Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

[Dinakar Guniguntala <dino@in.ibm.com>]

On Sat, Apr 23, 2005 at 03:30:59PM -0700, Paul Jackson wrote:
> The top cpuset holds the kernel threads that are pinned to a particular
> cpu or node.  It's not right that their cpusets cpus_allowed is empty,
> which is what I guess the "0" in the cpus_allowed column above means.
> (Even if the "0" means CPU 0, that still conflicts with kernel threads
> on CPUs 1-7.)

Yes, I meant cpus_allowed is empty

> 
> We might get away with it on cpus, because we don't change the tasks
> cpus_allowed to match the cpusets cpus_allowed (we don't call
> set_cpus_allowed, from kernel/cpuset.c) _except_ when someone rebinds
> that task to its cpuset by writing its pid into the cpuset tasks file.
> So for as long as no one tries to rebind the per-cpu or per-node
> kernel threads, no one will notice that they in a cpuset with an
> empty cpus_allowed.

True.

>  4) There are some tasks that _do_ require to run on the same cpus as
>     the tasks you would assign to isolated cpusets.  These kernel threads,
>     such as for example the migration and ksoftirqd threads, must be setup
>     well before user code is run that can configure job specific isolated
>     cpusets, so these tasks need a cpuset to run in that can be created
>     during the system boot, before init (pid == 1) starts up.  This cpuset
>     is the top cpuset.

And those processes (kernel threads) will continue to run on their cpus

> 
> I don't understand why what's there now isn't sufficient.  I don't see
> that this patch provides any capability that you can't get just by
> properly placing tasks in cpusets that have the desired cpus and nodes.
> This patch leaves the per-cpu kernel threads with no cpuset that allows
> what they need, and it complicates the semantics of things, in ways that
> I still don't entirely understand.

You are forgetting the fact that the scheduler is still load balancing
across all CPUs and tries to pull tasks only to find that the task's
cpus_allowed mask prevents it from being moved

> 
> You don't need to isolate a set of cpus; you need to isolate a set of
> processes.  So long as you can create non-overlapping cpusets, and
> assign processes to them, I don't see where it matters that you cannot
> prohibit the creation of overlapping cpusets, or in the case of the top
> cpuset, why it matters that you cannot _disallow_ allowed cpus
> or memory nodes in existing cpusets.
> 

I am working on a minimalistic design right now and will get back in
a day or two

	-Dinakar
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/