Re: [patch 2/2] cpusets: add interleave_over_allowed option

[Lee Schermerhorn <Lee.Schermerhorn@hp.com>]

On Sat, 2007-10-27 at 12:16 -0700, David Rientjes wrote:
> On Fri, 26 Oct 2007, David Rientjes wrote:
> 
> > Hacking and requiring an updated version of libnuma to allow empty 
> > nodemasks to be passed is a poor solution; if mempolicy's are supposed to 
> > be independent from cpusets, then what semantics does an empty nodemask 
> > actually imply when using MPOL_INTERLEAVE?  To me, it means the entire 
> > set_mempolicy() should be a no-op, and that's exactly how mainline 
> > currently treats it _as_well_ as libnuma.  So justifying this change in 
> > the man page is respectible, but passing an empty nodemask just doesn't 
> > make sense.
> > 
> 
> Another reason that passing an empty nodemask to set_mempolicy() doesn't 
> make sense is that libnuma uses numa_set_interleave_mask(&numa_no_nodes)
> to disable interleaving completely.
> 

David:  as we discussed when you contacted me off-list about this, the
libnuma API and the system call interface are two quite different APIs.
For example,  numa_set_interleave_mask(&numa_no_nodes) does not pass
MPOL_INTERLEAVE with an empty mask to set_mempolicy().  Rather it
"installs" an MPOL_DEFAULT policy which internally just deletes the
task's mempolicy, allowing fallback to system default policy.  I would
not propose to change this behavior, nor break libnuma in any way.

For other, who weren't involved in the off-list exchange, here's an
excerpt from my response to David:

[
At the libnuma level, I think we need an explicit
"numa_set_interleave_allowed()"--analogous to "numa_set_localalloc()".

The current "numa_alloc_interleaved()" should, I think, allocate on all
*allowed* nodes, rather than all nodes.  It can do this using the sys
call interface as defined.

Independent of cpuset-independent interleave, an application needs to
pass a valid subset of the current mems allowed to
"numa_alloc_interleaved_subset()".   An application can now obtain the
mems_allowed using the MPOL_F_MEMS_ALLOWED flag that I added, but we
need a libnuma wrapper for this as well.  [Yeah, this info can change at
any time, but that's always been the case....]

"numa_interleave_memory()" is essentially mbind(), I think [not looking
at the libnuma source code at this moment].  Maybe provide
"numa_interleave_memory_allowed(void *mem, size_t size)" ???

Finally, I think we need to add a query function:  
"nodemask_t numa_get_mems_allowed()" to return the mask of valid nodes
in the current context [cpuset].  This would just be a wrapper around
get_mempolicy() with the MPOL_F_MEMS_ALLOWED flag.
]

Couple of comments on the above:

1. "the sys call interface as defined" in the 2nd paragraph of the
except refers to my patch that uses null/empty nodemask to indicate "all
allowed".

2.  As this thread progresses, you've discussed relaxing the requirement
that applications pass a valid subset of mems_allowed.  I.e., something
that was illegal becomes legal.  An API change, I think.  But, a
backward compatible one, so that's OK, right? :-)

3. If we do change the semantics of the mempolicy system calls to allow
nodes outside of the cpuset, then maybe we don't need to query the mems
allowed.  I still find it useful, but not absolutely necessary--e.g., to
construct a nodemask that will be acceptable in the current cpuset.

4. I looked at libnuma source.  numa_interleave_memory() does use
mbind() which, again, does not complain about nodemasks that include
non-allowed nodes.

Another thing occurs to me:  perhaps numactl would need an additional
'nodes' specifier such as 'allowed'.  Alternatively, 'all' could be
redefined to me 'all allowed'.  This is independent of how you specify
'all allowed' to the system call.

Regards,
Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/