Re: [patch 3/3] cpusets: add memory_spread_user option

[David Rientjes <rientjes@google.com>]

On Fri, 26 Oct 2007, Paul Jackson wrote:

>  1) If you want the current behaviour, where set_mempolicy(MPOL_INTERLEAVE)
>     calls mean what they say, and cpusets tries as best it can (imperfectly)
>     to honor those memory policy calls, even in the face of changing cpusets,
>     then leave memory_spread_user turned off (the default, right?)
>  2) If you want MPOL_INTERLEAVE tasks to interleave user memory across all
>     nodes in the cpuset, whatever that might be, then enable memory_spread_user.
> 
> This is admittedly less flexible than your patch provided, but I am
> attracted to the simpler API - easier to explain.
> 

That seems to follow the convention better with respect to 
memory_spread_page and memory_spread_slab anyway.  Either all the tasks 
attached to the cpuset get that behavior or none of them do.  We can do 
the same for memory_spread_user.

Sounds good.

> This does beg yet another question: shouldn't memory_spread_user force
> interleaving of user memory -regardless- of mempolicy.
> 

Sure, I don't see any compelling reason why it shouldn't.

> And yet another question: what about the MPOL_BIND mempolicy?  It too,
> to a lesser extent, has the same problems with cpusets that shrink and
> then expand.  Several tasks in a cpuset with multiple nodes could carefully
> bind to a separate node each, but then end up collapsed all onto the same
> node if the cpuset was shrunk to one node and then expanded again.
> 

Hmm.  At some point we're going to have to just say that if you use 
mempolicies such as MPOL_BIND in your application and then you insanely 
take those nodes away from your application via cpusets, that you are 
actually getting exactly what you asked for.

There's two ways to fix that: try to remap the MPOL_BIND nodes onto the 
new set of mems_allowed regardless of the cardinality of the two sets, or 
refuse to update the nodemask of the cpuset if you're taking one or more 
nodes away from an attached task that has such a policy.  I favor the 
former because, in conjunction with a sane memory_migrate setting, it 
shouldn't actually matter that much.  The memory you previously allocated 
will still be on the removed nodes; only your future allocations will 
actually respect the new nodemask.

The MPOL_INTERLEAVE case is more interesting because we're trying to 
reduce bus contention and decrease our latency with quicker memory access.  
So, using the true definition of a node as a premise, we should get better 
results in terms of performance if we expand the nodemask as much as 
possible.  That's exactly what we've been trying to address: when an 
application's mems_allowed is expanded to allow more nodes, the 
application is unaware of the change and can't take advantage of the 
it (without the get_mempolicy() - set_mempolicy() loop).  That's my whole 
case for why cpusets should be modifying MPOL_INTERLEAVE policies in the 
first place: because they are the ones that allowed access to more memory.

> On a different point, we could, if it was worth the extra bit of code,
> improve the current code's handling of mempolicy rebinding when the
> cpuset adds memory nodes.  If we kept both the original cpusets
> mems_allowed, and the original MPOL_INTERLEAVE nodemask requested by
> the user in a call to set_mempolicy, then we could rebind (nodes_remap)
> the currently active policy v.nodes using that pair of saved masks to
> guide the rebinding.  This way, if say a cpuset shrunk, then regrew back
> to its original size (original number of nodes) we would end up
> replicating the original MPOL_INTERLEAVE request, cpuset relative.

Keeping a copy of the nodemask passed to set_mempolicy() in struct 
mempolicy is an interesting idea and could, with the logic you describe, 
help guide the remapping as the set of allowed nodes changes.  We'd have 
two interleaved nodemasks, the actual (pol->v.nodes) and the requested 
(something like pol->passed_nodemask).  get_mempolicy() would always 
return the actual nodemask so the application can be aware of what it has 
access to and what it doesn't.  I like it.

		David
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/