Re: [RFC 0/3] Recursive reclaim (on __PF_MEMALLOC)

[Peter Zijlstra <a.p.zijlstra@chello.nl>]

On Wed, 2007-09-12 at 15:47 -0700, Christoph Lameter wrote:
> On Wed, 12 Sep 2007, Peter Zijlstra wrote:
> 
> > > assumes single critical user of memory. There are other consumers of 
> > > memory and if you have a load that depends on other things than networking 
> > > then you should not kill the other things that want memory.
> > 
> > The VM is a _critical_ user of memory. And I dare say it is the _most_
> > important user. 
> 
> The users of memory are various subsystems. The VM itself of course also 
> uses memory to manage memory but the important thing is that the VM 
> provides services to other subsystems

Exactly, and because it services every other subsystem and userspace,
its the most important one, if it doesn't work, nothing else will.

> > Every user of memory relies on the VM, and we only get into trouble if
> > the VM in turn relies on one of these users. Traditionally that has only
> > been the block layer, and we special cased that using mempools and
> > PF_MEMALLOC.
> > 
> > Why do you object to me doing a similar thing for networking?
> 
> I have not seen you using mempools for the networking layer. I would not 
> object to such a solution. It already exists for other subsystems.

Dude, listen, how often do I have to say this: I cannot use mempools for
the network subsystem because its build on kmalloc! What I've done is
build a replacement for mempools - a reserve system - that does work
similar to mempools but also provides the flexibility of kmalloc.

That is all, no more, no less.

> > The problem of circular dependancies on and with the VM is rather
> > limited to kernel IO subsystems, and we only have a limited amount of
> > them. 
> 
> The kernel has to use the filesystems and other subsystems for I/O. These 
> subsystems compete for memory in order to make progress. I would not 
> consider strictly them part of the VM. The kernel reclaim may trigger I/O 
> in multiple I/O subsystems simultaneously.

I'm confused by this, I've never claimed part of, or such a thing. All
I'm saying is that because of the circular dependency between the VM and
the IO subsystem used for swap (not file backed paging [*], just swap)
you have to do something special to avoid deadlocks.

[*] the dirty limit along with 'atomic' swap ensures that file backed
paging does not get into this tight spot.

> > You talk about something generic, do you mean an approach that is
> > generic across all these subsystems?
> 
> Yes an approach that is fair and does not allow one single subsystem to 
> hog all of memory.

I do no such thing! My reserve system works much like mempools, you
reserve a certain amount of pages and use no more.

> > If so, my approach would be it, I can replace mempools as we have them
> > with the reserve system I introduce.
> 
> Replacing the mempools for the block layer sounds pretty good. But how do 
> these various subsystems that may live in different portions of the system 
> for various devices avoid global serialization and livelock through your 
> system? 

The reserves are spread over all kernel mapped zones, the slab allocator
is still per cpu, the page allocator tries to get pages from the nearest
node.

> And how is fairness addresses? I may want to run a fileserver on 
> some nodes and a HPC application that relies on a fiberchannel connection 
> on other nodes. How do we guarantee that the HPC application is not 
> impacted if the network services of the fileserver flood the system with 
> messages and exhaust memory?

The network system reserves A pages, the block layer reserves B pages,
once they start getting pages from the reserves they go bean counting,
once they reach their respective limit they stop.

The serialisation impact of the bean counting depends on how
fine-grained you place them, currently I only have a machine wide
network bean counter because the network subsystem is machine wide -
initially I tried to do something per net-device but that doesn't work
out. If someone more skilled in this area comes along and sees a better
way to place the bean counters they are free to do so.

But do notice that the bean counting is only done once we hit the
reserves, the normal mode of operation is not penalised by the extra
overhead thereof.

Also note that mempools also serialise their access once the backing
allocator fails, so I don't differ from them in that respect either.

Attachment: signature.asc
Description: This is a digitally signed message part