On Wed, 20 Jun 2007, Andi Kleen wrote:
> > - Device drivers may need to control memory allocations in
> > devices either because their DMA engines can only reach a
> > subsection of the system
>
> On what system does this happen? Not sure you can call that
> a "coherent system"
This is an issue for example on DSM systems where memory is virtualized by
transferring pages of memory on request. We are also getting into embedded
systems using NUMA for a variety of reason. Devices that can only reach
a subset of memory may come about there.
> > or because memory transfer
> > performance is superior in certain nodes of the system.
>
> Most architectures already give a sensible default for the coherent DMA
> mappings (node the device is attached to) For the others it is
> really not under driver control.
Yes I did some of that work. But that is still from the perspective of the
system as a whole. A device may have unique locality requirements. F.e. it
may be able to interweave memory transfers from multiple nodes for optimal
performance.
> Also controlling from the device where the submitted
> data is difficult unless you bind processes. If you do
> it just works, but if you don't want to (for most cases
> explicit binding is bad) it is hard.
It wont be difficult if the device has
1. A node number
2. An allocation policy
Then the allocation must be done as if we would be on that node.
> I would be definitely opposed to anything that exposes
> addresses as user interface.
Well its more the device driver telling the system where the stuff ought
to be best located.
> > 2. Memory policies need to support additional constraints
> >
> > - Restriction to a set of nodes. That is what we have today.
> >
> > - Restriction to a container or cpuset. Maybe restriction
> > to a set of containers?
>
> Why?
Because the sysadmin can set the containers up in a flexible way. Maybe we
want to segment a node into a couple of 100MB chunks and give various apps
access to it?
> > - Strict vs no strict allocations. A strict allocation needs
> > to fail if the constraints cannot be met. A non strict
> > allocation can fall back.
>
> That's already there -- that's the difference between PREFERED
> and BIND.
But its not available for interleave f.e.
> Regarding your page cache proposal: I think it's a bad
> idea, larger soft page sizes would be better.
I am not sure what you are talking about.
> > The esoteric
> > nature of memory policy semantics makes them difficult to comprehend.
>
> Exactly. It doesn't make sense to implement if you can't
> give it a good interface.
Right we need a clean interface and something that works in such a way
that people can understand it. The challenge is to boil down something
complex to a few simple mechanisms.
> > 7. Allocators must change
> >
> > Right now the policy is set by the process context which is bad because
> > one cannot specify a memory policy for an allocation. It must be possible
> > to pass a memory policy to the allocators and then get the memory
> > requested.
>
> We already can allocate on a node. If there is really demand
> we could also expose interleaved allocations, but again
> we would need a good user.
We have these bad hacks for shmem and for hugetlb where we have to set
policies in the context by creating a fake vma in order to get policy
applied.
If we want to allocate for a device then the device is the context and not
the process, same thing for shmem and hugetlb.
> Not sure it is useful for sl[aou]b.
If we do this then it needs to be consistently supported by the
allocators. Meaning the slab allocators would have to support a call where
you can pass a policy in and then objects need to be served in conformity
with that policy.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
