Re: [Lhms-devel] [PATCH 0/7] Fragmentation Avoidance V19 - Summary

[Nick Piggin <nickpiggin@yahoo.com.au>]

Mel Gorman wrote:

> Ok. To me, the rest of the thread are beating around the same points and
> no one is giving ground. The points are made so lets summarise. Apologies
> if anything is missing.

Thanks for attempting a summary of a difficult topic. I have a couple
of suggestions.

> Who cares
> =========
>   Physical hotplug remove: Vendors of the hardware that support this -
> 	Fujitsu, HP (I think), IBM etc
>
>   Virtualization hotplug remove: Sellers of virtualization software, some
> 	hardware like any IBM machine that lists LPAR in it's list of
> 	features.  Probably software solutions like Xen are also affected
> 	if they want to be able to grow and shrink the virtual machines on
> 	demand

Ingo said that Xen is fine with per page granular freeing - this covers
embedded, desktop and small server users of VMs into the future I'd say.

>   High order allocations: Ultimately, hugepage users. Today, that is a
> 	feature only big server users like Oracle care about. In the
> 	future I reckon applications will be able to use them for things
> 	like backing the heap by huge pages. Other users like GigE,
> 	loopback devices with large MTUs, some filesystem like CIFS are
> 	all interested although they are also been told use use smaller
> 	pages.

I think that saying its now OK to use higher order allocations is wrong
because as I said even with your patches they are going to run into
problems.

Actually I think one reason your patches may perform so well is because
there aren't actually a lot of higher order allocations in the kernel.

I think that probably leaves us realistically with demand hugepages,
hot unplug memory, and IBM lpars?


> Pros/Cons of Solutions
> ======================
>
> Anti-defrag Pros
>   o Aim9 shows no significant regressions (.37% on page_test). On some
>     tests, it shows performance gains (> 5% on fork_test)
>   o Stress tests show that it manages to keep fragmentation down to a far
>     lower level even without teaching kswapd how to linear reclaim

This sounds like a kind of funny test to me if nobody is actually
using higher order allocations.

When a higher order allocation is attempted, either you will satisfy
it from the kernel region, in which case the vanilla kernel would
have done the same. Or you satisfy it from an easy-reclaim contiguous
region, in which case it is no longer an easy-reclaim contiguous
region.

>   o Stress tests with a linear reclaim experimental patch shows that it
>     can successfully find large contiguous chunks of memory
>   o It is known to help hotplug on PPC64
>   o No tunables. The approach tries to manage itself as much as possible

But it has more dreaded heuristics :P

>   o It exists, heavily tested, and synced against the latest -mm1
>   o Can be compiled away be redefining the RCLM_* macros and the
>     __GFP_*RCLM flags
>
> Anti-defrag Cons
>   o More complexity within the page allocator
>   o Adds a new layer onto the allocator that effectively creates subzones
>   o Adding a new concept that maintainers have to work with
>   o Depending on the workload, it fragments anyway
>
> New Zone Pros
>   o Zones are a well known and understood concept
>   o For people that do not care about hotplug, they can easily get rid of it
>   o Provides reliable areas of contiguous groups that can be freed for
>     HugeTLB pages going to userspace
>   o Uses existing zone infrastructure for balancing
>
> New Zone Cons
>   o Zones historically have introduced balancing problems
>   o Been tried for hotplug and dropped because of being awkward to work with
>   o It only helps hotplug and potentially HugeTLB pages for userspace
>   o Tunable required. If you get it wrong, the system suffers a lot

Pro: it keeps IBM mainframe and pseries sysadmins in a job ;) Let
them get it right.

>   o Needs to be planned for and developed

Yasunori Goto had patches around from last year. Not sure what sort
of shape they're in now but I'd think most of the hard work is done.

> Scenarios
> =========
>
> Lets outline some situations then or workloads that can occur
>
> 1. Heavy job running that consumes 75% of physical memory. Like a kernel
>    build
>
>   Anti-defrag: It will not fragment as it will never have to fallback.High
> 	order allocations will be possible in the remaining 25%.
>   Zone-based: After been tuned to a kernel build load, it will not
> 	fragment. Get the tuning wrong, performance suffers or workload
> 	fails. High order allocations will be possible in the remaining 25%.

You don't need to continually tune things for each and every possible
workload under the sun. It is like how we currently drive 16GB highmem
systems quite nicely under most workloads with 1GB of normal memory.
Make that an 8:1 ratio if you're worried.

[snip]

> I've tried to be as objective as possible with the summary.
>
> > From the points above though, I think that anti-defrag gets us a lot of
> the way, with the complexity isolated in one place. It's downside is that
> it can still break down and future work is needed to stop it degrading
> (kswapd cleaning UserRclm areas and page migration when we get really
> stuck). Zone-based is more reliable but only addresses a limited
> situation, principally hotplug and it does not even go 100% of the way for
> hotplug. 

To me it seems like it solves the hotplug, lpar hotplug, and hugepages
problems which seem to be the main ones.

> It also depends on a tunable which is not cool and it is static.

I think it is very cool because it means the tiny minority of Linux
users who want this can do so without impacting the rest of the code
or users. This is how Linux has been traditionally run and I still
have a tiny bit of faith left :)

> If we make the zones growable+shrinkable, we run into all the same
> problems that anti-defrag has today.

But we don't have the extra zones layer that anti defrag has today.

And anti defrag needs limits if it is to be reliable anyway.

--
SUSE Labs, Novell Inc.

Send instant messages to your online friends http://au.messenger.yahoo.com 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/