On Friday 27 July 2007 06:25:18 Mike Galbraith wrote:
> On Fri, 2007-07-27 at 03:00 -0700, Andrew Morton wrote:
> > On Fri, 27 Jul 2007 01:47:49 -0700 Andrew Morton
<[email protected]> wrote:
> > > More sophisticated testing is needed - there's something in
> > > ext3-tools which will mmap, page in and hold a file for you.
> >
> > So much for that theory. afaict mmapped, active pagecache is immune to
> > updatedb activity. It just sits there while updatedb continues munching
> > away at the slab and blockdev pagecache which it instantiated. I assume
> > we're never getting the VM into enough trouble to tip it over the
> > start-reclaiming-mapped-pages threshold (ie: /proc/sys/vm/swappiness).
> >
> > Start the updatedb on this 128MB machine with 80MB of mapped pagecache,
> > it falls to 55MB fairly soon and then never changes.
> >
> > So hrm. Are we sure that updatedb is the problem? There are quite a few
> > heavyweight things which happen in the wee small hours.
>
> The balance in _my_ world seems just fine. I don't let any of those
> system maintenance things run while I'm using the system, and it doesn't
> bother me if my working set has to be reconstructed after heavy-weight
> maintenance things are allowed to run. I'm not seeing anything I
> wouldn't expect to see when running a job the size of updatedb.
>
> -Mike
Do you realize you've totally missed the point?
It isn't about what is fine in the Kernel Developers world, but what is fine
in the *USERS* world.
There are dozens of big businesses pushing Linux for Enterprise performance.
Rather than discussing the merit of those patches - some of which just
improve the performance of a specific application by 1 or 2 percent - they
get a nod and go into the kernel. But when a group of users that don't
represent one of those businesses says "Hey, this helps with problems I see
on my system" there is a big discussion and ultimately those patches get
rejected. Why? Because they'll give an example using a program that they see
causing part of the problem and be told "Use program X - it does things
differently and shouldn't cause the problem" or "But what causes the problem
to happen? The patch treats a symptom of a larger problem".
The fucked up part of that is that the (mass of) kernel developers will see a
similar report saying "mySQL has a performance problem because of X, this
fixes it" and not blink twice - even if it is "treating the symptom and not
the cause". It's this attitude more than anything that caused Con
to "retire" - at least that is the impression I got from the interviews he's
given. (The exact impression was "I'm sick of the kernel developers doing
everything they can to help enterprise users and ignoring the home users")
So...
The problem:
Updatedb or another process that uses the FS heavily runs on a users 256MB
P3-800 (when it is idle) and the VFS caches grow, causing memory pressure
that causes other applications to be swapped to disk. In the morning the user
has to wait for the system to swap those applications back in.
Questions about it:
Q) Does swap-prefetch help with this?
A) [From all reports I've seen (*)] Yes, it does.
Q) Why does it help?
A) Because it pro-actively swaps stuff back-in when the memory pressure that
caused it to be swapped out is gone.
Q) What causes the problem?
A) The VFS layer not keeping a limited cache. Instead the VFS will chew
through available memory in the name of "increasing performance".
Solution(s) to the problem:
1) Limit the amount of memory the pagecache and other VFS caches can consume
2) Implement swap prefetch
If I had a (more) complete understanding of how the VFS cache(s) work I'd try
to code a patch to do #1 myself. Patches to do #2 already exist and have been
shown to work for the users that have tried it. My question is thus, simply:
What is the reason that it is argued against?(**)
DRH
PS: Yes, I realize I've repeated some people from earlier in this thread, but
it seems that people have been forgetting the point.
(*) I've followed this thread and all of its splinters. The reports that are
in them, where the person making the report has a system that has the limited
memory needed for the problem to exhibit itself, all show that swap-prefetch
helps.
(**) No, I won't settle for "Its treating a symptom". The fact is that this
isn't a *SYMPTOM* of anything. It treats the cause of the lag the users that
have less than (for the sake of argument) 1G of memory are seeing. And no,
changing userspace isn't a solution - updatedb may be the program that has
been used as an example, but there are others. The proper fix is to change
the kernel to either make the situation impossible (limit the VFS and other
kernel caches) or make the situation as painless as possible (implement swap
prefetch to alleviate the lag of swapping data back in).
--
Dialup is like pissing through a pipette. Slow and excruciatingly painful.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
