On Friday 27 July 2007 19:29:19 Andi Kleen wrote:
> > Any faults in that reasoning?
>
> GNU sort uses a merge sort with temporary files on disk. Not sure
> how much it keeps in memory during that, but it's probably less
> than 150MB. At some point the dirty limit should kick in and write back the
> data of the temporary files; so it's not quite the same as anonymous
> memory. But it's not that different given.
Yes, this should occur. But how many programs use temporary files like that?
>From what I can tell FireFox and OpenOffice both keep all their data in
memory, only using a single file for some buffering purposes. When they get
pushed out by a memory hog (either short term or long term) it takes several
seconds for them to be swapped back in. (I'm on a P4-1.3GHz machine with 1G
of ram and rarely run more than four programs (Mail Client, XChat, FireFox
and a console window) and I've seen this lag in FireFox when switching to it
after starting OOo. I've also seen the same sort of lag when exiting OOo.
I'll see about getting some numbers about this)
> It would be better to measure than to guess. At least Andrew's measurements
> on 128MB actually didn't show updatedb being really that big a problem.
I agree. As I've said previously, it isn't updatedb itself which causes the
problem. It's the way the VFS cache seems to just expand and expand - to the
point of evicting pages to make room for itself. However, I may be wrong
about that - I haven't actually tested it for myself, just looked at the
numbers and other information that has been posted in this thread.
> Perhaps some people have much more files or simply a less efficient
> updatedb implementation?
Yes, it could be the proliferation of files. It could also be some other sort
of problem that is exposing a corner-case in the VFS cache or the MM. I,
personally, am of the opinion that it is likely the aforementioned corner
case for people reporting the "updatedb" problem. If it is, then
swap-prefetch is just papering over the problem. However I do not have the
knowledge and understanding of the subsystems involved to be able to do much
more than make a (probably wrong) guess.
> I guess the people who complain here that loudly really need to supply
> some real numbers.
I've seen numerous "real numbers" posted about this. As was said earlier in
the thread "every time numbers are posted they are claimed to be no good".
But hey, nobodies perfect :)
Anyway, the discussion seems to be turning to the technical merits of
swap-prefetch...
Now, a completely different question:
During the research (and lots of thinking) I've been doing while this thread
has been going on I've often wondered why swap prefetch wasn't already in the
kernel. The problem of slow swap-in has long been known, and, given current
hardware, the optimal solution would be some sort of data prefetch - similar
to what is done to speed up normal disk reads. Swap prefetch looks like it
does exactly that. The algo could be argued over and/or improved (to suggest
ways to do that I'd have to give it more than a 10 minute look) but it does
provide a speed-up.
This speed increase will probably be enjoyed more by the home users, but the
performance increase could also help on enterprise systems.
Now I'll be the first one to admit that there is a trade-off there - it will
cause more power to be used because the disk's don't get a chance to spin
down (or go through a cycle every time the prefetch system starts) but that
could, potentially, be alleviated by having "laptop mode" switch it off.
(And no, I'm not claiming that it is perfect - but then, what is when its
first merged into the kernel?)
DRH
--
Dialup is like pissing through a pipette. Slow and excruciatingly painful.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
