Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

[Paul Jackson <pj@sgi.com>]

Andi wrote:
> GNU sort uses a merge sort with temporary files on disk. Not sure
> how much it keeps in memory during that, but it's probably less
> than 150MB. 

If I'm reading the source code for GNU sort correctly, then the
following snippet of shell code displays how much memory it uses
for its primary buffer on typical GNU/Linux systems:

    head -2 /proc/meminfo | awk '
	NR == 1 { memtotal = $2 }
	NR == 2 { memfree = $2 }
	END     {
		   if (memfree > memtotal/8)
		       m = memfree
		   else
		       m = memtotal/8
		   print "sort size:", m/2, "kB"
	}
    '

That is, over simplifying, GNU sort looks at the first two entries
in /proc/meminfo, which for example on a machine near me happen to be:

  MemTotal:      2336472 kB
  MemFree:        110600 kB

and then uses one-half of whichever is -greater- of MemTotal/8 or
MemFree.

... However ... for the typical GNU locate updatedb run, it is sorting
the list of pathnames for almost all files on the system, which is
usually larger than fits in one of these sized buffers.   So it ends up
using quite a few of the temporary files you mention, which tends to
chew up easily freed memory.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/