Re: [00/41] Large Blocksize Support V7 (adds memmap support)

[Jörn Engel <joern@logfs.org>]

On Tue, 11 September 2007 04:52:19 +1000, Nick Piggin wrote:
> On Tuesday 11 September 2007 16:03, Christoph Lameter wrote:
> 
> > 5. VM scalability
> >    Large block sizes mean less state keeping for the information being
> >    transferred. For a 1TB file one needs to handle 256 million page
> >    structs in the VM if one uses 4k page size. A 64k page size reduces
> >    that amount to 16 million. If the limitation in existing filesystems
> >    are removed then even higher reductions become possible. For very
> >    large files like that a page size of 2 MB may be beneficial which
> >    will reduce the number of page struct to handle to 512k. The variable
> >    nature of the block size means that the size can be tuned at file
> >    system creation time for the anticipated needs on a volume.
> 
> The idea that there even _is_ a bug to fail when higher order pages
> cannot be allocated was also brushed aside by some people at the
> vm/fs summit. I don't know if those people had gone through the
> math about this, but it goes somewhat like this: if you use a 64K
> page size, you can "run out of memory" with 93% of your pages free.
> If you use a 2MB page size, you can fail with 99.8% of your pages
> still free. That's 64GB of memory used on a 32TB Altix.

While I agree with your concern, those numbers are quite silly.  The
chances of 99.8% of pages being free and the remaining 0.2% being
perfectly spread across all 2MB large_pages are lower than those of SHA1
creating a collision.  I don't see anyone abandoning git or rsync, so
your extreme example clearly is the wrong one.

Again, I agree with your concern, even though your example makes it look
silly.

Jörn

-- 
You can't tell where a program is going to spend its time. Bottlenecks
occur in surprising places, so don't try to second guess and put in a
speed hack until you've proven that's where the bottleneck is.
-- Rob Pike
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/