Re: Max Files Per Directory

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



At 8:53 AM -0400 7/16/05, Matthew Miller wrote:
>On Sat, Jul 16, 2005 at 08:04:22AM -0400, fredex wrote:
>> Now, for performance reasons, it is often not a good idea ot have
>> many thousands of files in a single directory. As the number of files
>> grows large the time it takes to access a file grows larger. I haven't
>> looked into this on any linux file system, but on other unixes I've
>> observed delays reaching up into the whole-second region when many thousands
>> of files are in a single directory.
>
>Shouldn't be a problem on a new install of ext3 on modern Linux. See
><http://lwn.net/Articles/11481/>. 

I suppose the simple test is to try it. Time (microseconds) per file to run
a script that creates files by touching them, to ls a few near the end, and
to remove all the files; see script at end of transcript.

#files    touch    ls       rm
------    -----    -----    -----
  1000      309      016      054
 10000      331      008      051
100000      326      007      357
200000      330      007     1065

I didn't want to try a million files.  Creation and listing seem to be
linear with the number of files, but rm seems quadratic.  I think this
indicates that in my current 2.6.12 FC3 kernel using Ext3 the directory
data structure is still a list and not a tree.

I expect that some sort of directory hierarchy to limit the number of files
per directory would still be a win.

Transcript follows.  Max line width 114 chars.

---------- cut here ----------
[tonyn@localhost ~]$ time ./manyfilestest 1000

real    0m0.309s
user    0m0.066s
sys     0m0.237s
[tonyn@localhost ~]$ time ls manyfiles/99*
manyfiles/99   manyfiles/991  manyfiles/993  manyfiles/995  manyfiles/997  manyfiles/999
manyfiles/990  manyfiles/992  manyfiles/994  manyfiles/996  manyfiles/998

real    0m0.016s
user    0m0.007s
sys     0m0.004s
[tonyn@localhost ~]$ time rm -rf manyfiles

real    0m0.054s
user    0m0.000s
sys     0m0.049s
[tonyn@localhost ~]$ time ./manyfilestest 10000

real    0m3.312s
user    0m0.728s
sys     0m2.518s
[tonyn@localhost ~]$ time ls manyfiles/999*
manyfiles/999   manyfiles/9991  manyfiles/9993  manyfiles/9995  manyfiles/9997  manyfiles/9999
manyfiles/9990  manyfiles/9992  manyfiles/9994  manyfiles/9996  manyfiles/9998

real    0m0.075s
user    0m0.051s
sys     0m0.021s
[tonyn@localhost ~]$ time rm -rf manyfiles

real    0m0.519s
user    0m0.006s
sys     0m0.501s
[tonyn@localhost ~]$ time ./manyfilestest 100000

real    0m32.561s
user    0m7.494s
sys     0m24.285s
[tonyn@localhost ~]$ time ls manyfiles/9999*
manyfiles/9999   manyfiles/99991  manyfiles/99993  manyfiles/99995  manyfiles/99997  manyfiles/99999
manyfiles/99990  manyfiles/99992  manyfiles/99994  manyfiles/99996  manyfiles/99998

real    0m0.686s
user    0m0.513s
sys     0m0.162s
[tonyn@localhost ~]$ time rm -rf manyfiles

real    0m35.653s
user    0m0.082s
sys     0m5.561s
[tonyn@localhost ~]$ time ./manyfilestest 200000

real    1m6.031s
user    0m15.243s
sys     0m47.962s
[tonyn@localhost ~]$ time ls manyfiles/9999*
manyfiles/9999   manyfiles/99991  manyfiles/99993  manyfiles/99995  manyfiles/99997  manyfiles/99999
manyfiles/99990  manyfiles/99992  manyfiles/99994  manyfiles/99996  manyfiles/99998

real    0m1.459s
user    0m1.113s
sys     0m0.269s
[tonyn@localhost ~]$ time rm -rf manyfiles

real    3m32.851s
user    0m0.159s
sys     0m12.205s
[tonyn@localhost ~]$ cat manyfilestest
#!/bin/bash

# Make lots of files in a directory.

TESTDIR="manyfiles"
MAXFILES=${1:-100000}

mkdir $TESTDIR
cd $TESTDIR
for (( i=0; i*10<$MAXFILES; i=i+1 )) ; do touch ${i}0 ${i}1 ${i}2 ${i}3 ${i}4 ${i}5 ${i}6 ${i}7 ${i}8 ${i}9 ; done
[tonyn@localhost ~]$
---------- cut here ----------
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson@xxxxxxxxxxxxxxxxx>
      '                              <http://www.georgeanelson.com/>


[Index of Archives]     [Current Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [Yosemite Photos]     [KDE Users]     [Fedora Tools]     [Fedora Docs]

  Powered by Linux