List members,
This post is a follow-up to another post titled "Large Ramdisk creation
causes crashes" in which the author states he was experiencing crashes
when filling his RAM disk to capacity. I have a process that is using a
large RAM disk to buffer I/O so I decided it would be a good idea to see
what happens on my system as the RAM disk fills, and to share the
results with the list.
I also have a few questions drawn from my observations at the end of
this post. I'd appreciate any information you could share with me
regarding them.
Hardware Profile
----------------
Machine: Dell PowerEdge 6850
CPU: Four Intel Xeon MP CPUs at 3.16 GHz - Hyperthreaded
RAM: 20 GB (4 GB System / 16 GB RAM Disk)
RAM Disk Setup
--------------
I edited the bootloader configuration file (/boot/grub/grub.conf) to
pass the ramdisk_size parameter to the kernel:
kernel /vmlinuz-2.6.12-1.1376_FC3smp ro root=LABEL=/ quiet
ramdisk_size=16777216
I appended lines to the /etc/rc.local init script to format and mount
the RAM disk at boot time:
# Formats and mounts the RAM disk used to buffer digital recordings.
/sbin/mke2fs -q -b 1024 -m 0 /dev/ram0
/bin/mount /dev/ram0 /digrec
RAM Disk Capacity Test
----------------------
As you can see, the machine has 20 GB of RAM with 4 GB available to the
system and a 16 GB RAM disk formatted to an ext2 filesystem with 1 KB
block sizes. To test filling the RAM disk to capacity, I used 'dd' to
create 16 files, each 1 GB in size. The filesystem itself poses some
overhead, so slightly less than 16 GB is available on the RAM disk.
This is fine, because it allowed the 16th file to show the results of
trying to push the RAM disk past capacity.
Prior to starting the test, I gathered a little information about the
state of the system:
=========================
[root@immlx15 ~]# uname -a
Linux immlx15.imm1 2.6.12-1.1376_FC3smp #1 SMP Fri Aug 26 23:51:16 EDT
2005 x86_64 x86_64 x86_64 GNU/Linux
[root@immlx15 ~]# ls -al /digrec
total 21
drwxr-xr-x 3 root root 1024 Nov 3 15:14 .
drwxr-xr-x 30 root root 4096 Nov 3 15:13 ..
drwx------ 2 root root 12288 Nov 3 15:14 lost+found
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 3.8M 16G 1% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 394064 20179568 0 281412 36568
-/+ buffers/cache: 76084 20497548
Swap: 2096472 0 2096472
=========================
Then I created each of the 16 files, gathering some information about
the state of the system after each one:
=========================
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test01 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 1.1G 15G 7% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 1558672 19014960 0 297980 1084960
-/+ buffers/cache: 175732 20397900
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test02 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 2.1G 14G 13% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 2727248 17846384 0 318496 2133824
-/+ buffers/cache: 274928 20298704
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test03 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 3.1G 13G 20% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 4795940 15777692 0 1239792 3182288
-/+ buffers/cache: 373860 20199772
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test04 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 4.1G 12G 26% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 6996320 13577312 0 2292160 4230720
-/+ buffers/cache: 473440 20100192
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test05 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 5.1G 11G 32% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 9196708 11376924 0 3344560 5279380
-/+ buffers/cache: 572768 20000864
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test06 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 6.1G 9.8G 39% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 11397080 9176552 0 4396888 6328112
-/+ buffers/cache: 672080 19901552
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test07 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 7.1G 8.8G 45% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 13597336 6976296 0 5449152 7376908
-/+ buffers/cache: 771276 19802356
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test08 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 8.1G 7.8G 52% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 15797592 4776040 0 6501480 8425380
-/+ buffers/cache: 870732 19702900
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test09 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 9.1G 6.8G 58% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 19162456 1411176 0 8717900 9473780
-/+ buffers/cache: 970776 19602856
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test10 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 11G 5.8G 64% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20326444 247188 0 8734332 10522308
-/+ buffers/cache: 1069804 19503828
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test11 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 12G 4.7G 71% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20536004 37628 0 9647868 9872152
-/+ buffers/cache: 1015984 19557648
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test12 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 13G 3.7G 77% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20534392 39240 0 10704276 8898944
-/+ buffers/cache: 931172 19642460
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test13 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 14G 2.7G 83% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20534268 39364 0 11756476 7931504
-/+ buffers/cache: 846288 19727344
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test14 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 15G 1.7G 90% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20535260 38372 0 12808764 6963976
-/+ buffers/cache: 762520 19811112
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test15 bs=16k count=65536
65536+0 records in
65536+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 16G 700M 96% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20536376 37256 0 13860900 5997380
-/+ buffers/cache: 678096 19895536
Swap: 2096472 0 2096472
[root@immlx15 ~]# dd if=/dev/zero of=/digrec/test16 bs=16k count=65536
dd: writing `/digrec/test16': No space left on device
44564+0 records in
44563+0 records out
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 16G 1.0K 100% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20534888 38744 0 14574392 5329124
-/+ buffers/cache: 631372 19942260
Swap: 2096472 4 2096468
=========================
After the test, I gathered more information about the state of the system:
=========================
[root@immlx15 ~]# ls -al /digrec
total 16506166
drwxr-xr-x 3 root root 1024 Nov 3 15:24 .
drwxr-xr-x 30 root root 4096 Nov 3 15:13 ..
drwx------ 2 root root 12288 Nov 3 15:14 lost+found
-rw-r--r-- 1 root root 1073741824 Nov 3 15:19 test01
-rw-r--r-- 1 root root 1073741824 Nov 3 15:20 test02
-rw-r--r-- 1 root root 1073741824 Nov 3 15:20 test03
-rw-r--r-- 1 root root 1073741824 Nov 3 15:20 test04
-rw-r--r-- 1 root root 1073741824 Nov 3 15:21 test05
-rw-r--r-- 1 root root 1073741824 Nov 3 15:21 test06
-rw-r--r-- 1 root root 1073741824 Nov 3 15:21 test07
-rw-r--r-- 1 root root 1073741824 Nov 3 15:21 test08
-rw-r--r-- 1 root root 1073741824 Nov 3 15:22 test09
-rw-r--r-- 1 root root 1073741824 Nov 3 15:22 test10
-rw-r--r-- 1 root root 1073741824 Nov 3 15:23 test11
-rw-r--r-- 1 root root 1073741824 Nov 3 15:23 test12
-rw-r--r-- 1 root root 1073741824 Nov 3 15:23 test13
-rw-r--r-- 1 root root 1073741824 Nov 3 15:24 test14
-rw-r--r-- 1 root root 1073741824 Nov 3 15:24 test15
-rw-r--r-- 1 root root 730124288 Nov 3 15:24 test16
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 16G 1.0K 100% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 20534268 39364 0 16774184 3229432
-/+ buffers/cache: 530652 20042980
Swap: 2096472 4 2096468
=========================
Finally, I deleted the files from the RAM disk and gathered more
information about the state of the system:
=========================
[root@immlx15 ~]# rm -f /digrec/*
rm: cannot remove `/digrec/lost+found': Is a directory
[root@immlx15 ~]# ls -al /digrec
total 21
drwxr-xr-x 3 root root 1024 Nov 3 15:26 .
drwxr-xr-x 30 root root 4096 Nov 3 15:13 ..
drwx------ 2 root root 12288 Nov 3 15:14 lost+found
[root@immlx15 ~]# df -h | grep /dev/ram0
/dev/ram0 16G 3.8M 16G 1% /digrec
[root@immlx15 ~]# free -k
total used free shared buffers cached
Mem: 20573632 17160724 3412908 0 16774300 10516
-/+ buffers/cache: 375908 20197724
Swap: 2096472 4 2096468
=========================
Conclusions
-----------
The system did not experience any problems filling the 16 GB RAM disk to
capacity. The creation of the 16th file failed gracefully with the
following message:
dd: writing `/digrec/test16': No space left on device
After the RAM disk was 50% full, adding files to it began to have a
significant impact on the memory available to the system itself. Note
the decline of free memory as the RAM disk goes from 52% capacity
(4,776,040 KB free) to 71% capacity (37,628 KB free) where free memory
bottoms out. The additional memory usage is accounted for by large
amounts of memory being used for buffers and cache.
Interestingly, even after deleting the files from the RAM disk almost 16
GB of memory was still being used for buffers.
This additional memory overhead may have been responsible for the
original poster's crashes. He reported out-of-memory errors so it seems
likely, but it is far from proven.
Questions
---------
As I stated before, the RAM disk is being used to buffer I/O that would
otherwise be going to disk. Writing to disk was causing a significant
hit to the performance of the application we were running, so the RAM
disk was a simple solution.
After the files are completely written to the RAM disk, they are
immediately moved to storage on a remote machine via NFS. I have
absolutely no interest in the contents of these files and they will
never be written to or read from by the local machine. This brings me
to a few questions:
Is the buffers/cache memory overhead necessary or of any benefit in my
scenario?
If not, how can I eliminate that overhead *only* when writing to the
RAM disk?
Is there any way to flush the memory being used for buffers/cache?
Is there any way to limit the memory available for buffers/cache?
How is memory usage prioritized? For instance, would memory needed
for a running process get priority over memory needed for buffers/cache?
How does the size of the files being written affect the amount of
memory that is allocated for buffers/cache?
If the memory being used for buffers/cache is not beneficial, I'd much
rather that it were available to the system instead. Since I'm trying
to minimize disk I/O, I really do *not* want any swapping to occur.
Note that 4 KB of swap memory is allocated once the RAM disk is filled
to capacity, but it is not deallocated after the RAM disk is cleared.
Please help me by providing answers to the questions above. I don't
claim to be an expert on RAM disks or Linux memory management, so there
is a good chance that I'm completely misinterpreting the data. If
that's the case and you can provide information to help me understand
the situation more clearly, it would be greatly appreciated.
Thank you very much,
Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer
