On Mon, Sep 13, 2004 at 11:36:28AM +0200, Axel Thimm wrote: > On Mon, Sep 13, 2004 at 03:05:44AM +0200, Axel Thimm wrote: > > On Fri, Sep 03, 2004 at 06:46:18AM -0400, Jakub Jelinek wrote: > > > On Fri, Sep 03, 2004 at 12:40:53PM +0200, Axel Thimm wrote: > > > > How can I debug the memory consumption on this box? Which figures are > > > > the ones to look for and which ones do accumulate for the OOM killer? > > > > > > IMHO best would be to install 32-bit and 64-bit httpd side by side, > > > > It turns out that memory gets consumed and not returned back to the > > system independent of httpd (the oom-killer just strikes there first). The bug has now been identified as a kernel memory leak in 2.6.8-1.521 and 2.6.8-1.541 on a x86_64 system running ia32 bit apps in IA32_EMULATION mode. https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132947 It is easily reproducable, and I wonder why there have been so few reports on the net on this (kernel 2.6.8-1.521 has been released before a month). Probably most x86_64 users are running in 64 bits only and don't observe the leak. Any references/fixes are more than welcome. :) > > On an FC2/x86_64 system (Tyan S2880 with one processor only) with > > untained 2.6.8-1.521 on 1GB RAM simple compilations can eat up all the > > memory. I trimmed down such a system up to basic networking to detect > > which processes were locking the memory, and no userland processes are > > holding the memory. But almost all memory is flagged as "used" (with > > negligible size of buffers and cache). > > > > Is this a kernel memory leak? Any other information I should collect? > > > > (I still cannot judge whether the change from kernel 2.4 to 2.6 or the > > architecture change i386 to x86_64 is responsible for this due to lack > > of different combinations) > > > > # free > > total used free shared buffers cached > > Mem: 1027016 1022600 4416 0 992 7288 > > -/+ buffers/cache: 1014320 12696 > > Swap: 2047992 4496 2043496 > > # vmstat -a > > procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- > > r b swpd free inact active si so bi bo in cs us sy id wa > > 0 0 4496 4352 4548 6556 1 1 399 80 1517 162 2 2 88 8 > > # cat /proc/meminfo > > MemTotal: 1027016 kB > > MemFree: 4352 kB > > Buffers: 1008 kB > > Cached: 7316 kB > > SwapCached: 1148 kB > > Active: 6528 kB > > Inactive: 4536 kB > > HighTotal: 0 kB > > HighFree: 0 kB > > LowTotal: 1027016 kB > > LowFree: 4352 kB > > SwapTotal: 2047992 kB > > SwapFree: 2043496 kB > > Dirty: 236 kB > > Writeback: 0 kB > > Mapped: 5296 kB > > Slab: 14388 kB > > Committed_AS: 535496 kB > > PageTables: 494900 kB > > VmallocTotal: 536870911 kB > > VmallocUsed: 1568 kB > > VmallocChunk: 536869323 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > Hugepagesize: 2048 kB > > # ps uaxwwf > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 1 0.0 0.0 3472 428 ? S Sep12 0:01 init [3] > > root 2 0.0 0.0 0 0 ? SWN Sep12 0:00 [ksoftirqd/0] > > root 3 0.0 0.0 0 0 ? SW< Sep12 0:00 [events/0] > > root 4 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [khelper] > > root 5 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kacpid] > > root 30 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kblockd/0] > > root 44 0.0 0.0 0 0 ? SW Sep12 0:00 \_ [pdflush] > > root 45 0.0 0.0 0 0 ? SW Sep12 0:02 \_ [pdflush] > > root 47 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [aio/0] > > root 186 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [ata/0] > > root 31 0.0 0.0 0 0 ? SW Sep12 0:00 [khubd] > > root 46 0.0 0.0 0 0 ? SW Sep12 0:01 [kswapd0] > > root 151 0.0 0.0 0 0 ? SW Sep12 0:00 [kseriod] > > root 188 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_0] > > root 189 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_1] > > root 204 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] > > root 339 0.0 0.0 2336 216 ? S< Sep12 0:00 udevd > > root 896 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] > > root 897 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] > > root 898 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] > > root 899 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] > > root 1637 0.0 0.0 0 0 ? SW< Sep12 0:00 [krfcommd] > > root 1946 0.0 0.0 18104 748 ? S Sep12 0:00 /usr/sbin/sshd > > root 5189 0.0 0.1 37540 1056 ? S 02:04 0:00 \_ sshd: root@pts/0 > > root 5195 0.0 0.0 45656 1020 pts/0 S 02:04 0:00 | \_ -bash > > root 5255 0.0 0.1 104764 1892 pts/0 S 02:04 0:00 | \_ gkrellm > > root 29075 0.0 0.0 44836 500 pts/0 S 02:38 0:00 | \_ sleep 10 > > root 6119 0.0 0.0 37284 1020 ? S 02:19 0:00 \_ sshd: root@pts/1 > > root 6133 0.0 0.1 45656 1120 pts/1 S 02:19 0:00 | \_ -bash > > root 29079 0.0 0.0 44476 924 pts/1 S 02:38 0:00 | \_ /bin/sh ./memory.sh > > root 29083 0.0 0.0 5228 784 pts/1 R 02:38 0:00 | \_ ps uaxwwf > > root 6193 0.0 0.0 37284 1020 ? S 02:20 0:00 \_ sshd: root@pts/2 > > root 6212 0.0 0.1 45656 1136 pts/2 S 02:20 0:00 | \_ -bash > > root 29077 0.0 0.1 35936 1932 ? S 02:38 0:00 \_ sshd: bin [priv] > > sshd 29078 0.0 0.1 19448 1120 ? S 02:38 0:00 \_ sshd: bin [net] > > root 2542 0.0 0.0 2344 272 tty1 S Sep12 0:00 /sbin/mingetty tty1 > > root 2543 0.0 0.0 2344 272 tty2 S Sep12 0:00 /sbin/mingetty tty2 > > root 2544 0.0 0.0 2344 272 tty3 S Sep12 0:00 /sbin/mingetty tty3 > > root 2545 0.0 0.0 2344 276 tty4 S Sep12 0:00 /sbin/mingetty tty4 > > root 2546 0.0 0.0 2344 276 tty5 S Sep12 0:00 /sbin/mingetty tty5 > > root 2547 0.0 0.0 2344 276 tty6 S Sep12 0:00 /sbin/mingetty tty6 > > > > > -- Axel.Thimm at ATrpms.net
Attachment:
pgplVo7KY6Xi2.pgp
Description: PGP signature
