Tracking down leaking applications

["Rahul Karnik" <rahul@genebrew.com>]

We are using an old P3 866 MHz with 512 MB of RAM to host a
development Apache server, and over the past few days have experienced
oom kills on a regular basis. Kernel being used is Fedora Core's
2.6.15-1_1831. A typical oom trace is shown below.

Apr  4 14:41:13 fedora kernel: oom-killer: gfp_mask=0x201d2, order=0
Apr  4 14:41:13 fedora kernel: Mem-info:
Apr  4 14:41:13 fedora kernel: DMA per-cpu:
Apr  4 14:41:13 fedora kernel: cpu 0 hot: low 0, high 0, batch 1 used:0
Apr  4 14:41:13 fedora kernel: cpu 0 cold: low 0, high 0, batch 1 used:0
Apr  4 14:41:13 fedora kernel: DMA32 per-cpu: empty
Apr  4 14:41:13 fedora kernel: Normal per-cpu:
Apr  4 14:41:13 fedora kernel: cpu 0 hot: low 0, high 186, batch 31 used:171
Apr  4 14:41:13 fedora kernel: cpu 0 cold: low 0, high 62, batch 15 used:49
Apr  4 14:41:13 fedora kernel: HighMem per-cpu: empty
Apr  4 14:41:13 fedora kernel: Free pages:        5320kB (0kB HighMem)
Apr  4 14:41:13 fedora kernel: Active:122584 inactive:645 dirty:0
writeback:0 unstable:0 free:1330 slab:2313 mapped:122684
pagetables:726
Apr  4 14:41:13 fedora kernel: DMA free:2068kB min:88kB low:108kB
high:132kB active:10280kB inactive:8kB present:16384kB
pages_scanned:12133 all_unreclaimable? yes
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 495 495
Apr  4 14:41:13 fedora kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable?
no
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 495 495
Apr  4 14:41:13 fedora kernel: Normal free:3252kB min:2800kB
low:3500kB high:4200kB active:480056kB inactive:2572kB
present:506880kB pages_scanned:671522 all_unreclaimable? yes
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 0 0
Apr  4 14:41:13 fedora kernel: HighMem free:0kB min:128kB low:128kB
high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 0 0
Apr  4 14:41:13 fedora kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2068kB
Apr  4 14:41:13 fedora kernel: DMA32: empty
Apr  4 14:41:13 fedora kernel: Normal: 137*4kB 2*8kB 2*16kB 1*32kB
1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3252kB
Apr  4 14:41:13 fedora kernel: HighMem: empty
Apr  4 14:41:13 fedora kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Apr  4 14:41:13 fedora kernel: Free swap  = 0kB
Apr  4 14:41:13 fedora kernel: Total swap = 0kB
Apr  4 14:41:13 fedora kernel: Free swap:            0kB
Apr  4 14:41:13 fedora kernel: 130816 pages of RAM
Apr  4 14:41:13 fedora kernel: 0 pages of HIGHMEM
Apr  4 14:41:13 fedora kernel: 2143 reserved pages
Apr  4 14:41:13 fedora kernel: 74708 pages shared
Apr  4 14:41:13 fedora kernel: 0 pages swap cached
Apr  4 14:41:13 fedora kernel: 0 pages dirty
Apr  4 14:41:13 fedora kernel: 0 pages writeback
Apr  4 14:41:13 fedora kernel: 122684 pages mapped
Apr  4 14:41:13 fedora kernel: 2313 pages slab
Apr  4 14:41:13 fedora kernel: 726 pages pagetables
Apr  4 14:41:13 fedora kernel: Out of Memory: Killed process 24188 (httpd).

The process killed has been either httpd or cronolog so far. For now,
I have upgraded to FC4's 2.6.16-1_1069 and added some swap, where
previously there was none.

Is there a way to:
- confirm that it is a userspace and not a kernel issue?
- track down the application that is leaking memory?

Thanks for the help,
Rahul
--
Rahul Karnik
rahul@genebrew.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/