Hi,
Once in a long while, oomkiller seems to decide my server is out of memory and
starts killing off processes, until I reboot. This machine runs stable most of
the time, so it's pretty confusing. I have the same issue occasionally on
another similar box. RAM consumption never goes over 3GB (out of 4GB). Swap
usage is virtually zero, and there's plenty of free disk space. Average load
is around 1.2.
I've been googling quite a bit, and the only relevant looking url was
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=837184&admit=-682735245+1131261645843+28353475
Unfortunately it was completely useless.
Can anyone give me some hints as to what might be causing this, and what I can
do about it?
TIA,
Rhesa Rozendaal
Hardware:
Dual Xeon 3.0GHz
4GB RAM
2x 100GB disk in software raid 0
Kernel version:
Linux version 2.6.10-1.760_FC3smp (bhcompile@xxxxxxxxxxxxxxxxxxxxx) (gcc
version 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)) #1 SMP Wed Feb 2 00:29:03 EST 2005
Oomkiller log message example:
Nov 5 18:15:20 web-fed-1 kernel: oom-killer: gfp_mask=0xd0
Nov 5 18:15:20 web-fed-1 kernel: Mem-info:
Nov 5 18:15:20 web-fed-1 kernel: DMA per-cpu:
Nov 5 18:15:20 web-fed-1 kernel: cpu 0 hot: low 2, high 6, batch 1
Nov 5 18:15:20 web-fed-1 kernel: cpu 0 cold: low 0, high 2, batch 1
Nov 5 18:15:20 web-fed-1 kernel: cpu 1 hot: low 2, high 6, batch 1
Nov 5 18:15:20 web-fed-1 kernel: cpu 1 cold: low 0, high 2, batch 1
Nov 5 18:15:20
web-fed-1 kernel: cpu 2 hot: low 2, high 6, batch 1
Nov 5 18:15:20 web-fed-1 kernel: cpu 2 cold: low 0, high 2, batch 1
Nov 5 18:15:20
web-fed-1 kernel: cpu 3 hot: low 2, high 6, batch 1
Nov 5 18:15:20 web-fed-1 kernel: cpu 3 cold: low 0, high 2, batch 1
Nov 5 18:15:20 web-fed-1 kernel: Normal per-cpu:
Nov 5 18:15:20 web-fed-1 kernel: cpu 0 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 0 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 1 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 1 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 2 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 2 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 3 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 3 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: HighMem per-cpu:
Nov 5 18:15:22 web-fed-1 kernel: cpu 0 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 0 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 1 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 1 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 2 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 2 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 3 hot: low 32, high 96, batch 16
Nov 5 18:15:22 web-fed-1 kernel: cpu 3 cold: low 0, high 32, batch 16
Nov 5 18:15:22 web-fed-1 kernel:
Nov 5 18:15:22 web-fed-1 kernel: Free pages: 1130660kB (1126848kB
HighMem)
Nov 5 18:15:22 web-fed-1 kernel: Active:439489 inactive:93772 dirty:0
writeback:0 unstable:0 free:282665 slab:216511 mapped:101178 pagetables:2553
Nov 5 18:15:22 web-fed-1 kernel: DMA free:68kB min:68kB low:84kB high:100kB
active:4kB inactive:0kB present:16384kB pages_scanned:2 all_unreclaimable? no
Nov 5 18:15:22 web-fed-1 kernel: protections[]: 0 0 0
Nov 5 18:15:22 web-fed-1 kernel: Normal free:3744kB min:3756kB low:4692kB
high:5632kB active:116kB inactive:456kB present:901120kB pages_scanned:910
all_unreclaimable? yes
Nov 5 18:15:22 web-fed-1 kernel: protections[]: 0 0 0
Nov 5 18:15:22 web-fed-1 kernel: HighMem free:1126848kB min:512kB low:640kB
high:768kB active:1757836kB inactive:374632kB present:3407872kB
pages_scanned:0 all_unreclaimable? no
Nov 5 18:15:22 web-fed-1 kernel: protections[]: 0 0 0
Nov 5 18:15:22 web-fed-1 kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
Nov 5 18:15:22 web-fed-1 kernel: Normal: 0*4kB 6*8kB 1*16kB 1*32kB 1*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3744kB
Nov 5 18:15:22 web-fed-1 kernel: HighMem: 1122*4kB 14951*8kB 10664*16kB
15430*32kB 4673*64kB 299*128kB 4*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
1126848kB
Nov 5 18:15:22 web-fed-1 kernel: Swap cache: add 57479, delete 57400, find
30261/33726, race 0+0
Nov 5 18:15:22 web-fed-1 kernel: Free swap: 8385360kB
Nov 5 18:15:22 web-fed-1 kernel: 1081344 pages of RAM
Nov 5 18:15:22 web-fed-1 kernel: 819184 pages of HIGHMEM
Nov 5 18:15:22 web-fed-1 kernel: 42639 reserved pages
Nov 5 18:15:22 web-fed-1 kernel: 271512 pages shared
Nov 5 18:15:22 web-fed-1 kernel: 79 pages swap cached
Nov 5 18:15:22 web-fed-1 kernel: Out of Memory: Killed process 9456 (httpd).
