About 2:10am tonite I noticed my linux box HD chugging away (I use it as
a server and was not using that system directly at the time). After a
few minutes I checked over to the console and it was showing out of
memory and killing processes. The box hadn't really have any problems
before. Any way to track down the issue? I'll post a log excerpt below.
The system's FC4, patched except for things the past few days, and
running some basic services.. dhcpd, named, ntpd, cups, and the standard
system services (sshd, autofs, crond, etc..). Actually, one non-typical
thing is it's running apcupsd to monitor a UPS. Any suggestions on what
or how to investigate what caused this?
Thanks,
Steve
Jul 29 02:01:01 zoo crond(pam_unix)[14124]: session opened for user root
by (uid=0)
Jul 29 02:01:01 zoo crond(pam_unix)[14124]: session closed for user root
Jul 29 02:12:31 zoo kernel: oom-killer: gfp_mask=0x1d2
Jul 29 02:12:31 zoo kernel: Mem-info:
Jul 29 02:12:31 zoo kernel: DMA per-cpu:
Jul 29 02:12:31 zoo kernel: cpu 0 hot: low 2, high 6, batch 1
Jul 29 02:12:31 zoo kernel: cpu 0 cold: low 0, high 2, batch 1
Jul 29 02:12:31 zoo kernel: cpu 1 hot: low 2, high 6, batch 1
Jul 29 02:12:31 zoo kernel: cpu 1 cold: low 0, high 2, batch 1
Jul 29 02:12:31 zoo kernel: Normal per-cpu:
Jul 29 02:12:31 zoo kernel: cpu 0 hot: low 62, high 186, batch 31
Jul 29 02:12:31 zoo kernel: cpu 0 cold: low 0, high 62, batch 31
Jul 29 02:12:31 zoo kernel: cpu 1 hot: low 62, high 186, batch 31
Jul 29 02:12:31 zoo kernel: cpu 1 cold: low 0, high 62, batch 31
Jul 29 02:12:31 zoo kernel: HighMem per-cpu: empty
Jul 29 02:12:31 zoo kernel:
Jul 29 02:12:31 zoo kernel: Free pages: 3872kB (0kB HighMem)
Jul 29 02:12:31 zoo kernel: Active:43884 inactive:43401 dirty:0
writeback:0 unstable:0 free:968 slab:5479 mapped:86081 pagetables:943
Jul 29 02:12:31 zoo kernel: DMA free:1572kB min:104kB low:128kB
high:156kB active:5168kB inactive:5160kB present:16384kB
pages_scanned:10779 a
ll_unreclaimable? yes
Jul 29 02:12:31 zoo kernel: lowmem_reserve[]: 0 367 367
Jul 29 02:12:31 zoo kernel: Normal free:2300kB min:2396kB low:2992kB
high:3592kB active:170620kB inactive:168320kB present:376272kB pages_scan
ned:350605 all_unreclaimable? yes
Jul 29 02:12:31 zoo kernel: oom-killer: gfp_mask=0xd2
Jul 29 02:12:31 zoo kernel: Mem-info:
Jul 29 02:12:31 zoo kernel: DMA per-cpu:
Jul 29 02:12:31 zoo kernel: cpu 0 hot: low 2, high 6, batch 1
Jul 29 02:12:31 zoo kernel: cpu 0 cold: low 0, high 2, batch 1
Jul 29 02:12:31 zoo kernel: cpu 1 hot: low 2, high 6, batch 1
Jul 29 02:12:31 zoo kernel: cpu 1 cold: low 0, high 2, batch 1
Jul 29 02:12:31 zoo kernel: Normal per-cpu:
Jul 29 02:12:31 zoo kernel: cpu 0 hot: low 62, high 186, batch 31
Jul 29 02:12:31 zoo kernel: cpu 0 cold: low 0, high 62, batch 31
Jul 29 02:12:31 zoo kernel: cpu 1 hot: low 62, high 186, batch 31
Jul 29 02:12:31 zoo kernel: cpu 1 cold: low 0, high 62, batch 31
Jul 29 02:12:31 zoo kernel: HighMem per-cpu: empty
Jul 29 02:12:31 zoo kernel:
Jul 29 02:12:31 zoo kernel: Free pages: 3872kB (0kB HighMem)
Jul 29 02:12:31 zoo kernel: Active:43947 inactive:43370 dirty:0
writeback:0 unstable:0 free:968 slab:5479 mapped:86081 pagetables:943
Jul 29 02:12:31 zoo kernel: DMA free:1572kB min:104kB low:128kB
high:156kB active:5168kB inactive:5160kB present:16384kB
pages_scanned:10779 a
ll_unreclaimable? yes
Jul 29 02:12:31 zoo kernel: lowmem_reserve[]: 0 367 367
Jul 29 02:12:31 zoo kernel: Normal free:2300kB min:2396kB low:2992kB
high:3592kB active:170620kB inactive:168320kB present:376272kB pages_scan
ned:350605 all_unreclaimable? yes
Jul 29 02:12:31 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:31 zoo kernel: HighMem free:0kB min:128kB low:160kB
high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaima
ble? no
Jul 29 02:12:31 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:31 zoo kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1572kB
Jul 29 02:12:31 zoo kernel: Normal: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB
1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2300kB
Jul 29 02:12:31 zoo kernel: HighMem: empty
Jul 29 02:12:31 zoo kernel: Swap cache: add 201839, delete 201614, find
20919/21519, race 0+0
Jul 29 02:12:31 zoo kernel: Free swap = 0kB
Jul 29 02:12:31 zoo kernel: Total swap = 786424kB
Jul 29 02:12:31 zoo kernel: Free swap: 0kB
Jul 29 02:12:31 zoo kernel: 98164 pages of RAM
Jul 29 02:12:31 zoo kernel: 0 pages of HIGHMEM
Jul 29 02:12:31 zoo kernel: 2162 reserved pages
Jul 29 02:12:31 zoo kernel: 6043 pages shared
Jul 29 02:12:31 zoo kernel: 225 pages swap cached
Jul 29 02:12:31 zoo kernel: Out of Memory: Killed process 25352 (named).
Jul 29 02:12:31 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:31 zoo kernel: HighMem free:0kB min:128kB low:160kB
high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaima
ble? no
Jul 29 02:12:31 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:31 zoo kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1572kB
Jul 29 02:12:31 zoo kernel: Normal: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB
1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2300kB
Jul 29 02:12:31 zoo kernel: HighMem: empty
Jul 29 02:12:31 zoo kernel: Swap cache: add 201847, delete 201614, find
20919/21520, race 0+0
Jul 29 02:12:31 zoo kernel: Free swap = 0kB
Jul 29 02:12:31 zoo kernel: Total swap = 786424kB
Jul 29 02:12:31 zoo kernel: Free swap: 0kB
Jul 29 02:12:31 zoo kernel: 98164 pages of RAM
Jul 29 02:12:31 zoo kernel: 0 pages of HIGHMEM
Jul 29 02:12:31 zoo kernel: 2162 reserved pages
Jul 29 02:12:31 zoo kernel: 6043 pages shared
Jul 29 02:12:31 zoo kernel: 233 pages swap cached
Jul 29 02:12:31 zoo kernel: Out of Memory: Killed process 25352 (named).
Jul 29 02:12:51 zoo kernel: oom-killer: gfp_mask=0x1d2
Jul 29 02:12:51 zoo kernel: Mem-info:
Jul 29 02:12:51 zoo kernel: DMA per-cpu:
Jul 29 02:12:51 zoo kernel: cpu 0 hot: low 2, high 6, batch 1
Jul 29 02:12:51 zoo kernel: cpu 0 cold: low 0, high 2, batch 1
Jul 29 02:12:51 zoo kernel: cpu 1 hot: low 2, high 6, batch 1
Jul 29 02:12:51 zoo kernel: cpu 1 cold: low 0, high 2, batch 1
Jul 29 02:12:51 zoo kernel: Normal per-cpu:
Jul 29 02:12:51 zoo kernel: cpu 0 hot: low 62, high 186, batch 31
Jul 29 02:12:51 zoo kernel: cpu 0 cold: low 0, high 62, batch 31
Jul 29 02:12:51 zoo kernel: cpu 1 hot: low 62, high 186, batch 31
Jul 29 02:12:51 zoo kernel: cpu 1 cold: low 0, high 62, batch 31
Jul 29 02:12:51 zoo kernel: HighMem per-cpu: empty
Jul 29 02:12:52 zoo kernel:
Jul 29 02:12:52 zoo kernel: Free pages: 3872kB (0kB HighMem)
Jul 29 02:12:52 zoo kernel: Active:44423 inactive:42838 dirty:0
writeback:0 unstable:0 free:968 slab:5467 mapped:86077 pagetables:928
Jul 29 02:12:52 zoo kernel: DMA free:1572kB min:104kB low:128kB
high:156kB active:5184kB inactive:5156kB present:16384kB
pages_scanned:11109 a
ll_unreclaimable? yes
Jul 29 02:12:52 zoo kernel: lowmem_reserve[]: 0 367 367
Jul 29 02:12:52 zoo kernel: Normal free:2300kB min:2396kB low:2992kB
high:3592kB active:172508kB inactive:166196kB present:376272kB pages_scan
ned:351253 all_unreclaimable? yes
Jul 29 02:12:52 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:52 zoo kernel: HighMem free:0kB min:128kB low:160kB
high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaima
ble? no
Jul 29 02:12:52 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:52 zoo kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1572kB
Jul 29 02:12:52 zoo kernel: Normal: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB
1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB oom-killer: gfp_mask=0xd0
Jul 29 02:12:52 zoo kernel: Mem-info:
Jul 29 02:12:52 zoo kernel: DMA per-cpu:
Jul 29 02:12:52 zoo kernel: cpu 0 hot: low 2, high 6, batch 1
Jul 29 02:12:52 zoo kernel: cpu 0 cold: low 0, high 2, batch 1
Jul 29 02:12:52 zoo kernel: cpu 1 hot: low 2, high 6, batch 1
Jul 29 02:12:52 zoo kernel: cpu 1 cold: low 0, high 2, batch 1
Jul 29 02:12:52 zoo kernel: Normal per-cpu:
Jul 29 02:12:52 zoo kernel: cpu 0 hot: low 62, high 186, batch 31
Jul 29 02:12:52 zoo kernel: cpu 0 cold: low 0, high 62, batch 31
Jul 29 02:12:52 zoo kernel: cpu 1 hot: low 62, high 186, batch 31
Jul 29 02:12:52 zoo kernel: cpu 1 cold: low 0, high 62, batch 31
Jul 29 02:12:52 zoo kernel: HighMem per-cpu: empty
Jul 29 02:12:52 zoo kernel:
Jul 29 02:12:52 zoo kernel: Free pages: 3872kB (0kB HighMem)
Jul 29 02:12:52 zoo kernel: Active:44423 inactive:42838 dirty:0
writeback:0 unstable:0 free:968 slab:5467 mapped:86077 pagetables:928
Jul 29 02:12:52 zoo kernel: DMA free:1572kB min:104kB low:128kB
high:156kB active:5184kB inactive:5156kB present:16384kB
pages_scanned:11109 a
ll_unreclaimable? yes
Jul 29 02:12:52 zoo kernel: lowmem_reserve[]: 0 367 367
Jul 29 02:12:52 zoo kernel: Normal free:2300kB min:2396kB low:2992kB
high:3592kB active:172508kB inactive:166196kB present:376272kB pages_scan
ned:351253 all_unreclaimable? yes
Jul 29 02:12:52 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:52 zoo kernel: HighMem free:0kB min:128kB low:160kB
high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaima
ble? no
Jul 29 02:12:52 zoo kernel: lowmem_reserve[]: 0 0 0
Jul 29 02:12:52 zoo kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1572kB
Jul 29 02:12:52 zoo kernel: Normal: 0*4096kB 1*4kB = 2300kB
Jul 29 02:12:52 zoo kernel: HighMem: empty
Jul 29 02:12:52 zoo kernel: Swap cache: add 202142, delete 201878, find
20968/21590, race 0+0
Jul 29 02:12:52 zoo kernel: Free swap = 0kB
Jul 29 02:12:52 zoo kernel: Total swap = 786424kB
Jul 29 02:12:52 zoo kernel: Free swap: 0kB
Jul 29 02:12:52 zoo kernel: 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB
0*512kB 0*1024kB 1*2048kB 0*4096kB = 2300kB
Jul 29 02:12:53 zoo kernel: HighMem: empty
Jul 29 02:12:53 zoo kernel: Swap cache: add 202142, delete 201878, find
20968/21590, race 0+0
Jul 29 02:12:53 zoo kernel: Free swap = 0kB
Jul 29 02:12:53 zoo kernel: Total swap = 786424kB
Jul 29 02:12:53 zoo kernel: Free swap: 0kB
Jul 29 02:12:53 zoo kernel: 98164 pages of RAM
Jul 29 02:12:53 zoo kernel: 98164 pages of RAM
Jul 29 02:12:53 zoo kernel: 0 pages of HIGHMEM
Jul 29 02:12:53 zoo kernel: 2162 reserved pages
Jul 29 02:12:53 zoo kernel: 5859 pages shared
Jul 29 02:12:53 zoo kernel: 264 pages swap cached
Jul 29 02:12:53 zoo kernel: Out of Memory: Killed process 2288
(mDNSResponder).
Jul 29 02:12:53 zoo kernel: 0 pages of HIGHMEM
Jul 29 02:12:53 zoo kernel: 2162 reserved pages
Jul 29 02:12:53 zoo kernel: 5859 pages shared
Jul 29 02:12:53 zoo kernel: 264 pages swap cached
Jul 29 02:12:53 zoo kernel: Out of Memory: Killed process 2288
(mDNSResponder).
Jul 29 02:12:53 zoo kernel: oom-killer: gfp_mask=0xd0
Jul 29 02:12:53 zoo kernel: Mem-info:
Jul 29 02:12:53 zoo kernel: DMA per-cpu:
Jul 29 02:12:53 zoo kernel: cpu 0 hot: low 2, high 6, batch 1
Jul 29 02:12:53 zoo kernel: cpu 0 cold: low 0, high 2, batch 1
Jul 29 02:12:53 zoo kernel: cpu 1 hot: low 2, high 6, batch 1
Jul 29 02:12:53 zoo kernel: cpu 1 cold: low 0, high 2, batch 1
Jul 29 02:12:53 zoo kernel: Normal per-cpu:
Jul 29 02:12:53 zoo kernel: cpu 0 hot: low 62, high 186, batch 31
Jul 29 02:12:53 zoo kernel: cpu 0 cold: low 0, high 62, batch 31
Jul 29 02:12:53 zoo kernel: cpu 1 hot: low 62, high 186, batch 31
Jul 29 02:12:53 zoo kernel: cpu 1 cold: low 0, high 62, batch 31
Jul 29 02:12:53 zoo kernel: HighMem per-cpu: empty
begin:vcard
fn:Steve Wardell
n:Wardell;Steve
org:Bantu Inc;Research and Development
adr:;;1819 L St NW #100;Washington;DC;20036;US
email;internet:steve@xxxxxxxxxxxxxx
tel;work:(202)822-3999
tel;fax:(202)822-9610
x-mozilla-html:TRUE
url:http://corp.bantu.com
version:2.1
end:vcard
