Re: 2.6.12.2 dies after 24 hours

[Lars Roland <lroland@gmail.com>]

On 7/12/05, Bron Gondwana <brong@fastmail.fm> wrote:
> We're also applying the attached patch.  There's a bug in reiserfs that
> gets tickled by our huge MMAP usage (it's amazing what really busy
> Cyrus daemons can do to a server, ouch).  It's fixed in generic_write,
> so we take the few percent performance hit for something that doesn't
> break!

Interesting - When I got the problem it was on mail servers under high
load (handling 60.000 emails pr. hour) with reiserfs as file system. I
have seen this problem on 5 different servers so I am confident that
it is not hardware failure.

Sometimes the server load just rises and then the server dies other
times the load rises but the kernel manages to get it back alive
filling up syslog with messages like this


---------
June 29 14:06:59 dkcphmx12 kernel: oom-killer: gfp_mask=0xd2
June 29 14:07:15 dkcphmx12 kernel: DMA per-cpu:
June 29 14:07:24 dkcphmx12 kernel: cpu 0 hot: low 2, high 6, batch 1
June 29 14:07:24 dkcphmx12 kernel: cpu 0 cold: low 0, high 2, batch 1
June 29 14:07:26 dkcphmx12 logger[17427]: *** SYSTEM UPDATE STARTED ***
June 29 14:07:26 dkcphmx12 kernel: cpu 1 hot: low 2, high 6, batch 1
June 29 14:07:26 dkcphmx12 kernel: cpu 1 cold: low 0, high 2, batch 1
June 29 14:07:26 dkcphmx12 kernel: cpu 2 hot: low 2, high 6, batch 1
June 29 14:07:26 dkcphmx12 kernel: cpu 2 cold: low 0, high 2, batch 1
June 29 14:07:26 dkcphmx12 kernel: cpu 3 hot: low 2, high 6, batch 1
June 29 14:07:26 dkcphmx12 kernel: cpu 3 cold: low 0, high 2, batch 1
June 29 14:07:26 dkcphmx12 kernel: Normal per-cpu:
June 29 14:07:26 dkcphmx12 kernel: cpu 0 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 0 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 1 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 1 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 2 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 2 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 3 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 3 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: HighMem per-cpu:
June 29 14:07:26 dkcphmx12 kernel: cpu 0 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 0 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 1 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 1 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 2 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 2 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 3 hot: low 62, high 186, batch 31
June 29 14:07:26 dkcphmx12 kernel: cpu 3 cold: low 0, high 62, batch 31
June 29 14:07:26 dkcphmx12 kernel:
June 29 14:07:26 dkcphmx12 kernel: Free pages:       49196kB (496kB HighMem)
June 29 14:07:26 dkcphmx12 kernel: Active:246580 inactive:244789
dirty:0 writeback:0 unstable:0 free:12299 slab:4271 mapped:494975
pagetables:2332

June 29 14:07:26 dkcphmx12 kernel: DMA free:8192kB min:68kB low:84kB
high:100kB active:2644kB inactive:2108kB present:16384kB
pages_scanned:35654 all_unreclaimable? yes

June 29 14:07:26 dkcphmx12 kernel: lowmem_reserve[]: 0 880 2031
June 29 14:07:26 dkcphmx12 kernel: Normal free:40508kB min:3756kB
low:4692kB high:5632kB active:400096kB inactive:396232kB
present:901120kB pages_scanned:1933718 all_unreclaimable? yes

June 29 14:07:26 dkcphmx12 kernel: lowmem_reserve[]: 0 0 9214
June 29 14:07:26 dkcphmx12 kernel: HighMem free:496kB min:512kB
low:640kB high:768kB active:583452kB inactive:580816kB
present:1179504kB pages_scanned:8651915 all_unreclaimable? yes

June 29 14:07:26 dkcphmx12 kernel: lowmem_reserve[]: 0 0 0
June 29 14:07:26 dkcphmx12 kernel: DMA: 0*4kB 2*8kB 1*16kB 1*32kB
1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8192kB

June 29 14:07:26 dkcphmx12 kernel: Normal: 251*4kB 38*8kB 6*16kB
0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 9*4096kB =
40508kB

June 29 14:07:26 dkcphmx12 kernel: HighMem: 0*4kB 0*8kB 1*16kB 1*32kB
1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 496kB

June 29 14:07:26 dkcphmx12 kernel: Swap cache: add 958067, delete
958073, find 182890/223964, race 0+1251
June 29 14:07:26 dkcphmx12 kernel: Free swap  = 0kB
June 29 14:07:26 dkcphmx12 kernel: Total swap = 2097136kB
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/