Crash with XFS on top of DRBD (DRBD 8.0.6 svn / Kernel 2.6.22)

[Laurent Caron <lcaron@lncsa.com>]

Hi,

I'm back with my crash, oomkiller..... problems on my DRBD cluster of 2
servers.

I compiled a 2.6.22 kernel with slab/slab debugging turned on.

Here is the last oom-killer message I got on that server.

I couldn't wait until a crash since a lot of users are working on it:

-----------------------------------------------
kernel: procmail invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
kernel: [<c0147dbe>] out_of_memory+0x69/0x197
kernel: [<c01492e1>] __alloc_pages+0x20a/0x294
kernel: [<c0149397>] __get_free_pages+0x2c/0x3a
kernel: [<c011ba87>] copy_process+0xa4/0x102d
kernel: [<c012aff2>] alloc_pid+0x16/0x240
kernel: [<c016048d>] kmem_cache_alloc+0x80/0x8a
kernel: [<c012aff2>] alloc_pid+0x16/0x240
kernel: [<c011cc57>] do_fork+0x9a/0x1c2
kernel: [<c01021d1>] sys_clone+0x36/0x3b
kernel: [<c0103cb2>] sysenter_past_esp+0x6b/0xa1
kernel: =======================
kernel: Mem-info:
kernel: DMA per-cpu:
kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    4: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    5: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    6: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: CPU    7: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0  
kernel: Normal per-cpu:
kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  63   Cold: hi:   62, btch:  15 usd:  51 
kernel: CPU    1: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:  54 
kernel: CPU    2: Hot: hi:  186, btch:  31 usd:  26   Cold: hi:   62, btch:  15 usd:  47 
kernel: CPU    3: Hot: hi:  186, btch:  31 usd:   8   Cold: hi:   62, btch:  15 usd:  61 
kernel: CPU    4: Hot: hi:  186, btch:  31 usd:  18   Cold: hi:   62, btch:  15 usd:  53 
kernel: CPU    5: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:  60 
kernel: CPU    6: Hot: hi:  186, btch:  31 usd:   6   Cold: hi:   62, btch:  15 usd:  57 
kernel: CPU    7: Hot: hi:  186, btch:  31 usd:  28   Cold: hi:   62, btch:  15 usd:  59 
kernel: HighMem per-cpu:
kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  98   Cold: hi:   62, btch:  15 usd:   8 
kernel: CPU    1: Hot: hi:  186, btch:  31 usd:   7   Cold: hi:   62, btch:  15 usd:  12 
kernel: CPU    2: Hot: hi:  186, btch:  31 usd:  85   Cold: hi:   62, btch:  15 usd:   4 
kernel: CPU    3: Hot: hi:  186, btch:  31 usd:  72   Cold: hi:   62, btch:  15 usd:   0 
kernel: CPU    4: Hot: hi:  186, btch:  31 usd:  21   Cold: hi:   62, btch:  15 usd:   6 
kernel: CPU    5: Hot: hi:  186, btch:  31 usd:  19   Cold: hi:   62, btch:  15 usd:  12 
kernel: CPU    6: Hot: hi:  186, btch:  31 usd: 175   Cold: hi:   62, btch:  15 usd:  12 
kernel: CPU    7: Hot: hi:  186, btch:  31 usd: 137   Cold: hi:   62, btch:  15 usd:   2 
kernel: Active:381850 inactive:9157 dirty:256 writeback:97 unstable:0
kernel: free:2519061 slab:22044 mapped:7487 pagetables:2163 bounce:0
kernel: DMA free:3564kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? yes 
kernel: lowmem_reserve[]: 0 873 12938
kernel: Normal free:7756kB min:3744kB low:4680kB high:5616kB active:116kB inactive:0kB present:894080kB pages_scanned:192 all_unreclaimable? yes 
kernel: lowmem_reserve[]: 0 0 96520
kernel: HighMem free:10064924kB min:512kB low:13456kB high:26404kB active:1527284kB inactive:36804kB present:12354560kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0
kernel: DMA: 9*4kB 15*8kB 3*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3564kB
kernel: Normal: 1495*4kB 10*8kB 1*16kB 0*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 7804kB
kernel: HighMem: 39234*4kB 109305*8kB 80630*16kB 52420*32kB 27267*64kB 9904*128kB 2822*256kB 871*512kB 377*1024kB 218*2048kB 257*4096kB = 10065264kB
kernel: Swap cache: add 43, delete 42, find 0/0, race 0+0 
kernel: Free swap  = 393204kB
kernel: Total swap = 393208kB
kernel: Free swap:       393204kB
kernel: 3342335 pages of RAM
kernel: 3112959 pages of HIGHMEM
kernel: 224356 reserved pages
kernel: 221558 pages shared
kernel: 1 pages swap cached
kernel: 256 pages dirty
kernel: 97 pages writeback
kernel: 7487 pages mapped
kernel: 22044 pages slab
kernel: 2163 pages pagetables
kernel: Out of memory: kill process 12069 (slapd) score 62386 or a child 
kernel: Killed process 12069 (slapd)
---------------------------------------------

Do anyone have any clue about what's happening ?

Thanks

Laurent



-- 
lcaron@lncsa.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/