After one of my servers had been running RH7.3 for over a year, with no problems, today I did a fresh installation of FC2 (fresh meaning I formatted all the partitions prior to installation,) and this is what I get after about five hours of uptime... [ see attached .txt file ]
Based on the time it happened, I'd say it was while there was some heavy NFS traffic. (This machine is a backup server for 13 others: they NFS mount one of its partitions, rsync, then unmount.) The time stamp corresponds with one of the other server (the 8th in fact) just starting off it's dump (the dumps happen sequentially.) And looking at the first OOPS, I see nfsd as the running process, and the second OOPS points at pdflush, but that's the extend of my knowledge of OOPSes. Can anyone else shed any light here?
It's frustrating to say the least, that 7.3 ran so well for so long, only to have FC2 blow up like this. So, any ideas anyone?
-- W | I haven't lost my mind; it's backed up on tape somewhere. +-------------------------------------------------------------------- Ashley M. Kirchner <mailto:ashley@xxxxxxxxxx> . 303.442.6410 x130 IT Director / SysAdmin / WebSmith . 800.441.3873 x130 Photo Craft Laboratories, Inc. . 3550 Arapahoe Ave. #6 http://www.pcraft.com ..... . . . Boulder, CO 80303, U.S.A.
May 31 22:50:37 cog kernel: Unable to handle kernel paging request at virtual address 88130004 May 31 22:50:37 cog kernel: printing eip: May 31 22:50:37 cog kernel: 0213124e May 31 22:50:37 cog kernel: *pde = 00000000 May 31 22:50:37 cog kernel: Oops: 0002 [#1] May 31 22:50:37 cog kernel: CPU: 0 May 31 22:50:37 cog kernel: EIP: 0060:[<0213124e>] Not tainted May 31 22:50:37 cog kernel: EFLAGS: 00010046 (2.6.5-1.358) May 31 22:50:37 cog kernel: EIP is at cache_alloc_refill+0xda/0x15f May 31 22:50:37 cog kernel: eax: 88130000 ebx: 21d81354 ecx: 05c0c000 edx: 21d3bc0c May 31 22:50:37 cog kernel: esi: 0000000f edi: 21d3bc0c ebp: 21d3bc00 esp: 0d358b08 May 31 22:50:37 cog kernel: ds: 007b es: 007b ss: 0068 May 31 22:50:37 cog kernel: Process nfsd (pid: 4249, threadinfo=0d358000 task=0d1331b0) May 31 22:50:37 cog kernel: Stack: 00000050 00000050 21d3bc00 00000246 216f1600 02131464 0044014c 216f1600 May 31 22:50:37 cog kernel: 21e38214 228ee7c4 021535d0 0044014c 216f1600 21e38214 0044014c 02153ecc May 31 22:50:37 cog kernel: 0044014c 208d6b80 216f1600 00000000 228ec0a9 1f90e02c 228fd500 208d6b80 May 31 22:50:37 cog kernel: Call Trace: May 31 22:50:37 cog kernel: [<02131464>] kmem_cache_alloc+0x3f/0x45 May 31 22:50:37 cog kernel: [<228ee7c4>] ext3_alloc_inode+0xf/0x3c [ext3] May 31 22:50:38 cog kernel: [<021535d0>] alloc_inode+0x13/0x175 May 31 22:50:38 cog kernel: [<02153ecc>] get_new_inode_fast+0xf/0x8b May 31 22:50:38 cog kernel: [<228ec0a9>] ext3_lookup+0x42/0x89 [ext3] May 31 22:50:38 cog kernel: [<0214b953>] __lookup_hash+0x70/0x89 May 31 22:50:38 cog kernel: [<0214b9c0>] lookup_one_len+0x4d/0x5b May 31 22:50:38 cog kernel: [<22a28005>] compose_entry_fh+0x74/0xb9 [nfsd] May 31 22:50:38 cog kernel: [<22a281c7>] encode_entry+0x17d/0x4c3 [nfsd] May 31 22:50:38 cog kernel: [<0210737b>] do_IRQ+0x15d/0x169 May 31 22:50:38 cog kernel: [<02122121>] in_group_p+0x30/0x56 May 31 22:50:38 cog kernel: [<228f4559>] ext3_permission+0x0/0x152 [ext3] May 31 22:50:38 cog kernel: [<228f4632>] ext3_permission+0xd9/0x152 [ext3] May 31 22:50:38 cog kernel: [<228f4559>] ext3_permission+0x0/0x152 [ext3] May 31 22:50:38 cog kernel: [<021faf85>] ide_build_sglist+0x2c/0x89 May 31 22:50:38 cog kernel: [<02116b21>] autoremove_wake_function+0x0/0x28 May 31 22:50:38 cog kernel: [<22a28537>] nfs3svc_encode_entry_plus+0x13/0x17 [nfsd] May 31 22:50:38 cog kernel: [<228e65f0>] ext3_readdir+0x305/0x3b5 [ext3] May 31 22:50:38 cog kernel: [<22a28524>] nfs3svc_encode_entry_plus+0x0/0x17 [nfsd] May 31 22:50:38 cog kernel: [<22a1dde6>] fh_verify+0x4a3/0x4bb [nfsd] May 31 22:50:38 cog kernel: [<0214eaba>] vfs_readdir+0x7a/0x9b May 31 22:50:38 cog kernel: [<22a28524>] nfs3svc_encode_entry_plus+0x0/0x17 [nfsd] May 31 22:50:38 cog kernel: [<228e65f0>] ext3_readdir+0x305/0x3b5 [ext3] May 31 22:50:38 cog kernel: [<22a28524>] nfs3svc_encode_entry_plus+0x0/0x17 [nfsd] May 31 22:50:38 cog kernel: [<22a1dde6>] fh_verify+0x4a3/0x4bb [nfsd] May 31 22:50:38 cog kernel: [<0214eaba>] vfs_readdir+0x7a/0x9b May 31 22:50:38 cog kernel: [<22a28524>] nfs3svc_encode_entry_plus+0x0/0x17 [nfsd] May 31 22:50:38 cog kernel: [<22a20d1c>] nfsd_readdir+0x59/0xaf [nfsd] May 31 22:50:38 cog kernel: [<22a25940>] nfsd3_proc_readdirplus+0xeb/0x199 [nfsd] May 31 22:50:38 cog kernel: [<22a28524>] nfs3svc_encode_entry_plus+0x0/0x17 [nfsd] May 31 22:50:38 cog kernel: [<22a27764>] nfs3svc_decode_readdirplusargs+0x0/0x154 [nfsd] May 31 22:50:38 cog kernel: [<22a1c54e>] nfsd_dispatch+0xbf/0x165 [nfsd] May 31 22:50:38 cog kernel: [<2297fc24>] svc_process+0x323/0x55f [sunrpc] May 31 22:50:38 cog kernel: [<22a1c355>] nfsd+0x18f/0x2c9 [nfsd] May 31 22:50:38 cog kernel: [<22a1c1c6>] nfsd+0x0/0x2c9 [nfsd] May 31 22:50:38 cog kernel: [<021041d9>] kernel_thread_helper+0x5/0xb May 31 22:50:38 cog kernel: May 31 22:50:38 cog kernel: Code: 89 50 04 89 02 83 79 14 ff c7 01 00 01 10 00 c7 41 04 00 02 May 31 22:50:40 cog kernel: <1>Unable to handle kernel paging request at virtual address 88130004 May 31 22:50:40 cog kernel: printing eip: May 31 22:50:40 cog kernel: 0213130d May 31 22:50:40 cog kernel: *pde = 00000000 May 31 22:50:40 cog kernel: Oops: 0002 [#2] May 31 22:50:40 cog kernel: CPU: 0 May 31 22:50:40 cog kernel: EIP: 0060:[<0213130d>] Not tainted May 31 22:50:40 cog kernel: EFLAGS: 00010012 (2.6.5-1.358) May 31 22:50:40 cog kernel: EIP is at free_block+0x3a/0xb8 May 31 22:50:40 cog kernel: eax: 88130000 ebx: 05c0c000 ecx: 05c0c680 edx: 21d3bc0c May 31 22:50:40 cog kernel: esi: 21d3bc00 edi: 0000001b ebp: 0000000a esp: 03500f1c May 31 22:50:40 cog kernel: ds: 007b es: 007b ss: 0068 May 31 22:50:40 cog kernel: Process pdflush (pid: 8, threadinfo=03500000 task=21de2bb0) May 31 22:50:40 cog kernel: Stack: 21d81364 0000001b 21d3bc00 05ca0d80 21d288dc 021313e5 21d81354 21d81354 May 31 22:50:40 cog kernel: 21d81364 05ca0d80 00000202 0213154a 05ca0e2c 03500f84 0000002b 00000000 May 31 22:50:40 cog kernel: 02153768 05ca0e2c 021539ae 0566f52c 0566f534 022caf24 02153ccd 00000098 May 31 22:50:40 cog kernel: Call Trace: May 31 22:50:40 cog kernel: [<021313e5>] cache_flusharray+0x5a/0x9a May 31 22:50:40 cog kernel: [<0213154a>] kmem_cache_free+0x21/0x2f May 31 22:50:40 cog kernel: [<02153768>] destroy_inode+0x36/0x45 May 31 22:50:40 cog kernel: [<021539ae>] dispose_list+0x4e/0x5e May 31 22:50:40 cog kernel: [<02153ccd>] prune_icache+0x164/0x1a1 May 31 22:50:40 cog kernel: [<0212ff39>] pdflush+0x0/0x1e May 31 22:50:40 cog kernel: [<0212feb7>] __pdflush+0xc3/0x145 May 31 22:50:40 cog kernel: [<0212ff53>] pdflush+0x1a/0x1e May 31 22:50:40 cog kernel: [<0212f89c>] wb_kupdate+0x0/0xf4 May 31 22:50:40 cog kernel: [<0212ff39>] pdflush+0x0/0x1e May 31 22:50:40 cog kernel: [<02125265>] kthread+0x69/0x91 May 31 22:50:40 cog kernel: [<021251fc>] kthread+0x0/0x91 May 31 22:50:40 cog kernel: [<021041d9>] kernel_thread_helper+0x5/0xb May 31 22:50:40 cog kernel: May 31 22:50:40 cog kernel: Code: 89 50 04 89 02 31 d2 2b 4b 0c c7 03 00 01 10 00 c7 43 04 00
