Hello, I think I found a bug in the kernel. I had 2 consecutive block ups. I'm running the system in a production environtment with a web application in my company with lots of users. I first installed FC2T3 to make some test, then I updated to FC2 Final with yum and then applied the security updates available in: [http-download.fedora.redhat.com-pub-fedora-linux-core-updates-2-i386-] name=http-download.fedora.redhat.com-pub-fedora-linux-core-updates-2-i386- baseurl=http://download.fedora.redhat.com/pub/fedora/linux/core/updates/2/i386/ I use Apache 2 (the one that comes with FC2), PHP 4.3.6 (compiled by myself), TUX and Oracle 9i Client. The portion of /var/log/messages is as follows: Jun 2 08:48:52 appweb2-coomeva kernel: ------------[ cut here ]------------ Jun 2 08:48:52 appweb2-coomeva kernel: kernel BUG at mm/page_alloc.c:786! Jun 2 08:48:52 appweb2-coomeva kernel: invalid operand: 0000 [#1] Jun 2 08:48:52 appweb2-coomeva kernel: SMP Jun 2 08:48:52 appweb2-coomeva kernel: CPU: 2 Jun 2 08:48:52 appweb2-coomeva kernel: EIP: 0060:[<02138aec>] Not tainted Jun 2 08:48:52 appweb2-coomeva kernel: EFLAGS: 00010246 (2.6.5-1.358smp) Jun 2 08:48:52 appweb2-coomeva kernel: EIP is at __free_pages+0x21/0x4c Jun 2 08:48:52 appweb2-coomeva kernel: eax: 00000000 ebx: 00000000 ecx: 03e128a0 edx: 03e128a0 Jun 2 08:48:52 appweb2-coomeva kernel: esi: 00000000 edi: 7c7c81f0 ebp: 7c7c802c esp: 7d09befc Jun 2 08:48:52 appweb2-coomeva kernel: ds: 007b es: 007b ss: 0068 Jun 2 08:48:52 appweb2-coomeva kernel: Process tux (pid: 1526, threadinfo=7d09b000 task=7dbbb410) Jun 2 08:48:52 appweb2-coomeva kernel: Stack: 00000000 82947f00 00000000 7c7c802c 00000000 82953380 00000022 8293aff6 Jun 2 08:48:52 appweb2-coomeva kernel: 7c7c802c 7c7c8054 8293bcd9 7d09bf40 82953380 00000000 00000005 fef73690 Jun 2 08:48:52 appweb2-coomeva kernel: 829457fd 00000000 7d09b000 00000000 82947345 0212a0f0 82953380 000041ed Jun 2 08:48:52 appweb2-coomeva kernel: Call Trace: Jun 2 08:48:53 appweb2-coomeva kernel: [<82947f00>] do_send_abuf+0x12c/0x147 [tux] Jun 2 08:48:53 appweb2-coomeva kernel: [<8293aff6>] tux_schedule_atom+0x23/0x34 [tux] Jun 2 08:48:53 appweb2-coomeva kernel: [<8293bcd9>] process_requests+0x90/0xa7 [tux] Jun 2 08:48:53 appweb2-coomeva kernel: [<829457fd>] event_loop+0x75/0x178 [tux] Jun 2 08:48:53 appweb2-coomeva kernel: [<82947345>] __sys_tux+0x343/0x77a [tux] Jun 2 08:48:53 appweb2-coomeva kernel: [<0212a0f0>] in_group_p+0x31/0x58 Jun 2 08:48:53 appweb2-coomeva kernel: [<021616a7>] dput+0x18/0x16c Jun 2 08:48:53 appweb2-coomeva kernel: [<02159672>] path_release+0xa/0x26 Jun 2 08:48:54 appweb2-coomeva kernel: [<0214da62>] sys_chdir+0x57/0x5f Jun 2 08:48:54 appweb2-coomeva kernel: [<0224b124>] sys_tux+0x25/0xb2 Jun 2 08:48:54 appweb2-coomeva kernel: Jun 2 08:48:54 appweb2-coomeva kernel: Code: 0f 0b 12 03 c4 80 2b 02 f0 ff 49 04 0f 94 c0 84 c0 74 16 85 Jun 2 08:48:54 appweb2-coomeva kernel: <5>TUX: thread 0 stopping ... Jun 2 08:50:07 appweb2-coomeva sshd(pam_unix)[1937]: session opened for user nalwalovaton by (uid=500) Jun 2 08:50:11 appweb2-coomeva su(pam_unix)[1965]: session opened for user root by nalwalovaton(uid=500) Jun 2 08:55:51 appweb2-coomeva kernel: Bad page state at free_hot_cold_page (in process 'async IO 1/6', page 03dfa860) Jun 2 08:55:51 appweb2-coomeva kernel: flags:0x20000000 mapping:00000000 mapped:0 count:0 Jun 2 08:55:51 appweb2-coomeva kernel: Backtrace: Jun 2 08:55:51 appweb2-coomeva kernel: Call Trace: Jun 2 08:55:51 appweb2-coomeva kernel: [<02137f0d>] bad_page+0x5c/0x7f Jun 2 08:55:51 appweb2-coomeva kernel: [<02138593>] free_hot_cold_page+0x67/0xe0 Jun 2 08:55:51 appweb2-coomeva kernel: [<82947d46>] get_abuf+0x9a/0x101 [tux] Jun 2 08:55:51 appweb2-coomeva kernel: [<82941008>] http_pre_header+0x32/0x3ec [tux] Jun 2 08:55:51 appweb2-coomeva kernel: [<82940ebf>] http_process_message+0x23a/0x266 [tux] Jun 2 08:55:51 appweb2-coomeva kernel: [<8293aff6>] tux_schedule_atom+0x23/0x34 [tux] Jun 2 08:55:51 appweb2-coomeva kernel: [<8293c039>] cachemiss_thread+0x128/0x1de [tux] Jun 2 08:55:51 appweb2-coomeva kernel: [<0211b419>] default_wake_function+0x0/0xc Jun 2 08:55:51 appweb2-coomeva kernel: [<0211b419>] default_wake_function+0x0/0xc Jun 2 08:55:51 appweb2-coomeva kernel: [<8293bf11>] cachemiss_thread+0x0/0x1de [tux] Jun 2 08:55:52 appweb2-coomeva kernel: [<021041f1>] kernel_thread_helper+0x5/0xb Jun 2 08:55:52 appweb2-coomeva kernel: Jun 2 08:55:52 appweb2-coomeva kernel: Trying to fix it up, but a reboot is needed Jun 2 08:55:52 appweb2-coomeva kernel: Unable to handle kernel paging request at virtual address 00100104 Jun 2 08:55:52 appweb2-coomeva kernel: printing eip: Jun 2 08:55:52 appweb2-coomeva kernel: 0213869f Jun 2 08:55:52 appweb2-coomeva kernel: *pde = 00003001 Jun 2 08:55:52 appweb2-coomeva kernel: Oops: 0002 [#2] Jun 2 08:55:53 appweb2-coomeva kernel: SMP Jun 2 08:55:53 appweb2-coomeva kernel: CPU: 0 Jun 2 08:55:53 appweb2-coomeva kernel: EIP: 0060:[<0213869f>] Not tainted Jun 2 08:55:53 appweb2-coomeva kernel: EFLAGS: 00010002 (2.6.5-1.358smp) Jun 2 08:55:53 appweb2-coomeva kernel: EIP is at buffered_rmqueue+0x82/0x1a7 Jun 2 08:55:53 appweb2-coomeva kernel: eax: 03dfa878 ebx: 022f3780 ecx: 00200200 edx: 00100100 Jun 2 08:55:53 appweb2-coomeva kernel: esi: 022f3b00 edi: 022f3780 ebp: 00000246 esp: 7ac81e30 Jun 2 08:55:54 appweb2-coomeva kernel: ds: 007b es: 007b ss: 0068 Jun 2 08:55:54 appweb2-coomeva kernel: Process httpd (pid: 2060, threadinfo=7ac81000 task=71b557b0) Jun 2 08:55:54 appweb2-coomeva kernel: Stack: 03dfa860 000000d2 00000000 022f3780 00000000 022f5fa0 000000d2 02138861 Jun 2 08:55:55 appweb2-coomeva kernel: 00000001 71b557b0 00000010 00000000 00000000 0038a025 00000000 7697c460 Jun 2 08:55:55 appweb2-coomeva kernel: 0988c574 02141c86 00000000 00000000 732e56c4 7b709080 00000000 7697c460 Jun 2 08:55:56 appweb2-coomeva kernel: Call Trace: Jun 2 08:55:56 appweb2-coomeva kernel: [<02138861>] __alloc_pages+0x9d/0x284 Jun 2 08:55:56 appweb2-coomeva kernel: [<02141c86>] do_anonymous_page+0x6d/0x192 Jun 2 08:55:57 appweb2-coomeva kernel: [<02141e14>] do_no_page+0x69/0x2b0 Jun 2 08:55:58 appweb2-coomeva kernel: [<021421ca>] handle_mm_fault+0xc0/0x16e Jun 2 08:55:58 appweb2-coomeva kernel: [<02118c2b>] do_page_fault+0x133/0x4b4 Jun 2 08:55:58 appweb2-coomeva kernel: [<8289eafd>] ext3_permission+0x0/0x153 [ext3] Jun 2 08:55:59 appweb2-coomeva kernel: [<021595ff>] permission+0x41/0x46 Jun 2 08:56:00 appweb2-coomeva kernel: [<02143043>] vma_merge+0x85/0x237 Jun 2 08:56:00 appweb2-coomeva kernel: [<02144121>] do_brk+0x127/0x1d5 Jun 2 08:56:00 appweb2-coomeva kernel: [<02118af8>] do_page_fault+0x0/0x4b4 Jun 2 08:56:00 appweb2-coomeva kernel: Jun 2 08:56:00 appweb2-coomeva kernel: Code: 89 4a 04 89 11 c7 40 04 00 02 20 00 c7 00 00 01 10 00 ff 8b Jun 2 09:00:02 appweb2-coomeva syslogd 1.4.1: restart. Jun 2 09:00:02 appweb2-coomeva syslog: Iniciación de syslogd succeeded Jun 2 09:00:03 appweb2-coomeva syslog: Iniciación de klogd succeeded Jun 2 09:00:03 appweb2-coomeva kernel: klogd 1.4.1, log source = /proc/kmsg started. Jun 2 09:00:03 appweb2-coomeva kernel: Linux version 2.6.5-1.358smp (bhcompile@xxxxxxxxxxxxxxxxxxxxx) (gcc version 3.3.3 20040412 (Red Hat Linux 3.3.3-7)) #1 SMP Sat May 8 09:25:36 EDT 2004 Previously, I had another Lock up and I guess it is for the same reason. This is the log: Jun 1 16:54:59 appweb2-coomeva kernel: ------------[ cut here ]------------ Jun 1 16:54:59 appweb2-coomeva kernel: kernel BUG at mm/page_alloc.c:786! Jun 1 16:54:59 appweb2-coomeva kernel: invalid operand: 0000 [#1] Jun 1 16:54:59 appweb2-coomeva kernel: SMP Jun 1 16:54:59 appweb2-coomeva kernel: CPU: 0 Jun 1 16:54:59 appweb2-coomeva kernel: EIP: 0060:[<02138aec>] Not tainted Jun 1 16:54:59 appweb2-coomeva kernel: EFLAGS: 00010246 (2.6.5-1.358smp) Jun 1 16:54:59 appweb2-coomeva kernel: EIP is at __free_pages+0x21/0x4c Jun 1 16:54:59 appweb2-coomeva kernel: eax: 00000000 ebx: 00000000 ecx: 03d89180 edx: 03d89180 Jun 1 16:54:59 appweb2-coomeva kernel: esi: 00000fd9 edi: 7708597c ebp: 82953380 esp: 7d9b6f40 Jun 1 16:54:59 appweb2-coomeva kernel: ds: 007b es: 007b ss: 0068 Jun 1 16:54:59 appweb2-coomeva kernel: Process async IO 0/1 (pid: 2182, threadinfo=7d9b6000 task=7c7b8750) Jun 1 16:54:59 appweb2-coomeva kernel: Stack: 039dcf80 82947d46 7708597c 00000001 00000000 7708597c 82941008 00000000 Jun 1 16:55:00 appweb2-coomeva kernel: 797e8054 00000000 00000000 7708597c 00000001 00000000 7d9b6000 82940ebf Jun 1 16:55:00 appweb2-coomeva kernel: 7708597c 00000001 80d769cc 8293aff6 80d769c4 7708597c 8293c039 00000001 Jun 1 16:55:00 appweb2-coomeva kernel: Call Trace: Jun 1 16:55:00 appweb2-coomeva kernel: [<82947d46>] get_abuf+0x9a/0x101 [tux] Jun 1 16:55:00 appweb2-coomeva kernel: [<82941008>] http_pre_header+0x32/0x3ec [tux] Jun 1 16:55:00 appweb2-coomeva kernel: [<82940ebf>] http_process_message+0x23a/0x266 [tux] Jun 1 16:55:00 appweb2-coomeva kernel: [<8293aff6>] tux_schedule_atom+0x23/0x34 [tux] Jun 1 16:55:00 appweb2-coomeva kernel: [<8293c039>] cachemiss_thread+0x128/0x1de [tux] Jun 1 16:55:00 appweb2-coomeva kernel: [<0211b419>] default_wake_function+0x0/0xc Jun 1 16:55:00 appweb2-coomeva kernel: [<0211b419>] default_wake_function+0x0/0xc Jun 1 16:55:00 appweb2-coomeva kernel: [<8293bf11>] cachemiss_thread+0x0/0x1de [tux] Jun 1 16:55:00 appweb2-coomeva kernel: [<021041f1>] kernel_thread_helper+0x5/0xb Jun 1 16:55:01 appweb2-coomeva kernel: Jun 1 16:55:01 appweb2-coomeva kernel: Code: 0f 0b 12 03 c4 80 2b 02 f0 ff 49 04 0f 94 c0 84 c0 74 16 85 Jun 1 17:04:55 appweb2-coomeva su(pam_unix)[7005]: session closed for user root Jun 1 17:04:57 appweb2-coomeva login(pam_unix)[6978]: session closed for user nalwalovaton Jun 2 08:38:42 appweb2-coomeva kernel: <0>Bad page state at free_hot_cold_page (in process 'async IO 0/2', page 0348b2a0) Jun 2 08:38:42 appweb2-coomeva kernel: flags:0x20000000 mapping:00000000 mapped:0 count:0 Jun 2 08:38:42 appweb2-coomeva kernel: Backtrace: Jun 2 08:38:42 appweb2-coomeva kernel: Call Trace: Jun 2 08:38:42 appweb2-coomeva kernel: [<02137f0d>] bad_page+0x5c/0x7f Jun 2 08:38:42 appweb2-coomeva kernel: [<02138593>] free_hot_cold_page+0x67/0xe0 Jun 2 08:38:42 appweb2-coomeva kernel: [<82947d46>] get_abuf+0x9a/0x101 [tux] Jun 2 08:38:42 appweb2-coomeva kernel: [<82941008>] http_pre_header+0x32/0x3ec [tux] Jun 2 08:38:42 appweb2-coomeva kernel: [<82940ebf>] http_process_message+0x23a/0x266 [tux] Jun 2 08:38:42 appweb2-coomeva kernel: [<8293aff6>] tux_schedule_atom+0x23/0x34 [tux] Jun 2 08:38:42 appweb2-coomeva kernel: [<8293c039>] cachemiss_thread+0x128/0x1de [tux] Jun 2 08:38:42 appweb2-coomeva kernel: [<0211b419>] default_wake_function+0x0/0xc Jun 2 08:38:42 appweb2-coomeva kernel: [<0211b419>] default_wake_function+0x0/0xc Jun 2 08:38:43 appweb2-coomeva kernel: [<8293bf11>] cachemiss_thread+0x0/0x1de [tux] Jun 2 08:38:43 appweb2-coomeva kernel: [<021041f1>] kernel_thread_helper+0x5/0xb Jun 2 08:38:43 appweb2-coomeva kernel: Jun 2 08:38:43 appweb2-coomeva kernel: Trying to fix it up, but a reboot is needed Jun 2 08:38:43 appweb2-coomeva kernel: Unable to handle kernel paging request at virtual address 00100104 Jun 2 08:38:44 appweb2-coomeva kernel: printing eip: Jun 2 08:38:44 appweb2-coomeva kernel: 0213869f Jun 2 08:38:44 appweb2-coomeva kernel: *pde = 00003001 Jun 2 08:38:44 appweb2-coomeva kernel: Oops: 0002 [#2] Jun 2 08:38:44 appweb2-coomeva kernel: SMP Jun 2 08:38:44 appweb2-coomeva kernel: CPU: 2 Jun 2 08:38:44 appweb2-coomeva kernel: EIP: 0060:[<0213869f>] Not tainted Jun 2 08:38:44 appweb2-coomeva kernel: EFLAGS: 00010002 (2.6.5-1.358smp) Jun 2 08:38:44 appweb2-coomeva kernel: EIP is at buffered_rmqueue+0x82/0x1a7 Jun 2 08:38:44 appweb2-coomeva kernel: eax: 0348b2b8 ebx: 022f3880 ecx: 022f3c10 edx: 00100100 Jun 2 08:38:44 appweb2-coomeva kernel: esi: 022f3c00 edi: 022f3780 ebp: 00000206 esp: 0ff05e30 Jun 2 08:38:44 appweb2-coomeva kernel: ds: 007b es: 007b ss: 0068 Jun 2 08:38:44 appweb2-coomeva kernel: Process httpd (pid: 7951, threadinfo=0ff05000 task=6b6c80b0) Jun 2 08:38:44 appweb2-coomeva kernel: Stack: 0348b2a0 000000d2 00000000 022f3780 00000000 022f5fa0 000000d2 02138861 Jun 2 08:38:44 appweb2-coomeva kernel: 00000001 6b6c80b0 00000010 00000000 00000000 0038a025 00000000 07220720 Jun 2 08:38:44 appweb2-coomeva kernel: ebae4004 02141c86 00000000 00000000 249c4bc4 80b1f080 00000000 07220720 Jun 2 08:38:45 appweb2-coomeva kernel: Call Trace: Jun 2 08:38:45 appweb2-coomeva kernel: [<02138861>] __alloc_pages+0x9d/0x284 Jun 2 08:38:45 appweb2-coomeva kernel: [<02141c86>] do_anonymous_page+0x6d/0x192 Jun 2 08:38:45 appweb2-coomeva kernel: [<02141e14>] do_no_page+0x69/0x2b0 Jun 2 08:38:45 appweb2-coomeva kernel: [<02140c53>] follow_page+0x128/0x134 Jun 2 08:38:45 appweb2-coomeva kernel: [<021421ca>] handle_mm_fault+0xc0/0x16e Jun 2 08:38:45 appweb2-coomeva kernel: [<02118c2b>] do_page_fault+0x133/0x4b4 Jun 2 08:38:45 appweb2-coomeva kernel: [<02143576>] do_mmap_pgoff+0x381/0x60c Jun 2 08:38:45 appweb2-coomeva kernel: [<0210b41f>] sys_mmap2+0x77/0xab Jun 2 08:38:45 appweb2-coomeva kernel: [<02118af8>] do_page_fault+0x0/0x4b4 Jun 2 08:38:45 appweb2-coomeva kernel: Jun 2 08:38:45 appweb2-coomeva kernel: Code: 89 4a 04 89 11 c7 40 04 00 02 20 00 c7 00 00 01 10 00 ff 8b Jun 2 08:38:53 appweb2-coomeva kernel: <1>Unable to handle kernel paging request at virtual address 00100104 Jun 2 08:38:53 appweb2-coomeva kernel: printing eip: Jun 2 08:38:53 appweb2-coomeva kernel: 021382ee Jun 2 08:38:53 appweb2-coomeva kernel: *pde = 00003001 Jun 2 08:47:45 appweb2-coomeva syslogd 1.4.1: restart. And 5 days ago I had the same problem... this time while I was executing gkrellm in a remote desktop. This is the log: May 28 07:43:59 appweb2-coomeva kernel: Unable to handle kernel paging request at virtual address 00100104 May 28 07:43:59 appweb2-coomeva kernel: printing eip: May 28 07:43:59 appweb2-coomeva kernel: 0213869f May 28 07:43:59 appweb2-coomeva kernel: *pde = 00003001 May 28 07:43:59 appweb2-coomeva kernel: Oops: 0002 [#1] May 28 07:43:59 appweb2-coomeva kernel: SMP May 28 07:43:59 appweb2-coomeva kernel: CPU: 0 May 28 07:43:59 appweb2-coomeva kernel: EIP: 0060:[<0213869f>] Not tainted May 28 07:43:59 appweb2-coomeva kernel: EFLAGS: 00010002 (2.6.5-1.358smp) May 28 07:43:59 appweb2-coomeva kernel: EIP is at buffered_rmqueue+0x82/0x1a7 May 28 07:43:59 appweb2-coomeva kernel: eax: 03c900b8 ebx: 022f3780 ecx: 022f3b10 edx: 00100100 May 28 07:43:59 appweb2-coomeva kernel: esi: 022f3b00 edi: 022f3780 ebp: 00000246 esp: 6b09fe1c May 28 07:43:59 appweb2-coomeva kernel: ds: 007b es: 007b ss: 0068 May 28 07:44:00 appweb2-coomeva kernel: Process gkrellm (pid: 2078, threadinfo=6b09f000 task=691d4bf0) May 28 07:44:00 appweb2-coomeva kernel: Stack: 03c900a0 000000d2 00000000 022f3780 00000000 022f5fa0 000000d2 02138861 May 28 07:44:00 appweb2-coomeva kernel: 00000001 691d4bf0 00000010 00000000 6b09feb3 0038a025 00000000 66222490 May 28 07:44:00 appweb2-coomeva kernel: f3e92000 02141c86 00000000 00000000 6f3f1d04 68b5a800 00000000 66222490 May 28 07:44:00 appweb2-coomeva kernel: Call Trace: May 28 07:44:00 appweb2-coomeva kernel: [<02138861>] __alloc_pages+0x9d/0x284 May 28 07:48:50 appweb2-coomeva syslogd 1.4.1: restart. Is this bug new? Should I file it in bugzilla?? Any help about this will be highly appreaciated. If you need more info don't hesitate to ask me. Thanks, -William
