Hi Clive, I saw your email in the Fedora archive and wasn't sure if you had received a reply. I ran into the same problem with my new database server and it would crash anytime under load. I did some searches and found a workaround by executing the following command: echo 0 > /proc/sys/kernel/randomize_va_space This basically turns off the virtual memory space randomizer that is a feature of the 2.6 kernel. The server's been running fine under the same high load conditions for a day now. BTW, the posts about this problem seem to be centered around AMD Opteron cpus. If you're running on Intel, this might not fix it, but it's worth a try. Good luck, Tony On Fri, 8 Jul 2005, Clive at Rational wrote: > Some diagnostic information I have found when Mozilla > fails under Linux Fedora 2.6.10 / 2.6.11 > > I do not have the skills to diagnose anything from > this > > Should I post the info on some other list? > > Thanks > > Clive > > Jul 8 18:00:47 localhost kernel: ------------[ cut > here ]------------ > Jul 8 18:00:47 localhost kernel: kernel BUG at > mm/rmap.c:483! > Jul 8 18:00:47 localhost kernel: invalid operand: > 0000 [#1] > Jul 8 18:00:47 localhost kernel: Modules linked in: > parport_pc lp parport md5 ipv6 autofs4 sunrpc > orinoco_cs orinoco hermes pcmcia microcode vfat fat > dm_mod video button battery ac ohci1394 ieee1394 > yenta_socket pcmcia_core ohci_hcd ehci_hcd i2c_sis96x > i2c_core snd_intel8x0 snd_ac97_codec snd_pcm_oss > snd_mixer_oss snd_pcm snd_timer snd soundcore > snd_page_alloc 8139too mii floppy ext3 jbd > Jul 8 18:00:47 localhost kernel: CPU: 0 > Jul 8 18:00:47 localhost kernel: EIP: > 0060:[<c01560dc>] Not tainted VLI > Jul 8 18:00:47 localhost kernel: EFLAGS: 00210286 > (2.6.10-1.771_FC2) > Jul 8 18:00:47 localhost kernel: EIP is at > page_remove_rmap+0x22/0x36 > Jul 8 18:00:47 localhost kernel: eax: ffffffff ebx: > c1171d40 ecx: c1171d40 edx: c1171d40 > Jul 8 18:00:47 localhost kernel: esi: 00000000 edi: > 000f4000 ebp: cbe71aa0 esp: d2925df0 > Jul 8 18:00:47 localhost kernel: ds: 007b es: 007b > ss: 0068 > Jul 8 18:00:47 localhost kernel: Process mozilla-bin > (pid: 5822, threadinfo=d2925000 task=cd991870) > Jul 8 18:00:47 localhost kernel: Stack: c014eb0b > 0b8ea067 0024c000 08db4000 c03e2e28 08db4000 09000000 > cdb64090 > Jul 8 18:00:47 localhost kernel: c03e2e28 > c014eba6 0024c000 00000000 08db4000 cdb64090 091b4000 > c03e2e28 > Jul 8 18:00:47 localhost kernel: c014ebfd > 00400000 00000000 d2925e98 00400000 08db4000 cd43a564 > c014ed09 > Jul 8 18:00:47 localhost kernel: Call Trace: > Jul 8 18:00:47 localhost kernel: [<c014eb0b>] > zap_pte_range+0x1bd/0x221 > Jul 8 18:00:47 localhost kernel: [<c014eba6>] > zap_pmd_range+0x37/0x52 > Jul 8 18:00:47 localhost kernel: [<c014ebfd>] > unmap_page_range+0x3c/0x57 > Jul 8 18:00:47 localhost kernel: [<c014ed09>] > unmap_vmas+0xf1/0x2a0 > Jul 8 18:00:47 localhost kernel: [<c03023d9>] > wait_for_completion+0x12d/0x229 > Jul 8 18:00:47 localhost kernel: [<c0153e97>] > exit_mmap+0xc2/0x1e5 > Jul 8 18:00:47 localhost kernel: [<c0118d40>] > mmput+0x1f/0x16a > Jul 8 18:00:47 localhost kernel: [<c011e68b>] > do_exit+0x22b/0x5bf > Jul 8 18:00:47 localhost kernel: [<c011ebcc>] > sys_exit_group+0x0/0xd > Jul 8 18:00:47 localhost kernel: [<c0128f87>] > get_signal_to_deliver+0x747/0x802 > Jul 8 18:00:47 localhost kernel: [<c01032ce>] > do_signal+0x55/0xce > Jul 8 18:00:47 localhost kernel: [<c0302abf>] > schedule_timeout+0x53/0x10f > Jul 8 18:00:47 localhost kernel: [<c016f756>] > pipe_poll+0x21/0x5e > Jul 8 18:00:47 localhost kernel: [<c01760a6>] > do_pollfd+0x5b/0x83 > Jul 8 18:00:47 localhost kernel: [<c0176117>] > do_poll+0x49/0xa5 > Jul 8 18:00:47 localhost kernel: [<c01756c0>] > poll_freewait+0x33/0x3a > Jul 8 18:00:47 localhost kernel: [<c0176322>] > sys_poll+0x1af/0x1bd > Jul 8 18:00:47 localhost kernel: [<c01756c7>] > __pollwait+0x0/0x94 > Jul 8 18:00:47 localhost kernel: [<c010336e>] > do_notify_resume+0x27/0x35 > Jul 8 18:00:47 localhost kernel: [<c01034be>] > work_notifysig+0x13/0x15 > Jul 8 18:00:47 localhost kernel: Code: ff 05 f0 89 40 > c0 50 9d c3 89 c2 8b 00 f6 c4 08 74 08 0f 0b e0 01 a3 > cb 31 c0 83 42 08 ff 0f 98 c0 84 c0 74 19 8b 42 08 40 > 79 08 <0f> 0b e3 01 a3 cb 31 c0 9c 58 fa ff 0d f0 89 > 40 c0 50 9d c3 55 > Jul 8 18:00:47 localhost kernel: <3>Debug: sleeping > function called from invalid context at > include/linux/rwsem.h:43 > Jul 8 18:00:47 localhost kernel: in_atomic():1, > irqs_disabled():0 > Jul 8 18:00:47 localhost kernel: [<c01188f3>] > __might_sleep+0x80/0x8a > Jul 8 18:00:47 localhost kernel: [<c011c6d4>] > profile_task_exit+0x18/0x41 > Jul 8 18:00:47 localhost kernel: [<c011e477>] > do_exit+0x17/0x5bf > Jul 8 18:00:47 localhost kernel: [<c0103e1b>] > do_trap+0x0/0xa2 > Jul 8 18:00:47 localhost kernel: [<c010400b>] > do_invalid_op+0x0/0x8b > Jul 8 18:00:47 localhost kernel: [<c010408a>] > do_invalid_op+0x7f/0x8b > Jul 8 18:00:47 localhost kernel: [<c02a70a6>] > skb_copy_datagram_iovec+0x4f/0x1e1 > Jul 8 18:00:47 localhost kernel: [<c01560dc>] > page_remove_rmap+0x22/0x36 > Jul 8 18:00:47 localhost kernel: [<c02ccb83>] > tcp_recvmsg+0x60b/0x646 > Jul 8 18:00:47 localhost kernel: [<e09eab03>] > hermes_bap_pwrite+0x34/0xa8 [hermes] > Jul 8 18:00:47 localhost kernel: [<c010361b>] > error_code+0x2b/0x30 > Jul 8 18:00:47 localhost kernel: [<c014007b>] > do_generic_mapping_read+0x1bc/0x357 > Jul 8 18:00:47 localhost kernel: [<c01560dc>] > page_remove_rmap+0x22/0x36 > Jul 8 18:00:47 localhost kernel: [<c014eb0b>] > zap_pte_range+0x1bd/0x221 > Jul 8 18:00:47 localhost kernel: [<c014eba6>] > zap_pmd_range+0x37/0x52 > Jul 8 18:00:47 localhost kernel: [<c014ebfd>] > unmap_page_range+0x3c/0x57 > Jul 8 18:00:47 localhost kernel: [<c014ed09>] > unmap_vmas+0xf1/0x2a0 > Jul 8 18:00:47 localhost kernel: [<c03023d9>] > wait_for_completion+0x12d/0x229 > Jul 8 18:00:47 localhost kernel: [<c0153e97>] > exit_mmap+0xc2/0x1e5 > Jul 8 18:00:47 localhost kernel: [<c0118d40>] > mmput+0x1f/0x16a > Jul 8 18:00:47 localhost kernel: [<c011e68b>] > do_exit+0x22b/0x5bf > Jul 8 18:00:47 localhost kernel: [<c011ebcc>] > sys_exit_group+0x0/0xd > Jul 8 18:00:47 localhost kernel: [<c0128f87>] > get_signal_to_deliver+0x747/0x802 > Jul 8 18:00:47 localhost kernel: [<c01032ce>] > do_signal+0x55/0xce > Jul 8 18:00:47 localhost kernel: [<c0302abf>] > schedule_timeout+0x53/0x10f > Jul 8 18:00:47 localhost kernel: [<c016f756>] > pipe_poll+0x21/0x5e > Jul 8 18:00:47 localhost kernel: [<c01760a6>] > do_pollfd+0x5b/0x83 > Jul 8 18:00:47 localhost kernel: [<c0176117>] > do_poll+0x49/0xa5 > Jul 8 18:00:47 localhost kernel: [<c01756c0>] > poll_freewait+0x33/0x3a > Jul 8 18:00:47 localhost kernel: [<c0176322>] > sys_poll+0x1af/0x1bd > Jul 8 18:00:47 localhost kernel: [<c01756c7>] > __pollwait+0x0/0x94 > Jul 8 18:00:47 localhost kernel: [<c010336e>] > do_notify_resume+0x27/0x35 > Jul 8 18:00:47 localhost kernel: [<c01034be>] > work_notifysig+0x13/0x15 > Jul 8 18:00:47 localhost kernel: note: > mozilla-bin[5822] exited with preempt_count 1 > Jul 8 18:01:01 localhost crond(pam_unix)[6180]: > session opened for user root by (uid=0) > Jul 8 18:01:01 localhost crond(pam_unix)[6180]: > session closed for user root > Jul 8 18:01:03 localhost kernel: ------------[ cut > here ]------------ > Jul 8 18:01:03 localhost kernel: kernel BUG at > mm/rmap.c:483! > Jul 8 18:01:03 localhost kernel: invalid operand: > 0000 [#2] > Jul 8 18:01:03 localhost kernel: Modules linked in: > parport_pc lp parport md5 ipv6 autofs4 sunrpc > orinoco_cs orinoco hermes pcmcia microcode vfat fat > dm_mod video button battery ac ohci1394 ieee1394 > yenta_socket pcmcia_core ohci_hcd ehci_hcd i2c_sis96x > i2c_core snd_intel8x0 snd_ac97_codec snd_pcm_oss > snd_mixer_oss snd_pcm snd_timer snd soundcore > snd_page_alloc 8139too mii floppy ext3 jbd > Jul 8 18:01:03 localhost kernel: CPU: 0 > Jul 8 18:01:03 localhost kernel: EIP: > 0060:[<c01560dc>] Not tainted VLI > Jul 8 18:01:03 localhost kernel: EFLAGS: 00210286 > (2.6.10-1.771_FC2) > Jul 8 18:01:03 localhost kernel: EIP is at > page_remove_rmap+0x22/0x36 > Jul 8 18:01:03 localhost kernel: eax: ffffffff ebx: > c1171d40 ecx: c1171d40 edx: c1171d40 > Jul 8 18:01:03 localhost kernel: esi: 00000000 edi: > 00000000 ebp: d525dc44 esp: d6eefed0 > Jul 8 18:01:03 localhost kernel: ds: 007b es: 007b > ss: 0068 > Jul 8 18:01:03 localhost kernel: Process nautilus > (pid: 5390, threadinfo=d6eef000 task=d918b100) > Jul 8 18:01:03 localhost kernel: Stack: c014eb0b > 0b8ea067 00001000 b7b11000 c03e2e28 b7b11000 b7b12000 > d64efb7c > Jul 8 18:01:03 localhost kernel: c03e2e28 > c014eba6 00001000 00000000 b7b11000 d64efb7c b7b12000 > c03e2e28 > Jul 8 18:01:03 localhost kernel: c014ebfd > 00001000 00000000 d6eeff78 00400000 b7b11000 cb0f675c > c014ed09 > Jul 8 18:01:03 localhost kernel: Call Trace: > Jul 8 18:01:03 localhost kernel: [<c014eb0b>] > zap_pte_range+0x1bd/0x221 > Jul 8 18:01:03 localhost kernel: [<c014eba6>] > zap_pmd_range+0x37/0x52 > Jul 8 18:01:03 localhost kernel: [<c014ebfd>] > unmap_page_range+0x3c/0x57 > Jul 8 18:01:03 localhost kernel: [<c014ed09>] > unmap_vmas+0xf1/0x2a0 > Jul 8 18:01:03 localhost kernel: [<c01520be>] > vma_link+0xf6/0x207 > Jul 8 18:01:03 localhost kernel: [<c0153781>] > unmap_region+0x61/0xc6 > Jul 8 18:01:03 localhost kernel: [<c0153a77>] > do_munmap+0x166/0x1dd > Jul 8 18:01:03 localhost kernel: [<c0153b39>] > sys_munmap+0x4b/0x63 > Jul 8 18:01:03 localhost kernel: [<c0103473>] > syscall_call+0x7/0xb > Jul 8 18:01:03 localhost kernel: Code: ff 05 f0 89 40 > c0 50 9d c3 89 c2 8b 00 f6 c4 08 74 08 0f 0b e0 01 a3 > cb 31 c0 83 42 08 ff 0f 98 c0 84 c0 74 19 8b 42 08 40 > 79 08 <0f> 0b e3 01 a3 cb 31 c0 9c 58 fa ff 0d f0 89 > 40 c0 50 9d c3 55 > Jul 8 18:01:03 localhost kernel: <3>Debug: sleeping > function called from invalid context at > include/linux/rwsem.h:43 > Jul 8 18:01:03 localhost kernel: in_atomic():1, > irqs_disabled():0 > Jul 8 18:01:03 localhost kernel: [<c01188f3>] > __might_sleep+0x80/0x8a > Jul 8 18:01:03 localhost kernel: [<c011c6d4>] > profile_task_exit+0x18/0x41 > Jul 8 18:01:03 localhost kernel: [<c011e477>] > do_exit+0x17/0x5bf > Jul 8 18:01:03 localhost kernel: [<c0103e1b>] > do_trap+0x0/0xa2 > Jul 8 18:01:03 localhost kernel: [<c010400b>] > do_invalid_op+0x0/0x8b > Jul 8 18:01:03 localhost kernel: [<c010408a>] > do_invalid_op+0x7f/0x8b > Jul 8 18:01:03 localhost kernel: [<c01560dc>] > page_remove_rmap+0x22/0x36 > Jul 8 18:01:03 localhost kernel: [<c014020e>] > do_generic_mapping_read+0x34f/0x357 > Jul 8 18:01:03 localhost kernel: [<c0140465>] > __generic_file_aio_read+0x15d/0x177 > Jul 8 18:01:03 localhost kernel: [<c010361b>] > error_code+0x2b/0x30 > Jul 8 18:01:03 localhost kernel: [<c01560dc>] > page_remove_rmap+0x22/0x36 > Jul 8 18:01:03 localhost kernel: [<c014eb0b>] > zap_pte_range+0x1bd/0x221 > Jul 8 18:01:03 localhost kernel: [<c014eba6>] > zap_pmd_range+0x37/0x52 > Jul 8 18:01:03 localhost kernel: [<c014ebfd>] > unmap_page_range+0x3c/0x57 > Jul 8 18:01:03 localhost kernel: [<c014ed09>] > unmap_vmas+0xf1/0x2a0 > Jul 8 18:01:03 localhost kernel: [<c01520be>] > vma_link+0xf6/0x207 > Jul 8 18:01:03 localhost kernel: [<c0153781>] > unmap_region+0x61/0xc6 > Jul 8 18:01:03 localhost kernel: [<c0153a77>] > do_munmap+0x166/0x1dd > Jul 8 18:01:03 localhost kernel: [<c0153b39>] > sys_munmap+0x4b/0x63 > Jul 8 18:01:03 localhost kernel: [<c0103473>] > syscall_call+0x7/0xb > Jul 8 18:01:03 localhost kernel: note: nautilus[5390] > exited with preempt_count 1 > Jul 8 18:01:03 localhost kernel: scheduling while > atomic: nautilus/0x00000001/5390 > Jul 8 18:01:03 localhost kernel: [<c0301dfb>] > schedule+0x3d/0x4ee > Jul 8 18:01:03 localhost kernel: [<c0302f81>] > rwsem_down_read_failed+0x1ff/0x21d > Jul 8 18:01:03 localhost kernel: [<c0134e2f>] > .text.lock.futex+0x7/0xb0 > Jul 8 18:01:03 localhost kernel: [<c0134d17>] > do_futex+0x37/0x5e > Jul 8 18:01:03 localhost kernel: [<c0134e06>] > sys_futex+0xc8/0xd7 > Jul 8 18:01:03 localhost kernel: [<c0119035>] > mm_release+0x7f/0x86 > Jul 8 18:01:03 localhost kernel: [<c011e53c>] > do_exit+0xdc/0x5bf > Jul 8 18:01:03 localhost kernel: [<c0103e1b>] > do_trap+0x0/0xa2 > Jul 8 18:01:03 localhost kernel: [<c010400b>] > do_invalid_op+0x0/0x8b > Jul 8 18:01:03 localhost kernel: [<c010408a>] > do_invalid_op+0x7f/0x8b > Jul 8 18:01:03 localhost kernel: [<c01560dc>] > page_remove_rmap+0x22/0x36 > Jul 8 18:01:03 localhost kernel: [<c014020e>] > do_generic_mapping_read+0x34f/0x357 > Jul 8 18:01:03 localhost kernel: [<c0140465>] > __generic_file_aio_read+0x15d/0x177 > Jul 8 18:01:03 localhost kernel: [<c010361b>] > error_code+0x2b/0x30 > Jul 8 18:01:03 localhost kernel: [<c01560dc>] > page_remove_rmap+0x22/0x36 > Jul 8 18:01:03 localhost kernel: [<c014eb0b>] > zap_pte_range+0x1bd/0x221 > Jul 8 18:01:03 localhost kernel: [<c014eba6>] > zap_pmd_range+0x37/0x52 > Jul 8 18:01:03 localhost kernel: [<c014ebfd>] > unmap_page_range+0x3c/0x57 > Jul 8 18:01:03 localhost kernel: [<c014ed09>] > unmap_vmas+0xf1/0x2a0 > Jul 8 18:01:03 localhost kernel: [<c01520be>] > vma_link+0xf6/0x207 > Jul 8 18:01:03 localhost kernel: [<c0153781>] > unmap_region+0x61/0xc6 > Jul 8 18:01:03 localhost kernel: [<c0153a77>] > do_munmap+0x166/0x1dd > Jul 8 18:01:03 localhost kernel: [<c0153b39>] > sys_munmap+0x4b/0x63 > Jul 8 18:01:03 localhost kernel: [<c0103473>] > syscall_call+0x7/0xb > Jul 8 18:02:44 localhost smartd[4231]: Device: > /dev/hda, 1 Currently unreadable (pending) sectors > Jul 8 18:02:55 localhost shutdown: shutting down for > system reboot > Jul 8 18:02:55 localhost init: Switching to runlevel: > 6 > Jul 8 18:02:56 localhost login(pam_unix)[4829]: > session closed for user root > Jul 8 18:02:57 localhost cups-config-daemon: > cups-config-daemon -TERM succeeded > Jul 8 18:02:57 localhost haldaemon: haldaemon -TERM > succeeded > Jul 8 18:02:57 localhost messagebus: messagebus -TERM > succeeded > Jul 8 18:02:57 localhost atd: atd shutdown succeeded > Jul 8 18:02:57 localhost cups: cupsd shutdown > succeeded > Jul 8 18:02:58 localhost webmin: Stopping Webmin > server in /usr/libexec/webmin > Jul 8 18:02:58 localhost rc: Stopping webmin: > succeeded > Jul 8 18:02:58 localhost xfs[4600]: terminating > Jul 8 18:02:58 localhost xfs: xfs shutdown succeeded > Jul 8 18:02:58 localhost gpm[4509]: *** info > [mice.c(1766)]: > Jul 8 18:02:58 localhost gpm[4509]: imps2: > Auto-detected intellimouse PS/2 > Jul 8 18:02:58 localhost gpm: gpm shutdown succeeded > Jul 8 18:03:00 localhost gconfd (root-5339): SIGHUP > received, reloading all databases > Jul 8 18:03:00 localhost gconfd (root-5339): Resolved > address "xml:readonly:/etc/gconf/gconf.xml.mandatory" > to a read-only configuration source at position 0 > Jul 8 18:03:00 localhost gconfd (root-5339): Resolved > address "xml:readwrite:/root/.gconf" to a writable > configuration source at position 1 > Jul 8 18:03:00 localhost gconfd (root-5339): Resolved > address "xml:readonly:/etc/gconf/gconf.xml.defaults" > to a read-only configuration source at position 2 > Jul 8 18:03:02 localhost httpd: httpd shutdown > succeeded > Jul 8 18:03:03 localhost sshd: sshd -TERM succeeded > Jul 8 18:03:03 localhost sendmail: sendmail shutdown > succeeded > Jul 8 18:03:03 localhost sendmail: sm-client shutdown > succeeded > Jul 8 18:03:04 localhost smb: smbd shutdown succeeded > Jul 8 18:03:04 localhost nmbd[4614]: [2005/07/08 > 18:03:04, 0] nmbd/nmbd.c:terminate(56) > Jul 8 18:03:04 localhost nmbd[4614]: Got SIGTERM: > going down... > Jul 8 18:03:04 localhost smb: nmbd shutdown succeeded > Jul 8 18:03:04 localhost vncserver: Shutting down VNC > server: > Jul 8 18:03:04 localhost vncserver: vncserver > shutdown succeeded > Jul 8 18:03:04 localhost vncserver: [60G > Jul 8 18:03:04 localhost vncserver: > Jul 8 18:03:04 localhost rc: Stopping vncserver: > succeeded > Jul 8 18:03:05 localhost mysqld: Stopping MySQL: > succeeded > Jul 8 18:07:49 localhost shutdown: shutting down for > system reboot > > -- A sadist is a masochist who follows the Golden Rule.
