I've had a FC4 install running fine for about 4 months. About 7 days ago there was a power outage and the box was rebooted. Ever since then every day at 1:07 am I get an oops email as follows. Because of the timing of this oops I suspect it to be some cron job that runs at the exact same time every day. In the cron log I see this every day at the exact same time and at no other time than this: May 31 01:07:09 vhost1 crontab[12219]: (root) LIST (root) May 31 01:07:09 vhost1 /usr/bin/crontab[12231]: (root) LIST (root) May 31 01:07:09 vhost1 crontab[12232]: (root) LIST (root) May 31 01:07:09 vhost1 crontab[12233]: (root) REPLACE (root) The only other thing that I notice out of what is the messages log. All the lines are intact and fine. They aren't garbled, but they just aren't logged in the right order. For example, about 8PM the logs stop and jump to some time after 1am the next morning and then you start seeing 8PM lines sprinkled in here and there with the 1am log entries. They progress like this until you see a large chunk of 11PM lines, and then they are sprinkled in until about 4am on it looks normal. None of the lines are chopped off or mixed in the middle of a line, it's as if syslog just queued up about 5 hours of log entries and decided to throw them in later, intermixing them with current log entries. I have a ssh session to the box all the time (it's remote) and the ssh never gets dropped through this oops. I've started top tonight and am having my terminal monitor for silence. If the top freezes I'll be able to see it in the morning when I come back to my local computer. Thanks in advance for any and all help. All of the oops emails look very similar to this: IMPORTANT: Do not ignore this email. Your kernel had an Oops! This is the result of bad hardware or a kernel bug. Your system may continue to function as normal, however there is a good chance bad things are happening right now. Bad things include: files disappearing, daemons crashing, complete server crashs, disk corruption and many others. You might want to check your RAM with memtest86 as this is usually the cause of the problem. http://www.memtest86.com/ The Oops is below: EIP: 0060:[<c0110a1f>] CPU: 0 EIP is at delay_tsc+0xb/0x13 EFLAGS: 00000297 Not tainted (2.6.11-1.1369_FC4smp) EAX: e39297b4 EBX: 00284015 ECX: e36ef174 EDX: 00000006 ESI: c048b7e0 EDI: 000088b8 EBP: c048b7e0 DS: 007b ES: 007b CR0: 8005003b CR2: 0809c880 CR3: 02f46c80 CR4: 000006f0 [<c01d6801>] __delay+0x9/0xa [<c026372b>] ide_wait_not_busy+0x3c/0xb1 [<c0110a1d>] delay_tsc+0x9/0x13 [<c0265114>] wait_hwif_ready+0xd7/0x1f1 [<c0265307>] probe_hwif+0x51/0x49c [<c0260011>] ide_register_subdriver+0xad/0xbf [<c0267934>] idedefault_attach+0x10/0x4c [<c025f885>] ata_attach+0xeb/0x177 [<c0265760>] probe_hwif_init_with_fixup+0xe/0x75 [<c0268c06>] ide_setup_pci_device+0x6d/0x7a [<c03ec4c1>] ide_scan_pcidev+0x31/0x57 [<c03ec504>] ide_scan_pcibus+0x1d/0x97 [<c03ebd80>] probe_for_hwifs+0xa/0x14 [<c03ebdd2>] ide_init+0x48/0x66 [<c03cb97b>] do_initcalls+0x55/0xb1 [<c01687dd>] kern_mount+0x15/0x19 [<c010036c>] init+0x79/0x19d [<c01002f3>] init+0x0/0x19d [<c01023d1>] kernel_thread_helper+0x5/0xb [<c0143897>] softlockup_tick+0x6f/0x80 [<c0108be9>] timer_interrupt+0x2d/0x75 [<c0143a3d>] handle_IRQ_event+0x2e/0x5a [<c0143b36>] __do_IRQ+0xcd/0x137 [<c010663a>] do_IRQ+0x4e/0x86 ======================= [<c01167e6>] smp_apic_timer_interrupt+0xcc/0xce [<c0104a5a>] common_interrupt+0x1a/0x20 [<c0110a1f>] delay_tsc+0xb/0x13 [<c01d6801>] __delay+0x9/0xa [<c026372b>] ide_wait_not_busy+0x3c/0xb1 [<c0110a1d>] delay_tsc+0x9/0x13 [<c0265114>] wait_hwif_ready+0xd7/0x1f1 [<c0265307>] probe_hwif+0x51/0x49c [<c0260011>] ide_register_subdriver+0xad/0xbf [<c0267934>] idedefault_attach+0x10/0x4c [<c025f885>] ata_attach+0xeb/0x177 [<c0265760>] probe_hwif_init_with_fixup+0xe/0x75 [<c0268c06>] ide_setup_pci_device+0x6d/0x7a [<c03ec4c1>] ide_scan_pcidev+0x31/0x57 [<c03ec504>] ide_scan_pcibus+0x1d/0x97 [<c03ebd80>] probe_for_hwifs+0xa/0x14 [<c03ebdd2>] ide_init+0x48/0x66 [<c03cb97b>] do_initcalls+0x55/0xb1 [<c01687dd>] kern_mount+0x15/0x19 [<c010036c>] init+0x79/0x19d [<c01002f3>] init+0x0/0x19d [<c01023d1>] kernel_thread_helper+0x5/0xb hdc: no response (status = 0x80), resetting drive hdc: CDU5211, ATAPI CD/DVD-ROM drive hdd: WDC WD1600JB-40GVC0, ATA DISK drive ide1 at 0x170-0x177,0x376 on irq 15 hda: max request size: 128KiB hda: 156301488 sectors (80026 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(33) hda: cache flushes supported hda: hda1 hda2 hdb: max request size: 128KiB hdb: Host Protected Area detected. current capacity is 156250000 sectors (80000 MB) native capacity is 156301488 sectors (80026 MB) hdb: Host Protected Area disabled. hdb: 156301488 sectors (80026 MB) w/2048KiB Cache, CHS=65535/16/63, UDMA(33) hdb: cache flushes not supported hdb: hdb1 hdb2 hdd: max request size: 1024KiB hdd: 312581808 sectors (160041 MB) w/8192KiB Cache, CHS=19457/255/63, UDMA(100) hdd: cache flushes supported hdd: hdd1 hdc: ATAPI 52X CD-ROM drive, 120kB Cache, UDMA(33) Uniform CD-ROM driver Revision: 3.20 ide-floppy driver 0.99.newide usbcore: registered new driver hiddev usbcore: registered new driver usbhid drivers/usb/input/hid-core.c: v2.01:USB HID core driver mice: PS/2 mouse device common for all mice md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27 NET: Registered protocol family 2 IP: routing cache hash table of 16384 buckets, 256Kbytes TCP established hash table entries: 262144 (order: 10, 4194304 bytes) TCP bind hash table entries: 65536 (order: 7, 786432 bytes) TCP: Hash tables configured (established 262144 bind 65536) Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 17 CPU0 attaching sched-domain: domain 0: span 00000001 groups: 00000001 domain 1: span 00000003 groups: 00000001 00000002 CPU1 attaching sched-domain: domain 0: span 00000002 groups: 00000002 domain 1: span 00000003 groups: 00000002 00000001 Freeing unused kernel memory: 232k freed input: AT Translated Set 2 keyboard on isa0060/serio0 SCSI subsystem initialized libata version 1.10 loaded. sata_promise version 1.01 sata_promise PATA port found ata1: SATA max UDMA/133 cmd 0xF8812200 ctl 0xF8812238 bmdma 0x0 irq 193 ata2: SATA max UDMA/133 cmd 0xF8812280 ctl 0xF88122B8 bmdma 0x0 irq 193 ata3: PATA max UDMA/133 cmd 0xF8812300 ctl 0xF8812338 bmdma 0x0 irq 193 ata1: no device found (phy stat 00000000) scsi0 : sata_promise ata2: no device found (phy stat 00000000) scsi1 : sata_promise ATA: abnormal status 0x8 on port 0xF881231C ata3: disabling port scsi2 : sata_promise EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. logips2pp: Detected unknown logitech mouse model 0 input: ImPS/2 Logitech Wheel Mouse on isa0060/serio1 kjournald starting. Commit interval 5 seconds EXT3-fs: hda2: orphan cleanup on readonly fs ext3_orphan_cleanup: deleting unreferenced inode 9635498 ext3_orphan_cleanup: deleting unreferenced inode 9569113 ext3_orphan_cleanup: deleting unreferenced inode 9247127 ext3_orphan_cleanup: deleting unreferenced inode 9243749 ext3_orphan_cleanup: deleting unreferenced inode 9249666 ext3_orphan_cleanup: deleting unreferenced inode 9247690 ext3_orphan_cleanup: deleting unreferenced inode 9247199 ext3_orphan_cleanup: deleting unreferenced inode 9243747 ext3_orphan_cleanup: deleting unreferenced inode 9252041 ext3_orphan_cleanup: deleting unreferenced inode 9250636 ext3_orphan_cleanup: deleting unreferenced inode 9252051 ext3_orphan_cleanup: deleting unreferenced inode 9244927 ext3_orphan_cleanup: deleting unreferenced inode 9244543 ext3_orphan_cleanup: deleting unreferenced inode 9247928 psmouse.c: Failed to enable mouse on isa0060/serio1 ext3_orphan_cleanup: deleting unreferenced inode 11042885 ext3_orphan_cleanup: deleting unreferenced inode 9243634 ext3_orphan_cleanup: deleting unreferenced inode 9243600 ext3_orphan_cleanup: deleting unreferenced inode 9251175 ext3_orphan_cleanup: deleting unreferenced inode 11042851 ext3_orphan_cleanup: deleting unreferenced inode 11042867 ext3_orphan_cleanup: deleting unreferenced inode 11042850 ext3_orphan_cleanup: deleting unreferenced inode 9252045 ext3_orphan_cleanup: deleting unreferenced inode 9252043 ext3_orphan_cleanup: deleting unreferenced inode 9245029 ext3_orphan_cleanup: deleting unreferenced inode 9252040 ext3_orphan_cleanup: deleting unreferenced inode 11042845 ext3_orphan_cleanup: deleting unreferenced inode 11042945 ext3_orphan_cleanup: deleting unreferenced inode 9251177 ext3_orphan_cleanup: deleting unreferenced inode 9252036 ext3_orphan_cleanup: deleting unreferenced inode 11043896 ext3_orphan_cleanup: deleting unreferenced inode 11043895 ext3_orphan_cleanup: deleting unreferenced inode 9252035 ext3_orphan_cleanup: deleting unreferenced inode 9252034 ext3_orphan_cleanup: deleting unreferenced inode 11042882 ext3_orphan_cleanup: deleting unreferenced inode 9252033 ext3_orphan_cleanup: deleting unreferenced inode 9243630 ext3_orphan_cleanup: deleting unreferenced inode 1310795 ext3_orphan_cleanup: deleting unreferenced inode 11042992 ext3_orphan_cleanup: deleting unreferenced inode 9246886 ext3_orphan_cleanup: deleting unreferenced inode 11042843 ext3_orphan_cleanup: deleting unreferenced inode 11042935 ext3_orphan_cleanup: deleting unreferenced inode 11042841 ext3_orphan_cleanup: deleting unreferenced inode 11042832 ext3_orphan_cleanup: deleting unreferenced inode 11042879 ext3_orphan_cleanup: deleting unreferenced inode 11042881 ext3_orphan_cleanup: deleting unreferenced inode 11043898 ext3_orphan_cleanup: deleting unreferenced inode 11042950 ext3_orphan_cleanup: deleting unreferenced inode 11043891 ext3_orphan_cleanup: deleting unreferenced inode 11043897 ext3_orphan_cleanup: deleting unreferenced inode 11042932 ext3_orphan_cleanup: deleting unreferenced inode 11043900 ext3_orphan_cleanup: deleting unreferenced inode 11042926 ext3_orphan_cleanup: deleting unreferenced inode 11043893 ext3_orphan_cleanup: deleting unreferenced inode 11043903 ext3_orphan_cleanup: deleting unreferenced inode 9306415 ext3_orphan_cleanup: deleting unreferenced inode 9249498 ext3_orphan_cleanup: deleting unreferenced inode 9306131 ext3_orphan_cleanup: deleting unreferenced inode 9637260 ext3_orphan_cleanup: deleting unreferenced inode 9570341 ext3_orphan_cleanup: deleting unreferenced inode 11043902 ext3_orphan_cleanup: deleting unreferenced inode 11043874 ext3_orphan_cleanup: deleting unreferenced inode 11042949 ext3_orphan_cleanup: deleting unreferenced inode 9248021 ext3_orphan_cleanup: deleting unreferenced inode 11042947 EXT3-fs: hda2: 64 orphan inodes deleted EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. security: 3 users, 6 roles, 890 types, 110 bools security: 55 classes, 237133 rules SELinux: Completing initialization. SELinux: Setting up existing superblocks. SELinux: initialized (dev hda2, type ext3), uses xattr SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses genfs_contexts SELinux: initialized (dev devpts, type devpts), uses transition SIDs SELinux: initialized (dev eventpollfs, type eventpollfs), uses genfs_contexts SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts SELinux: initialized (dev pipefs, type pipefs), uses task SIDs SELinux: initialized (dev sockfs, type sockfs), uses task SIDs SELinux: initialized (dev cpuset, type cpuset), not configured for labeling SELinux: initialized (dev proc, type proc), uses genfs_contexts SELinux: initialized (dev bdev, type bdev), uses genfs_contexts SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts Floppy drive(s): fd0 is 1.44M FDC 0 is a post-1991 82077 natsemi dp8381x driver, version 1.07+LK1.0.17, Sep 27, 2002 originally by Donald Becker <becker@xxxxxxxxx> http://www.scyld.com/network/natsemi.html 2.4.x kernel port by Jeff Garzik, Tjeerd Mulder natsemi eth0: NatSemi DP8381[56] at 0xfb025000 (0000:03:0d.0), 00:09:5b:61:e4:f5, IRQ 185, port TP. Intel(R) PRO/1000 Network Driver - version 6.0.54-k2-NAPI Copyright (c) 1999-2004 Intel Corporation. PCI: Setting latency timer of device 0000:02:01.0 to 64 e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection PCI: Setting latency timer of device 0000:00:1f.5 to 64 intel8x0_measure_ac97_clock: measured 49374 usecs intel8x0: clocking to 48000 hw_random hardware driver 1.0.0 loaded shpchp: acpi_shpchprm:get_device PCI ROOT HID fail=0x1001 shpchp: acpi_shpchprm:get_device PCI ROOT HID fail=0x1001 PCI: Setting latency timer of device 0000:00:1d.7 to 64 ehci_hcd 0000:00:1d.7: EHCI Host Controller ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1 ehci_hcd 0000:00:1d.7: irq 193, io mem 0xfc100000 PCI: cache line size of 128 is not supported by device 0000:00:1d.7 ehci_hcd 0000:00:1d.7: USB 2.0 initialized, EHCI 1.00, driver 10 Dec 2004 hub 1-0:1.0: USB hub found hub 1-0:1.0: 8 ports detected USB Universal Host Controller Interface driver v2.2 SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts PCI: Setting latency timer of device 0000:00:1d.0 to 64 uhci_hcd 0000:00:1d.0: UHCI Host Controller uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 uhci_hcd 0000:00:1d.0: irq 145, io base 0x0000cc00 hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected PCI: Setting latency timer of device 0000:00:1d.1 to 64 uhci_hcd 0000:00:1d.1: UHCI Host Controller uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3 uhci_hcd 0000:00:1d.1: irq 169, io base 0x0000c000 hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected PCI: Setting latency timer of device 0000:00:1d.2 to 64 uhci_hcd 0000:00:1d.2: UHCI Host Controller uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4 uhci_hcd 0000:00:1d.2: irq 161, io base 0x0000c400 hub 4-0:1.0: USB hub found hub 4-0:1.0: 2 ports detected PCI: Setting latency timer of device 0000:00:1d.3 to 64 uhci_hcd 0000:00:1d.3: UHCI Host Controller uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5 uhci_hcd 0000:00:1d.3: irq 145, io base 0x0000c800 hub 5-0:1.0: USB hub found hub 5-0:1.0: 2 ports detected ieee1394: Initialized config rom entry `ip1394' ohci1394: $Rev: 1250 $ Ben Collins <bcollins@xxxxxxxxxx> ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[177] MMIO=[fb024000-fb0247ff] Max Packet=[2048] md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. ieee1394: Host added: ID:BUS[0-00:1023] GUID[00e01800007c744e] device-mapper: 4.4.0-ioctl (2005-01-12) initialised: dm-devel@xxxxxxxxxx eth0: DSPCFG accepted after 0 usec. eth0: link up. eth0: Setting full-duplex based on negotiated link capability. EXT3 FS on hda2, internal journal kjournald starting. Commit interval 5 seconds EXT3 FS on hda1, internal journal EXT3-fs: mounted filesystem with ordered data mode. SELinux: initialized (dev hda1, type ext3), uses xattr SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs kjournald starting. Commit interval 5 seconds EXT3 FS on hdd1, internal journal EXT3-fs: mounted filesystem with ordered data mode. SELinux: initialized (dev hdd1, type ext3), uses xattr kjournald starting. Commit interval 5 seconds EXT3 FS on hdb1, internal journal EXT3-fs: mounted filesystem with ordered data mode. SELinux: initialized (dev hdb1, type ext3), uses xattr Adding 2048276k swap on /dev/hdb2. Priority:-1 extents:1 SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts NET: Registered protocol family 10 Disabled Privacy Extensions on device c037d640(lo) IPv6 over IPv4 tunneling driver SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts Bluetooth: Core ver 2.7 NET: Registered protocol family 31 Bluetooth: HCI device and connection manager initialized Bluetooth: HCI socket layer initialized Bluetooth: L2CAP ver 2.7 Bluetooth: L2CAP socket layer initialized Bluetooth: RFCOMM ver 1.5 Bluetooth: RFCOMM socket layer initialized Bluetooth: RFCOMM TTY layer initialized SELinux: initialized (dev autofs, type autofs), uses genfs_contexts parport0: PC-style at 0x378 [PCSPP,TRISTATE] lp0: using parport0 (polling). lp0: console ready eth0: no IPv6 routers present ip_tables: (C) 2000-2002 Netfilter core team loop: loaded (max 8 devices) kjournald starting. Commit interval 5 seconds EXT3 FS on loop0, internal journal ext3_orphan_cleanup: deleting unreferenced inode 6100 ext3_orphan_cleanup: deleting unreferenced inode 6099 ext3_orphan_cleanup: deleting unreferenced inode 6098 EXT3-fs: loop0: 3 orphan inodes deleted EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. SELinux: initialized (dev loop0, type ext3), uses xattr tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx> ip_conntrack version 2.1 (8192 buckets, 65536 max) - 256 bytes per conntrack Bridge firewalling registered device tap0 entered promiscuous mode br0: port 1(tap0) entering learning state br0: no IPv6 routers present tap0: no IPv6 routers present br0: topology change detected, propagating br0: port 1(tap0) entering forwarding state
