Problem with 2.6 kernel and lots of I/O

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I have a (well, at least one) show-stopping problems with the 2.6 kernel while doing heavy I/O. I have a (software) RAID1 of network block devices (nbd0 and nbd1) set up on two identical machines in an active-passive HA cluster configuration. When the "primary" node goes down and comes back up it recovers the RAID as follows:

	Start RAID in degraded mode with remote device (nbd1)
	Hot-add local device (nbd0)

This all works. Hot-adding the local device causes a resync and that is where the problems begin. Once the resync begins the system becomes unusable. Anything that wants to write something to the syslog socket ("/dev/log") syncronously hangs until the resync completes. The system load goes up to 18 or so. Writing stuff to the local disk ("/etc" for example, which is not part of the RAID) sometimes hangs. When the resync is complete everything is happy again. Resyncing takes about 25 minutes (20GB over a dedicated network interface to the client at 1000Mbps) and makes the recovery time unacceptable. Also, during this recovery the OOM killer will occasionally be invoked and kill something randomly even though there is typically plenty of unused swap lying around before (though perhaps "java" just decides to eat all of that VERY quickly and I don't notice this, since that is what the OOM killer choses to kill.)

Does anyone have any ideas ?


Information about the systems:

Info: Linux cog1 2.6.9-5.0.5.ELsmp #1 SMP Fri Apr 8 14:29:47 EDT 2005 i686 i686 i386 GNU/Linux
Dist: RedHat Enterprise Linux 4
Spec:
    2 x 3.2GHz Xeon (each system, with hyperthreading so 4 logical processors)
    4GB of physical RAM
    2GB of configured swap (partition, contigious)
    2 x 1000Mbps (Intel 82546GB) network cards (HA cluster link is
              provided by a cross over cable between the two nodes)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Index of Archives]     [Kernel Newbies]     [Netfilter]     [Bugtraq]     [Photo]     [Stuff]     [Gimp]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Video 4 Linux]     [Linux for the blind]     [Linux Resources]
  Powered by Linux