On Mon, 29 Aug 2005, Vojtech Pavlik wrote:
On Mon, Aug 29, 2005 at 06:20:56PM +0000, Holger Kiehl wrote:
Hello
I have a system with the following setup:
Board is Tyan S4882 with AMD 8131 Chipset
4 Opterons 848 (2.2GHz)
8 GB DDR400 Ram (2GB for each CPU)
1 onboard Symbios Logic 53c1030 dual channel U320 controller
2 SATA disks put together as a SW Raid1 for system, swap and spares
8 SCSI U320 (15000 rpm) disks where 4 disks (sdc, sdd, sde, sdf)
are on one channel and the other four (sdg, sdh, sdi, sdj) on
the other channel.
The U320 SCSI controller has a 64 bit PCI-X bus for itself, there is
no other device on that bus. Unfortunatly I was unable to determine at
what speed it is running, here the output from lspci -vv:
How does one determine the PCI-X bus speed?
Usually only the card (in your case the Symbios SCSI controller) can
tell. If it does, it'll be most likely in 'dmesg'.
There is nothing in dmesg:
Fusion MPT base driver 3.01.20
Copyright (c) 1999-2004 LSI Logic Corporation
ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 24 (level, low) -> IRQ 217
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator,Target}
ACPI: PCI Interrupt 0000:02:04.1[B] -> GSI 25 (level, low) -> IRQ 225
mptbase: Initiating ioc1 bringup
ioc1: 53C1030: Capabilities={Initiator,Target}
Fusion MPT SCSI Host driver 3.01.20
Anyway, I thought with this system I would get theoretically 640 MB/s using
both channels.
You can never use the full theoretical bandwidth of the channel for
data. A lot of overhead remains for other signalling. Similarly for PCI.
I tested several software raid setups to get the best possible write
speeds for this system. But testing shows that the absolute maximum I
can reach with software raid is only approx. 270 MB/s for writting.
Which is very disappointing.
I'd expect somewhat better (in the 300-400 MB/s range), but this is not
too bad.
To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.
Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
Here the results:
Each disk alone
/dev/sdc1 59.094636 MB/s
/dev/sdd1 58.686592 MB/s
/dev/sde1 55.282807 MB/s
/dev/sdf1 62.271240 MB/s
/dev/sdg1 60.872891 MB/s
/dev/sdh1 62.252781 MB/s
/dev/sdi1 59.145637 MB/s
/dev/sdj1 60.921119 MB/s
sdc + sdd in parallel (2 disks on same channel)
/dev/sdc1 42.512287 MB/s
/dev/sdd1 43.118483 MB/s
sdc + sdg in parallel (2 disks on different channels)
/dev/sdc1 42.938186 MB/s
/dev/sdg1 43.934779 MB/s
sdc + sdd + sde in parallel (3 disks on same channel)
/dev/sdc1 35.043501 MB/s
/dev/sdd1 35.686878 MB/s
/dev/sde1 34.580457 MB/s
Similar results for three disks (sdg + sdh + sdi) on the other channel
/dev/sdg1 36.381137 MB/s
/dev/sdh1 37.541758 MB/s
/dev/sdi1 35.834920 MB/s
sdc + sdd + sde + sdf in parallel (4 disks on same channel)
/dev/sdc1 31.432914 MB/s
/dev/sdd1 32.058752 MB/s
/dev/sde1 31.393455 MB/s
/dev/sdf1 33.208165 MB/s
And here for the four disks on the other channel
/dev/sdg1 31.873028 MB/s
/dev/sdh1 33.277193 MB/s
/dev/sdi1 31.910000 MB/s
/dev/sdj1 32.626744 MB/s
All 8 disks in parallel
/dev/sdc1 24.120545 MB/s
/dev/sdd1 24.419801 MB/s
/dev/sde1 24.296588 MB/s
/dev/sdf1 25.609548 MB/s
/dev/sdg1 24.572617 MB/s
/dev/sdh1 25.552590 MB/s
/dev/sdi1 24.575616 MB/s
/dev/sdj1 25.124165 MB/s
So from these results, I may assume that md is not the cause of the problem.
What comes as a big surprise is that I loose 25% performance with only
two disks and each hanging on its own channel!
Is this normal? I wonder if other people have the same problem with
other controllers or the same.
What can I do next to find out if this is a kernel, driver or hardware
problem?
Thanks,
Holger
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
| |
 |