Why are you booting with 'noapic'.. in my experience
that will seriously
impact interrupt performance. Use the APIC if you've
got it, which in this
case you definitely do.
Yes, having your gigabit NIC and RAID controller on
the same IRQ (in PIC
mode) could definitely me a source of trouble.
In your web server testing, were you using an
external traffic generator or
an on-host process? If you try on-host (eliminating
the network throughput
and related interrupts) does performance improve?
So two biggest suggestions:
- Use the APIC. It is your friend.
- It looks like the 3ware card and gigabit nic are
on different busses, but
the pirq lines are being routed to the same legacy
interrupt in PIC mode. So
APIC mode should avoid that problem. If the
controller and nic are actually
on the same bus, separate them.
Regards,
Ian Morgan
--
On Thu, 29 Sep 2005, subbie subbie wrote:
Dear list,
After almost two weeks of experimentation, google
searches and reading of posts, bug reports and
discussions I'm still far from an answer. I'm
hoping
someone on this list could shed some light on the
subject.
I'm using a 3Ware 9500S-12 card and am able to
produce
up to 400MB/s sustained read from my 12-disk 4.1TB
RAID5 SATA array, 128MB cache onboard, ext3
formatted.
All is well when performing a single read -- it
works nice and fast.
The system is a web server, serving mid-size files
(50MB, each, on average). All hell breaks loose
when
doing many concurrent reads, anywhere between 200
to
400 concurrent streams things simply grind to a
halt
and the system transfers a maximum of 12-14MB/s.
I'm in the process of clearing up the array (this
would take some time) and restructuring it to JBOD
mode in order to use each disk individually. I
will
use a filesystem more suitable to streaming large
files, such as XFS. But this would take time and
I
would very much appreciate the advice of people in
the
know if this is going to help at all. It's hard
for
me to make extreme experimentation (deleting,
formatting, reformatting) as this is a productio n
system with many files that I have no other place
to
dump until they can be safely removed. Though I'm
working on dumping them slowly to other, remote,
machines.
I'm running the latest kernel, 2.6.13.2 and the
latest
3Ware driver, taken from the 3ware.com web site
which
upon insmod, updates the card's firmware to the
latest
version as well.
In my experiments, I've tried using larger
readahead,
currently at 16k (this helps, higher values do not
seem to help much), using the deadline scheduler
for
this device, booting the system with the 'noapic'
option and playing with a bunch of VM tunable
parameters which I'm not sure that I should really
be
touching. At the moment only the readahead
modification is used as the other stuff simply
didn't
help at all.
With the stock kernel shipped with my
distribution,
2.6.8 and its old 3ware driver things were just as
worse but manifested themselves differently.
The
system was visibly (top, vmstat...) spending most
of
its time in io-wait and load average was extremely
high, in the area of 10 to 20. With the recent
kernel and driver mentioned above, the excessive
io-wait and load seems to have been resolved and
observed loadavg is between 1 and 4.
I don't have much experience with systems that are
supposed to stream many files concurrently off a
hardware RAID of this configuration, but my gut
feeling is that something is very wrong and I
should
be seeing a much higher read throughput.
Trying to preempt people's questions I've tried to
include as much information as possible, a lot of
stuff is pasted below.
I've just seen that the 3ware driver shares the
same
IRQ with my ethernet card, which has got me a
little
worried, should I be?
System uptime, exactly 1 day:
# cat /proc/interrupts
CPU0 CPU1
0: 21619638 0 XT-PIC timer
2: 0 0 XT-PIC cascade
8: 4 0 XT-PIC rtc
10: 268753224 0 XT-PIC
3w-9xxx,
eth0
14: 11 0 XT-PIC ide0
15: 337881 0 XT-PIC libata
NMI: 0 0
LOC: 21110402 21557685
ERR: 0
MIS: 0
# free
total used free shared
buffers cached
Mem: 2075260 2024724 50536
0
5184 1388408
-/+ buffers/cache: 631132 1444128
Swap: 3903784 0 3903784
# vmstat -n 1 (output of the last few seconds):
procs -----------memory---------- ---swap--
-----io---- --system-- ----cpu----
0 0 0 49932 4760 1392980 0 0 15636
32 3169 3697 4 6 30 60
0 0 0 50816 4752 1392376 0 0 5844
0 3114 3929 3 5 91 1
0 0 0 50924 4772 1391404 0 0 9360
0 3187 4348 6 6 76 13
0 2 0 50552 4780 1391532 0 0 24976
44 4077 3906 3 7 65 25
0 1 0 50444 4780 1392688 0 0 20192
0 5048 3914 7 8 56 30
0 1 0 50568 4756 1392508 0 0 21248
0 4060 3603 4 6 48 41
0 0 0 50704 4724 1392268 0 0 30004
0 3834 3369 4 9 65 22
0 3 0 50556 4728 1392468 0 0 3248
1832 2974 4514 2 5 58 35
0 3 0 50308 4724 1392200 0 0 1288
336 1766 1886 1 3 50 47
0 4 0 50308 4732 1391852 0 0 2300
408 1919 2158 0 3 51 46
0 4 0 50556 4736 1390692 0 0 1856
532 1488 1846 3 1 50 46
0 3 0 50680 4740 1390620 0 0 4016
1296 1577 1682 2 2 50 47
0 3 0 50432 4752 1391628 0 0 2180
72 1730 1945 2 2 51 46
2 2 0 49924 4772 1391540 0 0 44372
564 3403 2847 4 5 50 42
0 0 0 50684 4784 1391528 0 0 28640
216 3804 3847 7 8 69 16
