In article <[email protected]>,
subbie subbie <[email protected]> wrote:
>I'm using a 3Ware 9500S-12 card and am able to produce
>up to 400MB/s sustained read from my 12-disk 4.1TB
>RAID5 SATA array, 128MB cache onboard, ext3 formatted.
> All is well when performing a single read -- it
>works nice and fast.
>
>The system is a web server, serving mid-size files
>(50MB, each, on average). All hell breaks loose when
>doing many concurrent reads, anywhere between 200 to
>400 concurrent streams things simply grind to a halt
>and the system transfers a maximum of 12-14MB/s.
There are a couple of things you should do:
1. Use the CFQ I/O scheduler, and increase nr-requests:
echo cfq > /sys/block/hda/queue/scheduler
echo 1024 > /sys/block/hda/queue/nr_requests
2. Make sure that your filesystem knows about the stripe size
and number of disks in the array. E.g. for a raid5 array
with a stripe size of 64K and 6 disks (effectively 5,
because in every stripe-set there is on disk doing parity):
# ext3 fs, 5 disks, 64K stripe, units in 4K blocks
mkfs -text3 -E stride=$((64/4))
# xfs, 5 disks, 64K stripe, units in 512 bytes
mkfs -txfs -d sunit=$((64*2)) -d swidth=$((5*64*2))
3. Don't use partitions. Partions do not start on a multiple of
the (stripe_size * nr_disks), so your I/O will be misaligned
and the settings in (2) will have no or an adverse effect.
If you must use partitions, either build them manually
with sfdisk so that partitions do start on that multiple,
or use LVM.
4. Reconsider your stripe size for streaming large files.
If you have say 4 disks, and a 64K
stripe size, then a read of a block of 256K will busy all 4 disks.
Many simultaneous threads reading blocks of 256K will result in
trashings disks as they all want to read from all 4 disks .. so
in that case, using a stripesize of 256K will make things better.
One read of 256K (in the ideal, aligned case) will just keep one
disk busy. 4 reads can happen in parallel without trashing. Esp.
in this case, you need the alignment I talked about in (3).
5. Defragment the files.
If the files are written sequentially, they will not be fragmented.
But if they were stored by writing to thousands of them appending
a few K at a time in round-robin fashion, you need to defragment..
in the case of XFS, run xfs_fsr every so often.
Good luck,
Mike.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
