3ware + RAID5 (+ XFS performance)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

The purpose of this email is twofold:
- to share the results of the many tests I performed with
  a 3ware RAID card + RAID-5 + XFS, pushing for better file I/O,
- and to initiate some brainstorming on what parameters can be tuned for
  getting a good performance out of this hardware under 2.6.* kernels.

I started all these tests because the performance was quite poor, meaning
that the write speed was slow, the read speed was barely acceptable, and the
system load went very high (10.0) during bonnie++ tests.
My questions are marked below with "Q".

1.
There are many useful links related to the 3ware card and related anomalies.
The bugzilla page:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434
contains some 260 comments. It is mostly 2.4 kernel and RHEL specific.

2.
A newer description of the problem can be found in the thread:
http://lkml.org/lkml/2005/4/20/110
http://openlab-debugging.web.cern.ch/openlab-debugging/raid/
by Andreas Hirstius.
There was a nasty fls() bug, which was eliminated recently, and improved
performance and stability.

3.
There are recommendations by 3ware, which can be
summarized in one line: "blockdev --setra 16384".
http://www.3ware.com/reference/techlibrary.asp
"Maximum Performance for Linux Kernel 2.6 Combined with XFS File System",
which actually leads to a PDF that has a different title:
"Benchmarking the 9000 controller with linux 2.6".


Q: Any other useful links?

Briefly, the hardware setup I use
=================================
- Tyan S2882 Thunder K8S Pro motherboard
- Dual AMD opteron CPUs
- 4Gb RAM
- 3ware 9500-8S 8 port serial ATA controller
- 8 x 300GB ST3300831AS SATA Seagate disks in hardware RAID-5
More details at the end of this email.

OS/setup
=======
- Redhat FC3, first with 2.6.9-1.667smp kernel, then with all the upgrades,
  and finally a self-compiled 2.6.12.3 x86_64 kernel
- XFS filesystem
- Raid strip size = 64k, write-cache enabled
Kernel config attached.

==========================================================================
Tuneable parameters
====================
1. Kernel itself. I tried 2.6.9-1.667smp, 2.6.11-1.14_FC3smp, and 2.6.12.3
(self-compiled)

	1.a Kernel config (NUMA system, etc.)

2. Raid setup on the card.
	- Write-cache enabled? (I use "YES")
	- Raid strip size
	- firmware, bios, etc. on the card
	- staggered spinup (I use "YES", but the drives may not support it.
	  I always "warm up" the unit before the tests, )

3. 3ware driver version
- 3w-9xxx_2.26.02.002 the older version in the kernels
- 3w-9xxx_2.26.03.015fw from the 3ware website, containing the firmware as
  well.

4. Run-time kernel parameters (my device is /dev/sde):

	4.a
		/sys/class/scsi_host/host6/
		cmd_per_lun
		can_queue

	4.b
		/sys/block/sde/queue/, e.g.
		iosched            max_sectors_kb  read_ahead_kb
		max_hw_sectors_kb  nr_requests     scheduler

	4.c
		/sys/block/sde/device/ e.g.
		queue_depth

	4.d Other params from the 2.4 kernel, if they have an alternative in
		2.6:

		/proc/sys/vm/max-readahead

	Q: Anything else?

5. blockdev --setra
	This is possibly belongs to those points mentioned under 4.)

6. For not raw IO (dd), the XFS filesystem parameters.

7. Q: Anything crucial parameter i am missing?

==========================================================================
Tests
=====
I changed the following during the tests. It is not an orthogonal set of
parameters, and I did not try everything with every combination.

- kernel
- raid strip size: 64K and 256K
- 3ware driver and firmware
- /sys/block/sde/queue/nr_requests
- blockdev --setra xxx /dev/sde
- XFS filesystem parameters

I used 5 bonnie++ commands to do not only simple IO, but also combined
filesystem performance:

MOUNT=/mnt/3w1/un0
SIZE=20480
echo "Bonnie test for IO performance"
sync; time bonnie++ -m cfhat5 -n 0   -u 0 -r 4092 -s $SIZE -f -b -d $MOUNT
echo "Testing with zero size files"
sync; time bonnie++ -m cfhat5 -n 50:0:0:50 -u 0 -r 4092 -s 0 -b -d $MOUNT
echo "Testing with tiny files"
sync; time bonnie++ -m cfhat5 -n 20:10:1:20 -u 0 -r 4092 -s 0 -b -d $MOUNT
echo "Testing with 100Kb to 1Mb files"
sync; time bonnie++ -m cfhat5 -n 10:1000000:100000:10 -u 0 -r 4092 -s 0 -b -d $MOUNT
echo "Testing with 16Mb size files"
sync; time bonnie++ -m cfhat5 -n 1:17000000:17000000:10 -u 0 -r 4092 -s 0 -b -d $MOUNT

==========================================================================
System information during the tests
===================================
This is just to make sure the system is behaving OK, and to catch some
errors. Done only outside the recorded tests, so as not to affect the
results.

1. top, or cat /proc/loadavg
to see the load

2. iostat, iostat -x

3. vmstat

4. ps -eaf
If the system behaves strange, as if locked.

Q: Anything else recommended that can be useful to check healthy system
behaviour?

==========================================================================
Other testing tools?
====================

1.  iozone
mentioning an Excel table in the man page made me uncertain whether
to try it...

2. dd
for raw IO.

Q: What else?

==========================================================================
Conclusions in a nutshell
=========================
1. With any of the kernels below 2.6.12.3, on the ___ x86_64 ___
architecture, the performance is poor. Load becomes huge, system
unresponsive, kswapd0, kswapd1 running on top of the "top".

2. The blockdev --setra 16384 does almost nothing else than increases the
read speed from the disks by also consuming much more CPU time. The write
and re-write speed do not change considerably. It is not really a
solution, when a system is run in hw raid based on an expensive card so as
to save CPU cycles for other tasks. (Then we can use sw RAID-5 on JBOD,
which is just much faster with more CPU usage)

3. The best I got during normal operation (no kswapd anomaly and
unresponsive system) was about 80Mb/s write, 40Mb/s rewrite and 350Mb/s
read. However, this was with "blockdev --setra 4092" and 43% CPU usage.
I would rather quote a more conservative 180Mb/s at setra 256 and 20% CPU.

4. I made tests Migration from 64kb to 256kb stripe size on a 2Tb array would take
forever. The performance during this migration is really bad, indifferent
from what the IO priority is set up in the 3ware interface:
50Mb/s write, 8Mb/s rewrite (!) and 12Mb/s read.

As I had no data yet to loose, it was much faster to reboot, and delete
unit, create one with 256Kb stripe size, and initialize it.

5. The performance of the 3ware card seemed worse with the 256k strip size.
Write: 68 Rewrite: 21, read: 60Mb/s

6. Changing /sys/block/sde/queue/nr_requests from 128 to 512 does a moderate
improvement. Going to higher numbers, such as 1024 does not make it better
any more.

==========================================================================
QUESTIONS:
=========

Q: Where is useful information on how to tune the various /sys/*
   parameters.?
   What are recommended values for a 2Tb array running on 3ware card?
   What are the relation between these parameters?

   Notably: nr_requests, can_queue, command_per_lun, max-readahead, etc.

Q: Are there any benchmarks showing better (re)write performance on an eight
   disk SATA RAID-5 with similar capacity (2Tb)?

Q: (mostly to 3ware/amcc inc.) Why is the 256K strip size so inefficient
    compared to the 64k?

==========================================================================
TEST RESULTS
============

	---------------------------------------------------------------------------
	TEST2.1
	-------
	raid strip size = 64k
	blockdev --setra 256 /dev/sde
	/sys/block/sde/queue/nr_requests = 128
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2

	xfs_info /mnt/3w1/un0/
	meta-data=/mnt/3w1/un0           isize=1024   agcount=32, agsize=16021136 blks
	         =                       sectsz=512
	data     =                       bsize=4096   blocks=512676288, imaxpct=25
	         =                       sunit=16     swidth=112 blks, unwritten=1
	naming   =version 2              bsize=4096
	log      =internal               bsize=4096   blocks=32768, version=2
	         =                       sectsz=512   sunit=16 blks
	realtime =none                   extsz=65536  blocks=0, rtextents=0

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	            100/100   577   5 +++++ +++   914   5   763   6 +++++ +++    97   0
	real	24m32.187s
	user	0m0.365s
	sys	0m32.705s

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max            /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	       100:10:0/100   125   2 103182 100   824   7   127   2 84106  99    82   1
	real	49m47.104s
	user	0m0.494s
	sys	1m5.833s

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    42   5    75   5   685  11    41   5    24   1   212   4
	real	18m29.176s
	user	0m0.240s
	sys	0m45.138s

	16Mb files:
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000     4  14     7  14   461  39     4  15     5  10   562  43

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000     3  14     7  14   522  40     4  14     6  11   493  39
	real	13m43.331s
	user	0m0.455s
	sys	1m53.656s

	-----------------------------------------------------------------------------
	TEST 2.2
	--------
	-> change inode size

	Strip size 64Kb
	blockdev --setra 256 /dev/sde
	/sys/block/sde/queue/nr_requests = 128
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=2k -l version=2 /dev/sde1

	meta-data=/dev/sde1              isize=2048   agcount=32, agsize=16021136 blks
	         =                       sectsz=512
	data     =                       bsize=4096   blocks=512676288, imaxpct=25
	         =                       sunit=16     swidth=112 blks, unwritten=1
	naming   =version 2              bsize=4096
	log      =internal log           bsize=4096   blocks=32768, version=2
	         =                       sectsz=512   sunit=16 blks
	realtime =none                   extsz=65536  blocks=0, rtextents=0

	Disk IO
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G 57019  97 75887  16 47033  10 35907  61 192411  22 311.6   0

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   655   6 +++++ +++   944   5   717   6 +++++ +++   112   0
	real	10m58.033s
	user	0m0.182s
	sys	0m16.954s

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   111   2 +++++ +++   805   7   107   2 +++++ +++   126   1
	real	9m23.056s
	user	0m0.105s
	sys	0m12.835s

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    44   5   221  13   504   7    43   5    22   1   164   2
	real	17m25.308s
	user	0m0.207s
	sys	0m42.914s

	==> Seq. read speed increased to 3x, seq. delete decreased

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10  4  14    10  20   450  34     4  14     5   9   419  34
	real	13m24.856s
	user	0m0.483s
	sys	1m53.478s

	==> Delete speed decreased. Seq. read speed somewhat increased.
	==> No significant difference compared to smaller inode size.

	-----------------------------------------------------------------------------
	TEST2.3
	--------
	Tests done while migrating from Stripe 64kB to Stripe 256kB.
	/sys/block/sde/queue/nr_requests = 128
	blockdev --setra 256 /dev/sde
	Extremely slow.

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           53072  11  8848   1           12039   1 139.3   0

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50  289   3 +++++ +++   603   3   444   4 +++++ +++    77   0
	real	17m19.235s
	user	0m0.186s
	sys	0m17.566s

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   86   1 +++++ +++   564   5    86   1 +++++ +++    90   0
	real	12m16.227s
	user	0m0.099s
	sys	0m12.125s

	Testing with 100Kb to 1Mb files
	Delete files in random order...done.
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10  29   3    13   0   466   6    25   3    11   0   125   2
	real	41m4.151s
	user	0m0.255s
	sys	0m42.095s

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10 2   9     2   5   273  20     2   8     1   3   258  19
	real	29m20.672s
	user	0m0.469s
	sys	1m49.345s

	===> Disk IO becomes extreme slow when array is migrating strip size

	-----------------------------------------------------------------------------
	TEST 2.4
	--------
	Tests done with 256Kb RAID array size
	blockdev --setra 256 /dev/sde
	/sys/block/sde/queue/nr_requests = 128
	mkfs.xfs -f -b size=4k -d su=256k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1

	meta-data=/dev/sde1              isize=1024   agcount=32, agsize=16021184 blks
	         =                       sectsz=512
	data     =                       bsize=4096   blocks=512676288, imaxpct=25
	         =                       sunit=64     swidth=448 blks, unwritten=1
	naming   =version 2              bsize=4096
	log      =internal log           bsize=4096   blocks=32768, version=2
	         =                       sectsz=512   sunit=64 blks
	realtime =none                   extsz=65536  blocks=0, rtextents=0

	top - 11:54:04 up 11:31,  2 users,  load average: 8.52, 7.56, 5.07
	Tasks: 104 total,   1 running, 102 sleeping,   1 stopped,   0 zombie
	Cpu(s):  0.3% us,  4.0% sy,  0.0% ni,  0.7% id, 94.5% wa,  0.0% hi,  0.5% si
	Mem:   4010956k total,  3988284k used,    22672k free,        0k buffers
	Swap:  7823576k total,      224k used,  7823352k free,  3789640k cached
	  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
	30821 root      18   0  8312  916  776 D  5.3  0.0   1:21.60 bonnie++
	  175 root      15   0     0    0    0 D  1.3  0.0   0:16.35 kswapd1
	  176 root      15   0     0    0    0 S  1.0  0.0   0:18.38 kswapd0

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           68990  14 21157   5           60837   7 250.2   0
	real	27m58.805s
	user	0m1.118s
	sys	1m58.749s

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   255   3 +++++ +++   247   2   252   3 +++++ +++    61   0
	real	23m59.997s
	user	0m0.186s
	sys	0m26.721s

	==> Much slower than 64kb size with setra=256

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   110   3 +++++ +++   243   3   112   3 +++++ +++    77   1
	real	11m57.399s
	user	0m0.100s
	sys	0m17.356s

	==> Much slower than 64kb size with setra=256

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    36   5    77   5   232   4    40   5    35   2    92   2
	real	18m25.701s
	user	0m0.238s
	sys	0m45.724s

	==> Somewhat slower than 64kb size with setra=256

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10     4  15     3   6   227  18     3  14     2   4   155  13
	real	20m11.168s
	user	0m0.508s
	sys	1m55.892s

	==> Somewhat slower than 64kb size with setra=256

	==> Definitely inferior to the 64kb raid strip size

	------------------------------------------------------------------------------
	TEST2.5
	-------
	raid strip size = 256K
	Change su to 64k
	blockdev --setra 256 /dev/sde
	/sys/block/sde/queue/nr_requests = 128
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           72627  15 23325   5           63101   7 272.0   0
	real	25m56.324s
	user	0m1.097s
	sys	1m57.267s

	===> General IO was slightly faster with su=64k than su=256k

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   788   7 +++++ +++   989   6   781   7 +++++ +++    93   0
	real	12m8.633s
	user	0m0.158s
	sys	0m16.578s


	===> Filesystem is much faster with su=64k

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   135   2 +++++ +++   818   7   133   2 +++++ +++   145   1
	real	7m51.365s
	user	0m0.091s
	sys	0m12.182s

	===> Filesystem is somewhat faster with su=64k

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    41   5    91   5   787  12    41   5    24   1   224   4
	real	18m6.138s
	user	0m0.243s
	sys	0m42.042s

	===> For larger files, it becomes almost indifferent if we use su=64k or su=256k

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10     4  14     3   6   476  34     3  11     2   5   546  40
	real	19m37.665s
	user	0m0.548s
	sys	1m49.408s

	===> For larger files, it becomes almost indifferent if we use su=64k or su=256k

	------------------------------------------------------------------------------
	TEST 2.6
	---------
	Tests done with 256Kb RAID array size
	blockdev --setra 1024 /dev/sde
	/sys/block/sde/queue/nr_requests = 128
	blockdev --setra 1024 /dev/sde
	mkfs.xfs -f -b size=4k -d su=256k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1

	meta-data=/dev/sde1              isize=1024   agcount=32, agsize=16021184 blks
	         =                       sectsz=512
	data     =                       bsize=4096   blocks=512676288, imaxpct=25
	         =                       sunit=64     swidth=448 blks, unwritten=1
	naming   =version 2              bsize=4096
	log      =internal log           bsize=4096   blocks=32768, version=2
	         =                       sectsz=512   sunit=64 blks
	realtime =none                   extsz=65536  blocks=0, rtextents=0

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           68794  14 26139   6           118452  14 255.5   0
	real	22m2.101s
	user	0m1.268s
	sys	1m58.232s

	=> Speed increased compared to TEST 2.4 (setra 256). CPU % didn't increase.

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   253   3 +++++ +++   247   2   251   3 +++++ +++    60   0
	real	24m14.398s
	user	0m0.178s
	sys	0m27.186s

	=> No change compared to 2.4

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   112   3 +++++ +++   241   3   109   3 +++++ +++    71   1
	real	12m21.663s
	user	0m0.089s
	sys	0m17.502s

	=> No change.

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    39   5    90   5   237   4    37   5    32   1    82   1
	real	18m47.223s
	user	0m0.260s
	sys	0m45.430s

	=> No change.

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10 4  13     6  12   215  16     4  14     5   9   171  13
	real	14m21.865s
	user	0m0.474s
	sys	1m49.301s

	==> Improved.

	------------------------------------------------------------------------------
	TEST 2.6
	--------
	Back to raid-strip = 64k
	/sys/block/sde/queue/nr_requests = 128
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1
	blockdev --setra 256 /dev/sde

	top - 10:51:03 up  8:06,  3 users,  load average: 9.69, 4.18, 1.63
	Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
	Cpu(s):  0.2% us,  5.0% sy,  0.0% ni,  5.2% id, 88.5% wa,  0.0% hi,  1.2% si
	Mem:   4010956k total,  3987456k used,    23500k free,       52k buffers
	Swap:  7823576k total,      224k used,  7823352k free,  3677224k cached

	System stays responsive despite the giant load.

	  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
	 5757 root      18   0  8308  916  776 D  6.3  0.0   0:35.69 bonnie++
	  176 root      15   0     0    0    0 D  1.3  0.0   0:05.27 kswapd0
	  175 root      15   0     0    0    0 S  1.0  0.0   0:05.64 kswapd1

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           65322  14 46177  10           183637  21 293.2   0
	real	15m23.264s
	user	0m1.118s
	sys	1m58.544s

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   701   6 +++++ +++   983   5   733   6 +++++ +++   111   0
	real	10m56.735s
	user	0m0.171s
	sys	0m15.877s

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   109   2 +++++ +++   824   7   108   2 +++++ +++   147   1
	real	8m58.359s
	user	0m0.107s
	sys	0m12.546s

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    45   5   214  13   642   9    45   5    22   1   211   3
	real	16m59.573s
	user	0m0.230s
	sys	0m42.618s

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10     4  13    11  20   467  32     4  13     5   9   416  30
	real	13m15.243s
	user	0m0.534s
	sys	1m47.777s

	------------------------------------------------------------------------------
	TEST 2.7
	---------
	Change setra:
	blockdev --setra 4092 /dev/sde
	raid-strip = 64k
	/sys/block/sde/queue/nr_requests = 128
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1

	[root@cfhat5 diskio]# iostat -x /dev/sde
	Linux 2.6.12.3-GB2 (cfhat5)     07/25/2005

	avg-cpu:  %user   %nice    %sys %iowait   %idle
	           0.29    0.04    1.00    4.88   93.80
	Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
	sde          0.04 903.28 19.74 44.03 4757.48 8632.40  2378.74  4316.20   209.94     7.73  121.17   1.96  12.51

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           66303  13 41254   9           345730  41 274.7   0
	real	15m21.055s
	user	0m1.114s
	sys	1m57.199s

	==> Write does not change. Rewrite decreases. Read increases.

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   624   6 +++++ +++   904   5   727   6 +++++ +++   113   0
	real	10m59.528s
	user	0m0.189s
	sys	0m16.520s

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   111   2 +++++ +++   798   7   102   2 +++++ +++   143   1
	real	9m12.536s
	user	0m0.120s
	sys	0m12.467s

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10  46   6   323  20   686  10    43   5    30   1   207   3
	real	14m42.960s
	user	0m0.262s
	sys	0m42.090s

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10     4  14    20  40   524  38     4  13    11  21   492  35
	real	10m42.784s
	user	0m0.453s
	sys	1m51.078s

	------------------------------------------------------------------------------
	TEST 2.8
	---------
	echo 512 > /sys/block/sde/queue/nr_requests
	raid-strip = 64k
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1
	blockdev --setra 4092 /dev/sde

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           78573  16 42444   9           353894  42 284.6   0
	real	14m14.938s
	user	0m1.213s
	sys	1m55.382s

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   623   6 +++++ +++   894   5   739   6 +++++ +++   123   0
	real	10m25.379s
	user	0m0.186s
	sys	0m16.846s

	Testing with tiny files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20   107   2 +++++ +++   835   7   100   1 +++++ +++   159   1
	real	9m7.268s
	user	0m0.104s
	sys	0m12.589s

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10    47   6   324  19   697  10    44   5    35   2   232   4
	real	13m41.706s
	user	0m0.234s
	sys	0m42.614s

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10     4  14    19  38   448  32     4  13    11  21   506  36
	real	10m40.404s
	user	0m0.469s
	sys	1m51.098s

	------------------------------------------------------------------------------
	TEST 2.9
	---------
	echo 1024 > /sys/block/sde/queue/nr_requests
	raid-strip = 64k
	mkfs.xfs -f -b size=4k -d su=64k,sw=7 -i size=1k -l version=2 -L cfhat5_1_un0 /dev/sde1
	blockdev --setra 4092 /dev/sde

	Bonnie test for IO performance
	Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
	                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
	Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
	cfhat5          20G           79546  16 41227   9           351637  43 285.0   0
	real	14m26.609s
	user	0m1.136s
	sys	1m57.398s

	==> No improvement

	Testing with zero size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	              50/50   616   5 +++++ +++   880   5   748   6 +++++ +++   123   0
	real	10m25.469s
	user	0m0.186s
	sys	0m16.723s

	Testing with tiny files
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	         20:10:1/20    99   2 +++++ +++   779   7   104   2 +++++ +++   165   1
	real	9m12.385s
	user	0m0.111s
	sys	0m12.947s

	Testing with 100Kb to 1Mb files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	10:1000000:100000/10  47   6   316  20   616   9    47   6    36   2   248   4
	real	13m22.360s
	user	0m0.231s
	sys	0m43.679s

	Testing with 16Mb size files
	Version  1.03       ------Sequential Create------ --------Random Create--------
	cfhat5              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
	files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
	1:17000000:17000000/10 3  13    16  31   386  27     4  13    11  22   558  40
	real	11m1.018s
	user	0m0.464s
	sys	1m49.534s



============================================================================
Hardware info
=============

[root@cfhat5 diskio]# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 5
model name      : AMD Opteron(tm) Processor 246
stepping        : 10
cpu MHz         : 1991.008
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips        : 3915.77
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 5
model name      : AMD Opteron(tm) Processor 246
stepping        : 10
cpu MHz         : 1991.008
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
bogomips        : 3973.12
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

-----------------------------------------------------------
[root@cfhat5 diskio]# cat /sys/class/scsi_host/host6/stats
3w-9xxx Driver version: 2.26.03.015fw
Current commands posted:      0
Max commands posted:         79
Current pending commands:     0
Max pending commands:         1
Last sgl length:              2
Max sgl length:              32
Last sector count:            0
Max sector count:           256
SCSI Host Resets:             0
AEN's:                        0

--------------------------
3ware card info

Model   9500S-8
Serial #      L19403A5100293
Firmware      FE9X 2.06.00.009
Driver        2.26.03.015fw
BIOS  BE9X 2.03.01.051
Boot Loader   BL9X 2.02.00.001
Memory Installed      112 MB
# of Ports    8
# of Units    1
# of Drives   8

Write cache enabled
Auto-spin up enabled, 2 sec between spin-up
Drives, however, probably do not support spinup.

-------------------------------
Disks:
Drive Information (Controller ID 6)
Port 	Model 	Capacity 	Serial # 	Firmware 	Unit 	Status
0 	ST3300831AS 	279.46 GB 	3NF0BZYJ 	3.02 	0 	OK
1 	ST3300831AS 	279.46 GB 	3NF0AC04 	3.01 	0 	OK
2 	ST3300831AS 	279.46 GB 	3NF0A7JE 	3.01 	0 	OK
3 	ST3300831AS 	279.46 GB 	3NF0ABT1 	3.01 	0 	OK
4 	ST3300831AS 	279.46 GB 	3NF0A63J 	3.01 	0 	OK
5 	ST3300831AS 	279.46 GB 	3NF0ACC5 	3.01 	0 	OK
6 	ST3300831AS 	279.46 GB 	3NF09FLP 	3.01 	0 	OK
7 	ST3300831AS 	279.46 GB 	3NF046WY 	3.01 	0 	OK

----------------------------------
[root@cfhat5 diskio]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0    380 3781540      0  58004    0    0  2712  3781  243   216  0  2 91  7

[root@cfhat5 diskio]# free
             total       used       free     shared    buffers     cached
Mem:       4010956     229532    3781424          0          0      58004
-/+ buffers/cache:     171528    3839428
Swap:      7823576        380    7823196


============================================================================
Kernel config

See at
http://www.cfa.harvard.edu/~gbakos/diskio/


[Index of Archives]     [Current Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [Yosemite Photos]     [KDE Users]     [Fedora Tools]     [Fedora Docs]

  Powered by Linux