I have seen an 80GB/sec limitation in the kernel unless this value is
changed in the SCSI I/O layer
for 3Ware and other controllers during testing of 2.6.X series kernels.
Change these values in include/linux/blkdev.h and performance goes from
80MB/S to over 670MB/S on the 3Ware controller.
//#define BLKDEV_MIN_RQ 4
//#define BLKDEV_MAX_RQ 128 /* Default maximum */
#define BLKDEV_MIN_RQ 4096
#define BLKDEV_MAX_RQ 8192 /* Default maximum */
Jeff
Jens Axboe wrote:
On Wed, Aug 31 2005, Holger Kiehl wrote:
On Wed, 31 Aug 2005, Jens Axboe wrote:
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us sy id
wa^M
3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22
0 78
1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23
4 74
0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22
12 66
1 7 4348 38912 7803700 9636 0 0 322432 0 3526 5078 0 23
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#define __USE_GNU
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#define BS (131072)
#define ALIGN(buf) (char *) (((unsigned long) (buf) + 4095) & ~(4095))
#define BLOCKS (8192)
int main(int argc, char *argv[])
{
char *p;
int fd, i;
if (argc < 2) {
printf("%s: <dev>\n", argv[0]);
return 1;
}
fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd == -1) {
perror("open");
return 1;
}
p = ALIGN(malloc(BS + 4095));
for (i = 0; i < BLOCKS; i++) {
int r = read(fd, p, BS);
if (r == BS)
continue;
else {
if (r == -1)
perror("read");
break;
}
}
return 0;
}
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
| |
 |