Re: Where is the performance bottleneck?

[Holger Kiehl <Holger.Kiehl@dwd.de>]

On Wed, 31 Aug 2005, Jens Axboe wrote:

> On Wed, Aug 31 2005, Holger Kiehl wrote:
> > On Wed, 31 Aug 2005, Jens Axboe wrote:
> >
> > > Nothing sticks out here either. There's plenty of idle time. It smells
> > > like a driver issue. Can you try the same dd test, but read from the
> > > drives instead? Use a bigger blocksize here, 128 or 256k.
> > I used the following command reading from all 8 disks in parallel:
> >
> >    dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
> >
> > Here vmstat output (I just cut something out in the middle):
> >
> > procs -----------memory---------- ---swap-- -----io---- --system--
> > ----cpu----^M
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
> >  wa^M
> >  3  7   4348  42640 7799984   9612    0    0 322816     0 3532  4987  0 22
> >  0 78
> >  1  7   4348  42136 7800624   9584    0    0 322176     0 3526  4987  0 23
> >  4 74
> >  0  8   4348  39912 7802648   9668    0    0 322176     0 3525  4955  0 22
> >  12 66
> >  1  7   4348  38912 7803700   9636    0    0 322432     0 3526  5078  0 23
>
> Ok, so that's somewhat better than the writes but still off from what
> the individual drives can do in total.
>
> > > You might want to try the same with direct io, just to eliminate the
> > > costly user copy. I don't expect it to make much of a difference though,
> > > feels like the problem is elsewhere (driver, most likely).
> > Sorry, I don't know how to do this. Do you mean using a C program
> > that sets some flag to do direct io, or how can I do that?
>
> I've attached a little sample for you, just run ala
>
> # ./oread /dev/sdX
>
> and it will read 128k chunks direct from that device. Run on the same
> drives as above, reply with the vmstat info again.
Using kernel 2.6.12.5 again, here the results:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0      0 8009648   4764  40592    0    0     0     0 1011    32  0  0 100  0
 0  0      0 8009648   4764  40592    0    0     0     0 1011    34  0  0 100  0
 0  0      0 8009648   4764  40592    0    0     0     0 1008    61  0  0 100  0
 0  0      0 8009648   4764  40592    0    0     0     0 1006    26  0  0 100  0
 0  8      0 8006372   4764  40592    0    0 120192     0 1944  1929  0  1 89 10
 2  8      0 8006372   4764  40592    0    0 319488     0 3502  4999  0  2 75 24
 0  8      0 8006372   4764  40592    0    0 319488     0 3506  4995  0  2 75 24
 0  8      0 8006372   4764  40592    0    0 319744     0 3504  4999  0  1 75 24
 0  8      0 8006372   4764  40592    0    0 319488     0 3507  5009  0  2 75 23
 0  8      0 8006372   4764  40592    0    0 319616     0 3506  5011  0  2 75 24
 0  8      0 8005124   4800  41100    0    0 319976     0 3536  4995  0  2 73 25
 0  8      0 8005124   4800  41100    0    0 323584     0 3534  5000  0  2 75 23
 0  8      0 8005124   4800  41100    0    0 323968     0 3540  5035  0  1 75 24
 0  8      0 8005124   4800  41100    0    0 319232     0 3506  4811  0  1 75 24
 0  8      0 8005504   4800  41100    0    0 317952     0 3498  4747  0  1 75 24
 0  8      0 8005504   4800  41100    0    0 318720     0 3495  4672  0  2 75 23
 1  8      0 8005504   4800  41100    0    0 318720     0 3509  4707  0  1 75 24
 0  8      0 8005504   4800  41100    0    0 318720     0 3499  4667  0  2 75 23
 0  8      0 8005504   4808  41092    0    0 318848    40 3509  4674  0  1 75 24
 0  8      0 8005380   4808  41092    0    0 318848     0 3497  4693  0  2 72 26
 0  8      0 8005380   4808  41092    0    0 318592     0 3500  4646  0  2 75 23
 0  8      0 8005380   4808  41092    0    0 318592     0 3495  4828  0  2 61 37
 0  8      0 8005380   4808  41092    0    0 318848     0 3499  4827  0  1 62 37
 1  8      0 8005380   4808  41092    0    0 318464     0 3495  4642  0  2 75 23
 0  8      0 8005380   4816  41084    0    0 318848    32 3511  4672  0  1 75 24
 0  8      0 8005380   4816  41084    0    0 320640     0 3512  4877  0  2 75 23
 0  8      0 8005380   4816  41084    0    0 322944     0 3533  5047  0  2 75 24
 0  8      0 8005380   4816  41084    0    0 322816     0 3531  5053  0  1 75 24
 0  8      0 8005380   4816  41084    0    0 322944     0 3531  5048  0  2 75 23
 0  8      0 8005380   4816  41084    0    0 322944     0 3529  5043  0  1 75 24
 0  0      0 8008360   4816  41084    0    0 266880     0 3112  4224  0  2 78 20
 0  0      0 8008360   4816  41084    0    0     0     0 1012    28  0  0 100  0

Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/