Re: Where is the performance bottleneck?

[Holger Kiehl <Holger.Kiehl@dwd.de>]

On Wed, 31 Aug 2005, Jens Axboe wrote:

> Nothing sticks out here either. There's plenty of idle time. It smells
> like a driver issue. Can you try the same dd test, but read from the
> drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:

   dd if=/dev/sd?1 of=/dev/null bs=256k count=78125

Here vmstat output (I just cut something out in the middle):

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----^M
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa^M
 3  7   4348  42640 7799984   9612    0    0 322816     0 3532  4987  0 22  0 78
 1  7   4348  42136 7800624   9584    0    0 322176     0 3526  4987  0 23  4 74
 0  8   4348  39912 7802648   9668    0    0 322176     0 3525  4955  0 22 12 66
 1  7   4348  38912 7803700   9636    0    0 322432     0 3526  5078  0 23  7 70
 2  6   4348  37552 7805120   9644    0    0 322432     0 3527  4908  0 23 12 64
 0  8   4348  41152 7801552   9608    0    0 322176     0 3524  5018  0 24  6 70
 1  7   4348  41644 7801044   9572    0    0 322560     0 3530  5175  0 23  0 76
 1  7   4348  37184 7805396   9640    0    0 322176     0 3525  4914  0 24 18 59
 3  7   4348  41704 7800376   9832    0    0 322176    20 3531  5080  0 23  4 73
 1  7   4348  40652 7801700   9732    0    0 323072     0 3533  5115  0 24 13 64
 1  7   4348  40284 7802224   9616    0    0 322560     0 3527  4967  0 23  1 76
 0  8   4348  40156 7802356   9688    0    0 322560     0 3528  5080  0 23  2 75
 6  8   4348  41896 7799984   9816    0    0 322176     0 3530  4945  0 24 20 57
 0  8   4348  39540 7803124   9600    0    0 322560     0 3529  4811  0 24 21 55
 1  7   4348  41520 7801084   9600    0    0 322560     0 3532  4843  0 23 22 55
 0  8   4348  40408 7802116   9588    0    0 322560     0 3527  5010  0 23  4 72
 0  8   4348  38172 7804300   9580    0    0 322176     0 3526  4992  0 24  7 69
 4  7   4348  42264 7799784   9812    0    0 322688     0 3529  5003  0 24  8 68
 1  7   4348  39908 7802520   9660    0    0 322700     0 3529  4963  0 24 14 62
 0  8   4348  37428 7805076   9620    0    0 322420     0 3528  4967  0 23 15 62
 0  8   4348  37056 7805348   9688    0    0 322048     0 3525  4982  0 24 26 50
 1  7   4348  37804 7804456   9696    0    0 322560     0 3528  5072  0 24 16 60
 0  8   4348  38416 7804084   9660    0    0 323200     0 3533  5081  0 24 23 53
 0  8   4348  40160 7802300   9676    0    0 323200    28 3543  5095  0 24 17 59
 1  7   4348  37928 7804612   9608    0    0 323072     0 3532  5175  0 24  7 68
 2  6   4348  38680 7803724   9612    0    0 322944     0 3531  4906  0 25 24 51
 1  7   4348  40408 7802192   9648    0    0 322048     0 3524  4947  0 24 19 57

Full vmstat session can be found under:

  ftp://ftp.dwd.de/pub/afd/linux_kernel_debug/vmstat-256k-read

And here the profile data:

2106577 total                                      0.9469
1638177 default_idle                             34128.6875
179615 copy_user_generic_c                      4726.7105
 27670 end_buffer_async_read                    108.0859
 26055 shrink_zone                                7.1111
 23199 __make_request                            17.2612
 17221 kmem_cache_free                          153.7589
 11796 drop_buffers                              52.6607
 11016 add_to_page_cache                         52.9615
  9470 __wake_up_bit                            197.2917
  8760 buffered_rmqueue                          12.4432
  8646 find_get_page                             90.0625
  8319 __do_page_cache_readahead                 11.0625
  7976 kmem_cache_alloc                         124.6250
  7463 scsi_request_fn                            6.2192
  7208 try_to_free_buffers                       40.9545
  6716 create_empty_buffers                      41.9750
  6432 __end_that_request_first                  11.8235
  6044 test_clear_page_dirty                     25.1833
  5643 scsi_dispatch_cmd                          9.7969
  5588 free_hot_cold_page                        19.4028
  5479 submit_bh                                 18.0230
  3903 __alloc_pages                              3.2965
  3671 file_read_actor                            9.9755
  3425 thread_return                             14.2708
  3333 generic_make_request                       5.6301
  3294 bio_alloc_bioset                           7.6250
  2868 bio_put                                   44.8125
  2851 mpt_interrupt                              2.8284
  2697 mempool_alloc                              8.8717
  2642 block_read_full_page                       3.9315
  2512 do_generic_mapping_read                    2.1216
  2394 set_page_refs                            149.6250
  2235 alloc_page_buffers                         9.9777
  1992 __pagevec_lru_add                          8.3000
  1859 __memset                                   9.6823
  1791 page_waitqueue                            15.9911
  1783 scsi_end_request                           6.9648
  1348 dma_unmap_sg                               6.4808
  1324 bio_endio                                 11.8214
  1306 unlock_page                               20.4062
  1211 mptscsih_freeChainBuffers                  7.5687
  1141 alloc_pages_current                        7.9236
  1136 __mod_page_state                          35.5000
  1116 radix_tree_preload                         8.7188
  1061 __pagevec_release_nonlru                   6.6312
  1043 set_bh_page                                9.3125
  1024 release_pages                              2.9091
  1023 mempool_free                               6.3937
   832 alloc_buffer_head                         13.0000

Full profile data can be found under:

   ftp://ftp.dwd.de/pub/afd/linux_kernel_debug/dd-256k-8disk-read.profile

> You might want to try the same with direct io, just to eliminate the
> costly user copy. I don't expect it to make much of a difference though,
> feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?

> If we still can't get closer to this, it would be interesting to try my
> block tracing stuff so we can see what is going on at the queue level.
> But lets gather some more info first, since it requires testing -mm.
Ok, please then just tell me what I must do.

Thanks,
Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/