Re: Where is the performance bottleneck?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, 31 Aug 2005, Jens Axboe wrote:

On Wed, Aug 31 2005, Vojtech Pavlik wrote:
On Tue, Aug 30, 2005 at 08:06:21PM +0000, Holger Kiehl wrote:
How does one determine the PCI-X bus speed?

Usually only the card (in your case the Symbios SCSI controller) can
tell. If it does, it'll be most likely in 'dmesg'.

There is nothing in dmesg:

   Fusion MPT base driver 3.01.20
   Copyright (c) 1999-2004 LSI Logic Corporation
   ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 24 (level, low) -> IRQ 217
   mptbase: Initiating ioc0 bringup
   ioc0: 53C1030: Capabilities={Initiator,Target}
   ACPI: PCI Interrupt 0000:02:04.1[B] -> GSI 25 (level, low) -> IRQ 225
   mptbase: Initiating ioc1 bringup
   ioc1: 53C1030: Capabilities={Initiator,Target}
   Fusion MPT SCSI Host driver 3.01.20

To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.

Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.

Ok, I did run the following dd command in different combinations:

   dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000

I think a bs of 4k is way too small and will cause huge CPU overhead.
Can you try with something like 4M? Also, you can use /dev/full to avoid
the pre-zeroing.

That was my initial thought as well, but since he's writing the io side
should look correct. I doubt 8 dd's writing 4k chunks will gobble that
much CPU as to make this much difference.

Holger, we need vmstat 1 info while the dd's are running. A simple
profile would be nice as well, boot with profile=2 and do a readprofile
-r; run tests; readprofile > foo and send the first 50 lines of foo to
this list.

Here vmstat for 8 dd's still with 4k blocksize:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 9  2   5244  38272 7738248  10400    0    0     3 11444  390    24  0  5 75 20
 5 10   5244  30824 7747680   8684    0    0     0 265672 2582  1917  1 95  0  4
 2 12   5244  30948 7747248   8708    0    0     0 222620 2858   292  0 33  0 67
 4 10   5244  31072 7747516   8644    0    0     0 236400 3132   326  0 43  0 57
 2 12   5244  31320 7747792   8512    0    0     0 250204 3225   285  0 37  0 63
 1 13   5244  30948 7747412   8552    0    0    24 227600 3261   312  0 41  0 59
 2 12   5244  32684 7746124   8616    0    0     0 235392 3219   274  0 32  0 68
 1 13   5244  30948 7747940   8568    0    0     0 228020 3394   296  0 37  0 63
 0 14   5244  31196 7747680   8624    0    0     0 232932 3389   300  0 32  0 68
 3 12   5244  31072 7747904   8536    0    0     0 233096 3545   312  0 33  0 67
 1 13   5244  31072 7747852   8520    0    0     0 226992 3381   290  0 31  0 69
 1 13   5244  31196 7747704   8396    0    0     0 230112 3372   265  0 28  0 72
 0 14   5244  31072 7747928   8512    0    0     0 240652 3491   295  0 33  0 67
 3 13   5244  31072 7748104   8608    0    0     0 222944 3433   269  0 27  0 73
 1 13   5244  31072 7748000   8508    0    0     0 207944 3470   294  0 28  0 72
 0 14   5244  31072 7747980   8528    0    0     0 234608 3496   272  0 31  0 69
 2 12   5244  31196 7748148   8496    0    0     0 228760 3480   280  0 28  0 72
 0 14   5244  30948 7748568   8620    0    0     0 214372 3551   302  0 29  0 71
 1 13   5244  31072 7748392   8524    0    0     0 226732 3494   284  0 29  0 71
 0 14   5244  31072 7748004   8640    0    0     0 229628 3604   273  0 26  0 74
 1 13   5244  30948 7748392   8660    0    0     0 212868 3563   266  0 28  0 72
 1 13   5244  30948 7748600   8520    0    0     0 228244 3568   294  0 30  0 70
 1 13   5244  31196 7748228   8416    0    0     0 221692 3543   258  0 27  0 73
 1 13   5244  31072 7748192   8520    0    0     0 241040 3983   330  0 25  0 74
 1 13   5244  31196 7748288   8560    0    0     0 217108 3676   276  0 28  0 72
                             .
                             .
                             .
               This goses on up to the end.
                             .
                             .
                             .
 0  3   5244 825096 6949252   8596    0    0     0 241244 2683   223  0  7 71 22
 0  2   5244 825108 6949252   8596    0    0     0 229764 2683   214  0  7 73 20
 0  3   5244 826348 6949252   8596    0    0     0 116840 2046   450  0  4 71 26
 0  3   5244 826976 6949252   8596    0    0     0 141992 1887    97  0  4 73 23
 0  3   5244 827100 6949252   8596    0    0     0 137716 1871    93  0  4 70 26
 0  3   5244 827100 6949252   8596    0    0     0 137032 1894    96  0  4 75 21
 0  3   5244 827224 6949252   8596    0    0     0 131332 1860   288  0  4 73 23
 0  1   5244 1943732 5833756   8620    0    0     0 72404 1560   481  0 24 61 16
 0  2   5244 1943732 5833756   8620    0    0     0 71680 1450    60  0  2 61 38
 0  2   5244 1943736 5833756   8620    0    0     0 71680 1464    70  0  2 52 46
 0  2   5244 1943736 5833756   8620    0    0     0 66560 1436    66  0  2 50 48
 0  2   5244 1943984 5833756   8620    0    0     0 71680 1454    72  0  2 50 48
 0  2   5244 1943984 5833756   8620    0    0     0 71680 1450    70  0  2 50 48
 1  0   5244 2906484 4872176   8612    0    0     0 12760 1240   321  0 13 68 19
 0  0   5244 3306732 4472300   8580    0    0     0     0 1109    31  0  9 91  0
 0  0   5244 3306732 4472300   8580    0    0     0     0 1008    22  0  0 100  0

And here the profile output (I assume you meant sorted):

3236497 total                                      1.4547
2507913 default_idle                             52248.1875
158752 shrink_zone                               43.3275
121584 copy_user_generic_c                      3199.5789
 34271 __wake_up_bit                            713.9792
 31131 __make_request                            23.1629
 22096 scsi_request_fn                           18.4133
 21915 rotate_reclaimable_page                   80.5699
 20641 end_buffer_async_write                    86.0042
 18701 __clear_user                             292.2031
 13562 __block_write_full_page                   18.4266
 12981 test_set_page_writeback                   47.7243
 10772 kmem_cache_free                           96.1786
 10216 unlock_page                              159.6250
  9492 free_hot_cold_page                        32.9583
  9478 add_to_page_cache                         45.5673
  9117 page_waitqueue                            81.4018
  8671 drop_buffers                              38.7098
  8584 __set_page_dirty_nobuffers                31.5588
  8444 release_pages                             23.9886
  8204 scsi_dispatch_cmd                         14.2431
  8191 buffered_rmqueue                          11.6349
  7966 page_referenced                           22.6307
  7093 generic_file_buffered_write                4.1431
  6953 __pagevec_lru_add                         28.9708
  6740 __alloc_pages                              5.6926
  6369 __end_that_request_first                  11.7077
  5940 dnotify_parent                            30.9375
  5880 kmem_cache_alloc                          91.8750
  5797 submit_bh                                 19.0691
  4720 find_lock_page                            21.0714
  4612 __generic_file_aio_write_nolock            4.8042
  4559 __do_softirq                              20.3527
  4337 end_page_writeback                        54.2125
  4090 create_empty_buffers                      25.5625
  3985 bio_alloc_bioset                           9.2245
  3787 mempool_alloc                             12.4572
  3708 set_page_refs                            231.7500
  3545 __block_commit_write                      17.0433
  3037 system_call                               23.1832
  2968 zone_watermark_ok                         15.4583
  2966 cond_resched                              26.4821
  2828 generic_make_request                       4.7770
  2766 __mod_page_state                          86.4375
  2759 fget_light                                15.6761
  2692 test_clear_page_dirty                     11.2167
  2523 vfs_write                                  8.2993
  2406 generic_file_aio_write_nolock             15.0375
  2335 bio_put                                   36.4844
  2287 bad_range                                 23.8229

Under ftp://ftp.dwd.de/pub/afd/linux_kernel_debug/ I put the full vmstat
and profile output (also with -v). There is also dmesg and my kernel.config
from this system.

I will also do some test with 4M instead of 4k and as Al Boldi hinted
do a test together with some CPU load.

Thanks,
Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Index of Archives]     [Kernel Newbies]     [Netfilter]     [Bugtraq]     [Photo]     [Gimp]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Video 4 Linux]     [Linux for the blind]
  Powered by Linux