Re: Where is the performance bottleneck?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, 31 Aug 2005, Nick Piggin wrote:

Holger Kiehl wrote:

3236497 total                                      1.4547
2507913 default_idle                             52248.1875
158752 shrink_zone                               43.3275
121584 copy_user_generic_c                      3199.5789
 34271 __wake_up_bit                            713.9792
 31131 __make_request                            23.1629
 22096 scsi_request_fn                           18.4133
 21915 rotate_reclaimable_page                   80.5699
          ^^^^^^^^^

I don't think this function should be here. This indicates that
lots of writeout is happening due to pages falling off the end
of the LRU.

There was a bug recently causing memory estimates to be wrong
on Opterons that could cause this I think.

Can you send in 2 dumps of /proc/vmstat taken 10 seconds apart
while you're writing at full speed (with 2.6.13 or the latest
-git tree).

I took 2.6.13, there where no git snapshots at www.kernel.org when
I looked. With 2.6.13 I must load the Fusion MPT driver as module.
Compiling it in it does not detect the drive correctly, as module
there is no problem.

Here is what I did:

   #!/bin/bash

   time dd if=/dev/full of=/dev/sdc1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sdd1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sde1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sdf1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sdg1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sdh1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sdi1 bs=4M count=4883 &
   time dd if=/dev/full of=/dev/sdj1 bs=4M count=4883 &

   sleep 20

   cat /proc/vmstat > /root/vmstat-1.dump

   sleep 10

   cat /proc/vmstat > /root/vmstat-2.dump
   cat /proc/zoneinfo > /root/zoneinfo.dump
   cat /proc/meminfo > /root/meminfo.dump

   exit 0

vmstat-1.dump:

   nr_dirty 787282
   nr_writeback 44317
   nr_unstable 0
   nr_page_table_pages 633
   nr_mapped 6373
   nr_slab 53030
   pgpgin 263362
   pgpgout 5260352
   pswpin 0
   pswpout 0
   pgalloc_high 0
   pgalloc_normal 2448628
   pgalloc_dma 1041
   pgfree 2457343
   pgactivate 5775
   pgdeactivate 2113
   pgfault 465679
   pgmajfault 321
   pgrefill_high 0
   pgrefill_normal 5940
   pgrefill_dma 33
   pgsteal_high 0
   pgsteal_normal 148759
   pgsteal_dma 0
   pgscan_kswapd_high 0
   pgscan_kswapd_normal 153813
   pgscan_kswapd_dma 1089
   pgscan_direct_high 0
   pgscan_direct_normal 0
   pgscan_direct_dma 0
   pginodesteal 0
   slabs_scanned 0
   kswapd_steal 148759
   kswapd_inodesteal 0
   pageoutrun 5304
   allocstall 0
   pgrotated 0
   nr_bounce 0

vmstat-2.dump:

   nr_dirty 786397
   nr_writeback 44233
   nr_unstable 0
   nr_page_table_pages 640
   nr_mapped 6406
   nr_slab 53027
   pgpgin 263382
   pgpgout 7835732
   pswpin 0
   pswpout 0
   pgalloc_high 0
   pgalloc_normal 3091687
   pgalloc_dma 2420
   pgfree 3101327
   pgactivate 5817
   pgdeactivate 2918
   pgfault 466269
   pgmajfault 322
   pgrefill_high 0
   pgrefill_normal 28265
   pgrefill_dma 150
   pgsteal_high 0
   pgsteal_normal 789909
   pgsteal_dma 1388
   pgscan_kswapd_high 0
   pgscan_kswapd_normal 904101
   pgscan_kswapd_dma 4950
   pgscan_direct_high 0
   pgscan_direct_normal 0
   pgscan_direct_dma 0
   pginodesteal 0
   slabs_scanned 1152
   kswapd_steal 791297
   kswapd_inodesteal 0
   pageoutrun 28299
   allocstall 0
   pgrotated 562
   nr_bounce 0

zoneinfo.dump:

   Node 3, zone   Normal
     pages free     899
           min      726
           low      907
           high     1089
           active   3996
           inactive 490989
           scanned  0 (a: 16 i: 0)
           spanned  524287
           present  524287
           protection: (0, 0, 0)
     pagesets
       cpu: 0 pcp: 0
                 count: 2
                 low:   62
                 high:  186
                 batch: 31
       cpu: 0 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       10186
               numa_miss:      3313
               numa_foreign:   0
               interleave_hit: 10136
               local_node:     0
               other_node:     13499
       cpu: 1 pcp: 0
                 count: 13
                 low:   62
                 high:  186
                 batch: 31
       cpu: 1 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       6559
               numa_miss:      1668
               numa_foreign:   0
               interleave_hit: 6559
               local_node:     0
               other_node:     8227
       cpu: 2 pcp: 0
                 count: 84
                 low:   62
                 high:  186
                 batch: 31
       cpu: 2 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       5579
               numa_miss:      12806
               numa_foreign:   0
               interleave_hit: 5579
               local_node:     0
               other_node:     18385
       cpu: 3 pcp: 0
                 count: 93
                 low:   62
                 high:  186
                 batch: 31
       cpu: 3 pcp: 1
                 count: 55
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       834769
               numa_miss:      1
               numa_foreign:   940192
               interleave_hit: 5563
               local_node:     834770
               other_node:     0
     all_unreclaimable: 0
     prev_priority:     12
     temp_priority:     12
     start_pfn:         1572864
   Node 2, zone   Normal
     pages free     1036
           min      726
           low      907
           high     1089
           active   360
           inactive 501700
           scanned  0 (a: 26 i: 0)
           spanned  524287
           present  524287
           protection: (0, 0, 0)
     pagesets
       cpu: 1 pcp: 0
                 count: 91
                 low:   62
                 high:  186
                 batch: 31
       cpu: 1 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       6002
               numa_miss:      15490
               numa_foreign:   0
               interleave_hit: 6002
               local_node:     0
               other_node:     21492
       cpu: 2 pcp: 0
                 count: 75
                 low:   62
                 high:  186
                 batch: 31
       cpu: 2 pcp: 1
                 count: 56
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       410692
               numa_miss:      0
               numa_foreign:   76064
               interleave_hit: 5223
               local_node:     410692
               other_node:     0
       cpu: 3 pcp: 0
                 count: 73
                 low:   62
                 high:  186
                 batch: 31
       cpu: 3 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       5163
               numa_miss:      288909
               numa_foreign:   1
               interleave_hit: 5152
               local_node:     0
               other_node:     294072
     all_unreclaimable: 0
     prev_priority:     12
     temp_priority:     12
     start_pfn:         1048576
   Node 1, zone   Normal
     pages free     859
           min      703
           low      878
           high     1054
           active   1224
           inactive 485043
           scanned  0 (a: 14 i: 0)
           spanned  507903
           present  507760
           protection: (0, 0, 0)
     pagesets
       cpu: 0 pcp: 0
                 count: 1
                 low:   62
                 high:  186
                 batch: 31
       cpu: 0 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       9443
               numa_miss:      15475
               numa_foreign:   18446604437880297808
               interleave_hit: 18446604437880307200
               local_node:     1
               other_node:     24917
       cpu: 1 pcp: 0
                 count: 181
                 low:   62
                 high:  186
                 batch: 31
       cpu: 1 pcp: 1
                 count: 38
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       368191
               numa_miss:      0
               numa_foreign:   39046
               interleave_hit: 5967
               local_node:     368191
               other_node:     0
       cpu: 2 pcp: 0
                 count: 85
                 low:   62
                 high:  186
                 batch: 31
       cpu: 2 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       5139
               numa_miss:      18963
               numa_foreign:   0
               interleave_hit: 5139
               local_node:     0
               other_node:     24102
       cpu: 3 pcp: 0
                 count: 92
                 low:   62
                 high:  186
                 batch: 31
       cpu: 3 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       5124
               numa_miss:      363472
               numa_foreign:   0
               interleave_hit: 5115
               local_node:     0
               other_node:     368596
     all_unreclaimable: 0
     prev_priority:     12
     temp_priority:     12
     start_pfn:         524288
   Node 0, zone      DMA
     pages free     2045
           min      5
           low      6
           high     7
           active   0
           inactive 992
           scanned  0 (a: 2 i: 2)
           spanned  4096
           present  3994
           protection: (0, 2031, 2031)
     pagesets
       cpu: 0 pcp: 0
                 count: 1
                 low:   2
                 high:  6
                 batch: 1
       cpu: 0 pcp: 1
                 count: 1
                 low:   0
                 high:  2
                 batch: 1
               numa_hit:       18446604437880298786
               numa_miss:      18446604442220017848
               numa_foreign:   0
               interleave_hit: 0
               local_node:     7567460
               other_node:     0
     all_unreclaimable: 0
     prev_priority:     12
     temp_priority:     12
     start_pfn:         0
   Node 0, zone   Normal
     pages free     1052
           min      721
           low      901
           high     1081
           active   845
           inactive 480162
           scanned  0 (a: 2 i: 0)
           spanned  520191
           present  520191
           protection: (0, 0, 0)
     pagesets
       cpu: 0 pcp: 0
                 count: 96
                 low:   62
                 high:  186
                 batch: 31
       cpu: 0 pcp: 1
                 count: 50
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       18446604437880708763
               numa_miss:      18446604439958819000
               numa_foreign:   29590
               interleave_hit: 9679
               local_node:     7977309
               other_node:     0
       cpu: 1 pcp: 0
                 count: 88
                 low:   62
                 high:  186
                 batch: 31
       cpu: 1 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       6206
               numa_miss:      21831
               numa_foreign:   0
               interleave_hit: 6206
               local_node:     0
               other_node:     28037
       cpu: 2 pcp: 0
                 count: 65
                 low:   62
                 high:  186
                 batch: 31
       cpu: 2 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       5367
               numa_miss:      44135
               numa_foreign:   0
               interleave_hit: 5365
               local_node:     0
               other_node:     49502
       cpu: 3 pcp: 0
                 count: 92
                 low:   62
                 high:  186
                 batch: 31
       cpu: 3 pcp: 1
                 count: 0
                 low:   0
                 high:  62
                 batch: 31
               numa_hit:       5544
               numa_miss:      286378
               numa_foreign:   0
               interleave_hit: 5507
               local_node:     0
               other_node:     291922
     all_unreclaimable: 0
     prev_priority:     12
     temp_priority:     12
     start_pfn:         4096

meminfo.dump:

   MemTotal:      8124172 kB
   MemFree:         23564 kB
   Buffers:       7825944 kB
   Cached:          19216 kB
   SwapCached:          0 kB
   Active:          25708 kB
   Inactive:      7835548 kB
   HighTotal:           0 kB
   HighFree:            0 kB
   LowTotal:      8124172 kB
   LowFree:         23564 kB
   SwapTotal:    15631160 kB
   SwapFree:     15631160 kB
   Dirty:         3145604 kB
   Writeback:      176452 kB
   Mapped:          25624 kB
   Slab:           212116 kB
   CommitLimit:  19693244 kB
   Committed_AS:    85112 kB
   PageTables:       2560 kB
   VmallocTotal: 34359738367 kB
   VmallocUsed:     16288 kB
   VmallocChunk: 34359721635 kB

Thanks,
Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[Index of Archives]     [Kernel Newbies]     [Netfilter]     [Bugtraq]     [Photo]     [Stuff]     [Gimp]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Video 4 Linux]     [Linux for the blind]     [Linux Resources]
  Powered by Linux