On Thu, Jan 11, 2007 at 10:13:55AM +1100, Nick Piggin wrote:
> David Chinner wrote:
> >On Wed, Jan 10, 2007 at 03:04:15PM -0800, Christoph Lameter wrote:
> >
> >>On Thu, 11 Jan 2007, David Chinner wrote:
> >>
> >>
> >>>The performance and smoothness is fully restored on 2.6.20-rc3
> >>>by setting dirty_ratio down to 10 (from the default 40), so
> >>>something in the VM is not working as well as it used to....
> >>
> >>dirty_background_ratio is left as is at 10?
> >
> >
> >Yes.
> >
> >
> >>So you gain performance by switching off background writes via pdflush?
> >
> >
> >Well, pdflush appears to be doing very little on both 2.6.18 and
> >2.6.20-rc3. In both cases kswapd is consuming 10-20% of a CPU and
> >all of the pdflush threads combined (I've seen up to 7 active at
> >once) use maybe 1-2% of cpu time. This occurs regardless of the
> >dirty_ratio setting.
>
> Hi David,
>
> Could you get /proc/vmstat deltas for each kernel, to start with?
Sure, but that doesn't really show the how erratic the per-filesystem
throughput is because the test I'm running is PCI-X bus limited in
it's throughput at about 750MB/s. Each dm device is capable of about
340MB/s write, so when one slows down, the others will typically
speed up.
So, what I've attached is three files which have both
'vmstat 5' output and 'iostat 5 |grep dm-' output in them.
- 2.6.18.out - 2.6.18 behaviour near start of writes.
Behaviour does not change over the couse of the test,
just gets a bit slower as the test moves from the outer
edge of the disk to the inner. erractic behaviour is
highlighted.
- 2.6.20-rc3.out - 2.6.20-rc3 behaviour near start of writes.
Somewhat more erratic than 2.6.18, but about 100-150GB into
the write test, things change with dirty_ratio=40. erractic
behaviour is highlighted.
- 2.6.20-rc3-worse.out - 2.6.20-rc3 behavour when things go
bad. We're not keeping the disks or the PCI-X bus fully
utilised (each dm device can do about 300MB/s at this offset)
and aggregate throughput has dropped to 500-600MB/s.
With 2.6.20-rc3 and dirty_ratio = 10, the performance drop-off part way
into the test does not occur and the output is almost identical to
2.6.18.out.
> I'm guessing CPU time isn't a problem, but if it is then I guess
> profiles as well.
Plenty of idle cpu so I don't think it's a problem.
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
2.6.18-1-amd64 from debian unstable
/proc/sys/vm at defaults
$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 4 0 1807512 200 14052784 0 0 4 1638 76 41 0 1 99 0
7 2 0 91300 40 15726136 0 0 6 750452 2060 7153 1 53 13 33
3 3 0 89144 40 15719372 0 0 0 765470 2028 8306 0 57 16 27
3 3 0 90620 40 15737432 0 0 8 734020 2231 8449 1 55 16 28
1 4 0 93400 40 15732968 0 0 0 754329 2200 8208 0 55 15 30
1 4 0 94684 24 15737896 0 0 5 743548 2227 8862 1 56 14 29
2 1 0 87684 8 15828324 0 0 0 669015 2071 8571 0 46 27 27 <<<<<
3 3 0 91028 8 15735904 0 0 2 754430 2099 8252 1 46 30 24
2 4 0 89868 8 15778236 0 0 0 707134 2174 9668 0 47 24 29
2 8 0 88980 8 15740856 0 0 0 762258 2275 7690 1 54 15 30
3 7 0 92736 8 15734196 0 0 5 747985 2248 7365 0 56 17 26
3 6 0 90172 8 15744740 0 0 0 740884 2312 8348 0 57 15 28
2 7 0 89380 8 15741924 0 0 0 754114 2264 10184 0 57 15 28
3 5 0 94296 8 15807056 0 0 8 696285 2367 10537 0 54 14 32
2 6 0 88644 8 15791416 0 0 2 693826 2210 16981 0 44 19 37
2 7 0 93136 8 15793116 0 0 5 725302 2371 17737 1 52 14 34
3 10 0 87428 8 15753932 0 0 0 773513 2285 17724 0 56 7 37
3 8 0 90800 8 15753124 0 0 6 739142 2375 16656 1 58 10 31
4 8 0 93096 8 15754324 0 0 0 749338 2415 18957 0 60 10 30
2 11 0 86628 8 15760316 0 0 9 746439 2462 17413 1 60 11 29
3 5 0 87772 8 15829824 0 0 4 650377 2404 18220 0 49 16 35 <<<<<
1 6 0 90944 8 15803152 0 0 6 709598 2402 11936 1 48 13 38
7 7 0 89092 8 15783484 0 0 0 736173 2451 12959 0 51 14 35
1 11 0 85788 8 15752500 0 0 1 760675 2467 10508 1 54 11 34
4 9 0 88324 8 15748964 0 0 5 738161 2496 6939 0 57 13 30
3 8 0 91804 8 15751276 0 0 2 740564 2554 9801 1 59 12 29
2 9 0 91708 8 15761608 0 0 0 738580 2474 16851 0 60 11 28
4 8 0 87148 8 15774140 0 0 0 735378 2598 24892 0 60 9 30
5 9 0 87820 8 15767248 0 0 2 712053 2715 25149 0 56 13 31
2 9 0 94568 8 15756724 0 0 5 751153 2525 19390 1 60 8 31
3 8 0 88996 8 15764540 0 0 0 739513 2554 17517 0 60 12 28
4 9 0 88060 8 15770108 0 0 0 733779 2617 14532 1 59 10 31
1 11 0 84372 8 15767840 0 0 0 748831 2575 16348 0 60 12 28
4 10 0 85276 8 15769740 0 0 44 739195 2570 15844 1 60 10 30
3 10 0 90464 8 15757564 0 0 50 742243 2587 17061 0 58 9 32
3 6 0 92368 8 15781128 0 0 4 655346 2599 14666 1 48 18 33 <<<<<
3 7 0 94220 8 15812424 0 0 0 593861 2587 15113 0 38 31 31 <<<<<
8 8 0 90684 8 15786132 0 0 0 690888 2682 15278 1 47 15 38 <<<<<
1 10 0 89076 8 15754540 0 0 4 735318 2602 11575 0 55 10 36
2 9 0 91340 8 15752164 0 0 1 736788 2671 11438 0 59 14 26
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 9 0 89000 8 15756816 0 0 0 746851 2543 10281 0 60 15 25
3 10 0 89420 8 15753108 0 0 1 745210 2619 12884 1 61 13 25
1 11 0 84588 8 15762012 0 0 2 695561 2718 13985 0 54 10 36
3 9 0 87012 8 15759400 0 0 10 728164 2722 11159 1 59 10 30
2 9 0 91472 8 15757548 0 0 6 723808 2669 13573 0 57 13 29
4 8 0 89160 8 15759376 0 0 0 742918 2661 15718 1 60 12 27
2 10 0 85084 8 15771616 0 0 0 724117 2671 18962 0 61 9 30
4 9 0 92472 8 15771296 0 0 1 725714 2811 24185 1 60 12 28
2 10 0 92068 8 15773208 0 0 4 734559 2726 27220 0 60 11 29
2 11 0 85592 8 15773564 0 0 2 737879 2714 27623 1 60 11 28
3 10 0 91700 8 15766072 0 0 2 718346 2646 22216 0 56 15 29
3 10 0 83776 8 15775852 0 0 0 727938 2753 23132 1 61 10 29
3 8 0 93516 8 15771240 0 0 0 734910 2711 23445 0 62 12 26
2 9 0 88968 8 15776408 0 0 5 732068 2734 23289 1 62 11 26
$ iostat 5 |grep dm-
dm-0 226.47 3.28 6213.88 11668 22084021
dm-1 216.82 3.29 5681.85 11692 20193189
dm-2 230.18 3.29 5762.51 11692 20479829
dm-0 31150.80 0.00 498411.24 0 2482088
dm-1 29964.66 0.00 479434.54 0 2387584
dm-2 32157.23 0.00 514515.66 0 2562288
dm-0 31355.53 0.00 501688.53 0 2493392
dm-1 30267.40 0.00 484276.86 0 2406856
dm-2 32690.14 0.00 523042.25 0 2599520
dm-0 30640.85 0.00 490329.18 0 2436936
dm-1 31119.32 0.00 497986.32 0 2474992
dm-2 31146.88 0.00 498401.61 0 2477056
dm-0 11263.65 0.00 361658.43 0 1801059 <<<<<
dm-1 23827.51 0.00 543884.94 0 2708547 <<<<<
dm-2 15443.37 0.00 376792.57 0 1876427 <<<<<
dm-0 24433.06 0.00 488340.73 0 2422170
dm-1 30285.08 0.00 542182.26 0 2689224
dm-2 14798.59 0.00 469650.00 0 2329464
dm-0 23780.96 0.00 566244.49 0 2825560 <<<<<
dm-1 31165.93 0.00 560659.12 0 2797689 <<<<<
dm-2 12451.50 0.00 440997.60 0 2200578 <<<<<
dm-0 29361.77 0.00 469781.89 0 2334816
dm-1 31076.26 0.00 497215.29 0 2471160
dm-2 31230.18 0.00 499679.68 0 2483408
dm-0 32040.32 0.00 512641.94 0 2542704
dm-1 31585.69 0.00 505369.35 0 2506632
dm-2 30485.08 0.00 487758.06 0 2419280
dm-0 31143.95 0.00 498300.00 0 2471568
dm-1 32371.37 0.00 517941.94 0 2568992
dm-2 30724.60 0.00 491593.55 0 2438304
dm-0 30964.92 0.00 495433.87 0 2457352
dm-1 31821.17 0.00 509137.10 0 2525320
dm-2 31084.88 0.00 497353.23 0 2466872
dm-0 22213.86 0.00 431683.94 0 2149786 <<<<<
dm-1 19303.82 3.21 406148.19 16 2022618 <<<<<
dm-2 24112.85 0.00 481736.55 0 2399048 <<<<<
dm-0 32586.52 0.00 521364.99 0 2591184
dm-1 26891.15 0.00 533442.25 0 2651208
dm-2 30924.35 0.00 494713.88 0 2458728
dm-0 32124.95 0.00 513909.94 0 2533576
dm-1 31073.63 0.00 497169.98 0 2451048
dm-2 30958.42 0.00 495117.24 0 2440928
dm-0 31779.44 0.00 508403.23 0 2521680
dm-1 30971.57 0.00 495524.19 0 2457800
dm-2 31085.08 0.00 497133.87 0 2465784
dm-0 32254.03 0.00 515925.81 0 2558992
dm-1 30127.62 0.00 481991.94 0 2390680
dm-2 32006.05 0.00 512008.06 0 2539560
dm-0 22819.92 4.83 526683.70 24 2617618 <<<<<
dm-1 15316.30 4.83 411950.50 24 2047394 <<<<<
dm-2 16564.59 8.05 434072.03 40 2157338 <<<<<
dm-0 18948.69 3.23 481028.28 16 2381090
dm-1 19329.90 0.00 445191.52 0 2203698
dm-2 24194.75 0.00 454280.81 0 2248690
dm-0 38064.65 0.00 608997.17 0 3014536 <<<<<
dm-1 35084.65 0.00 561273.54 0 2778304 <<<<<
dm-2 723.84 0.00 321928.08 0 1593544 <<<<<
dm-0 31022.29 0.00 496204.02 0 2471096
dm-1 30877.11 0.00 493871.49 0 2459480
dm-2 23065.26 0.00 516738.96 0 2573360
dm-0 31469.90 0.00 503505.45 0 2492352
dm-1 29785.86 0.00 476305.45 0 2357712
dm-2 32217.17 0.00 515424.65 0 2551352
dm-0 31458.87 0.00 503259.68 0 2496168
dm-1 29787.70 0.00 476203.23 0 2361968
dm-2 31912.50 0.00 510553.23 0 2532344
dm-0 31722.78 0.00 507559.68 0 2517496
dm-1 31753.02 0.00 507954.84 0 2519456
dm-2 30015.52 0.00 479920.97 0 2380408
dm-0 31761.13 0.00 508152.63 0 2510274
dm-1 29887.25 0.00 477944.94 0 2361048
dm-2 30137.85 0.00 482254.25 0 2382336
dm-0 21502.83 0.00 488791.92 0 2419520
dm-1 30429.49 0.00 486623.03 0 2408784
dm-2 30606.87 0.00 489622.63 0 2423632
dm-0 32404.22 0.00 518364.66 0 2581456
dm-1 29427.91 0.00 470522.09 0 2343200
dm-2 31702.01 0.00 507203.21 0 2525872
dm-0 31864.04 0.00 509805.25 0 2523536
dm-1 31660.20 0.00 506454.95 0 2506952
dm-2 29938.99 0.00 478516.36 0 2368656
dm-0 31114.75 0.00 497692.12 0 2463576
dm-1 30767.07 0.00 492061.41 0 2435704
dm-2 31053.94 0.00 496738.59 0 2458856
dm-0 32305.04 0.00 516856.45 0 2563608
dm-1 31119.96 0.00 497622.58 0 2468208
dm-2 30513.10 0.00 487833.87 0 2419656
dm-0 32361.62 0.00 517675.96 0 2562496
dm-1 29420.61 0.00 470370.91 0 2328336
dm-2 31926.06 0.00 510792.73 0 2528424
dm-0 30932.39 0.00 494989.47 0 2445248
dm-1 32021.46 0.00 512369.23 0 2531104
dm-2 30555.67 0.00 488898.79 0 2415160
dm-0 25578.14 0.00 529783.00 0 2617128
dm-1 30213.97 0.00 483229.15 0 2387152
dm-2 31086.64 0.00 506953.85 0 2504352
dm-0 30769.76 0.00 492275.81 0 2441688
dm-1 31664.31 0.00 506608.06 0 2512776
dm-2 30774.19 0.00 492314.52 0 2441880
dm-0 31153.72 0.00 498317.91 0 2476640
dm-1 30733.40 0.00 491517.10 0 2442840
dm-2 31614.29 0.00 505740.04 0 2513528
dm-0 31665.52 0.00 506700.00 0 2513232
dm-1 31119.35 0.00 497848.39 0 2469328
dm-2 30385.08 0.00 485975.81 0 2410440
dm-0 24149.60 4.86 464011.74 24 2292218
dm-1 21799.19 0.00 463535.63 0 2289866
dm-2 28986.84 0.00 484500.40 0 2393432
dm-0 28882.66 0.00 461990.32 0 2291472
dm-1 29911.49 1.61 478564.52 8 2373680
dm-2 29888.91 0.00 487735.48 0 2419168
dm-0 30865.73 0.00 493837.10 0 2449432
dm-1 28708.87 3.23 503904.84 16 2499368
dm-2 30902.22 4.84 504001.61 24 2499848
dm-0 31659.27 0.00 506483.87 0 2512160
dm-1 31373.39 0.00 501814.52 0 2489000
dm-2 30631.25 0.00 489846.77 0 2429640
dm-0 29666.80 0.00 474531.99 0 2358424
dm-1 30355.33 0.00 485514.69 0 2413008
dm-2 30551.31 0.00 488505.43 0 2427872
dm-0 29918.15 0.00 478454.84 0 2373136
dm-1 30949.80 0.00 495179.03 0 2456088
dm-2 31393.15 0.00 502204.84 0 2490936
dm-0 30278.38 0.00 484286.06 0 2397216
dm-1 31032.73 0.00 496501.01 0 2457680
dm-2 31428.08 0.00 502841.21 0 2489064
dm-0 30813.08 0.00 493051.11 0 2450464 <<<<<
dm-1 31612.88 0.00 505817.51 0 2513913 <<<<<
dm-2 27513.28 3.22 440259.76 16 2188091 <<<<<
dm-0 31584.85 0.00 505252.53 0 2501000
dm-1 32325.86 0.00 517141.01 0 2559848
dm-2 20020.20 0.00 464043.64 0 2297016
dm-0 30549.19 0.00 488669.35 0 2423800
dm-1 31501.21 0.00 503943.55 0 2499560
dm-2 30221.17 0.00 483325.81 0 2397296
dm-0 29849.60 0.00 477311.29 0 2367464
dm-1 31148.79 0.00 498250.00 0 2471320
dm-2 30749.40 0.00 496754.84 0 2463904
dm-0 31116.30 0.00 497820.52 0 2474168
dm-1 30522.13 0.00 488333.20 0 2427016
dm-2 30494.57 0.00 487793.96 0 2424336
2.6.20-rc3, default vm tunings
$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 5 0 87864 8 15723492 0 0 48 64452 253 908 0 18 72 9
3 6 0 87520 8 15726764 0 0 5 747371 2295 15063 0 57 19 24
3 6 0 91612 8 15726768 0 0 0 743760 2343 15321 1 57 17 25
4 5 0 87912 8 15744920 0 0 2 727226 2312 13431 0 57 20 24
1 7 0 87848 8 15798020 0 0 1 495273 1878 7874 1 24 61 14 <<<<<<<
2 10 0 89364 8 15756660 0 0 0 681229 2401 24021 0 54 20 26
2 10 0 93892 8 15749864 0 0 5 736260 2514 26324 1 59 11 30
5 8 0 88716 8 15749160 0 0 0 751747 2389 27372 0 59 11 29
2 10 0 90880 8 15743488 0 0 0 746821 2384 26088 1 59 9 32
3 9 0 91520 8 15747588 0 0 0 741946 2345 28706 0 60 13 27
4 10 0 86928 8 15756244 0 0 4 744362 2425 28480 1 61 14 25
3 8 0 91932 8 15757916 0 0 9 590591 2141 14653 0 36 31 33 <<<<<<<
3 9 0 88180 8 15778396 0 0 2 534214 2311 20443 1 37 26 37 <<<<<<<
2 9 0 88976 8 15762576 0 0 2 717699 2516 30072 0 56 17 27
7 8 0 92892 8 15757448 0 0 0 747566 2485 32363 1 61 12 27
1 10 0 86364 8 15759024 0 0 0 749395 2453 30159 0 60 13 27
5 8 0 92860 8 15752016 0 0 6 745046 2462 29773 1 60 11 28
4 10 0 91700 8 15754044 0 0 0 745897 2422 31167 0 60 12 28
2 9 0 93368 8 15773256 0 0 0 636481 2736 29868 1 52 7 41 <<<<<<<
0 11 0 92920 8 15752060 0 0 0 737382 2639 32313 0 58 13 28
1 10 0 87772 8 15759076 0 0 0 742549 2478 30909 1 61 13 26
3 8 0 91632 8 15759116 0 0 5 742481 2464 31315 0 61 13 27
2 10 0 85992 8 15760688 0 0 14 740988 2639 31975 1 61 9 30
3 8 0 92352 8 15751444 0 0 0 742023 2557 29505 0 61 12 27
3 8 0 90332 8 15747752 0 0 0 748272 2537 27328 1 60 18 21
4 8 0 92128 8 15769772 0 0 0 677031 2614 27487 0 56 13 31 <<<<<<<
2 9 0 92028 8 15760824 0 0 5 724803 2652 27075 1 57 10 32
5 8 0 86600 8 15763456 0 0 0 739023 2557 31041 0 61 15 24
2 9 0 93508 8 15751448 0 0 1 747915 2629 28523 1 62 11 26
5 9 0 90224 8 15755416 0 0 0 744098 2509 27222 0 63 10 27
1 11 0 91948 8 15758920 0 0 0 744949 2561 27062 1 61 11 28
1 10 0 89108 8 15755400 0 0 5 749241 2553 28086 0 60 12 28
12 8 0 90292 8 15778824 0 0 2 690422 2696 28210 1 56 16 27 <<<<<<<
2 9 0 94316 8 15762820 0 0 2 621403 2741 26233 0 47 22 31 <<<<<<<
0 11 0 91864 8 15759612 0 0 1 734336 2671 30539 1 59 14 26
4 9 0 88592 8 15755156 0 0 2 743448 2600 27277 0 61 16 24
3 8 0 92236 8 15746248 0 0 7 740407 2643 26521 1 61 17 21
3 9 0 87024 8 15755208 0 0 0 748968 2587 31585 0 63 12 25
6 8 0 91032 8 15748884 0 0 0 745108 2569 27812 1 62 10 28
2 8 0 89656 8 15776280 0 0 2 697020 2639 26038 0 57 12 30
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 8 0 91660 8 15770328 0 0 1 722922 2719 28936 1 56 15 28
2 10 0 89032 8 15779088 0 0 5 679131 2871 29320 0 55 20 24 <<<<<<<
4 8 0 87044 8 15777288 0 0 0 718871 2747 30824 1 59 19 21
3 9 0 89644 8 15770644 0 0 0 746693 2574 32736 0 64 9 27
4 8 0 92260 8 15773276 0 0 0 742171 2631 33458 1 66 7 27
1 11 0 87176 8 15772488 0 0 0 746909 2627 34393 0 66 13 21
1 7 0 87696 8 15792288 0 0 6 711137 2683 31292 0 61 12 26
2 7 0 92700 8 15778184 0 0 0 616361 2652 23023 0 48 23 29 <<<<<<<
3 10 0 84792 8 15752604 0 0 0 750780 2683 24771 0 63 11 26
5 9 0 92944 8 15743256 0 0 0 749332 2681 25124 0 64 17 19
1 10 0 94048 8 15741900 0 0 0 735865 2675 29609 0 65 9 26
9 9 0 90672 8 15740272 0 0 5 747928 2688 25178 0 66 8 26
3 8 0 87356 8 15749596 0 0 1 730961 2718 26207 1 64 16 20
2 9 0 88708 8 15750372 0 0 0 740069 2649 25111 0 65 16 19
2 10 0 83480 8 15770688 0 0 2 628456 2845 23485 1 52 22 25 <<<<<<<
3 8 0 92748 8 15751520 0 0 0 733477 2788 33597 0 64 12 24
2 9 0 94188 8 15749032 0 0 5 740086 2724 34426 1 66 13 20
3 9 0 89168 8 15750168 0 0 0 738250 2723 31540 0 65 16 19
4 8 0 89388 8 15758480 0 0 3 706912 3034 32462 1 63 7 29
3 8 0 88364 8 15759132 0 0 2 738317 2745 34635 0 66 9 26
5 9 0 89300 8 15755912 0 0 5 737039 2824 33647 1 65 7 27
2 8 0 86640 8 15765904 0 0 0 622271 3029 24950 0 52 17 30 <<<<<<<
3 9 0 87144 8 15758820 0 0 0 694992 3123 26889 1 62 7 31
1 10 0 86968 8 15756064 0 0 0 736023 2867 24487 0 65 3 32
11 9 0 89468 8 15756400 0 0 6 721520 2904 23735 1 64 3 32
9 9 0 91564 8 15756660 0 0 1 710507 3029 25829 0 64 3 33
2 9 0 88196 8 15754908 0 0 0 724214 3002 26844 1 66 2 31
$ iostat 5 |grep dm-
dm-0 9134.66 11.56 166305.84 1903 27380594
dm-1 8620.94 11.56 158204.25 1903 26046748
dm-2 8916.15 11.56 165543.95 1903 27255156
dm-0 34293.75 0.00 548691.07 0 2458136
dm-1 35079.02 0.00 561233.93 0 2514328
dm-2 35228.79 0.00 563646.43 0 2525136
dm-0 34237.64 0.00 547802.23 0 2459632
dm-1 34295.55 0.00 548728.73 0 2463792
dm-2 34899.78 0.00 558392.87 0 2507184
dm-0 34518.30 0.00 552292.86 0 2474272
dm-1 34491.07 0.00 551850.00 0 2472288
dm-2 34769.20 0.00 556307.14 0 2492256
dm-0 33206.70 3.57 531393.30 16 2380642
dm-1 33909.38 0.00 542646.88 0 2431058
dm-2 35174.55 0.00 562889.73 0 2521746
dm-0 3334.30 0.00 299309.64 0 1334921 <<<<<<<
dm-1 3046.64 0.00 326809.19 0 1457569 <<<<<<<
dm-2 3493.50 0.00 448256.73 0 1999225 <<<<<<<
dm-0 19059.56 0.00 325194.67 0 1463376 <<<<<<<
dm-1 31279.11 0.00 561379.56 0 2526208 <<<<<<<
dm-2 35930.44 0.00 639664.00 0 2878488 <<<<<<<
dm-0 22602.02 0.00 361628.70 0 1612864 <<<<<<<
dm-1 40571.75 0.00 649147.98 0 2895200 <<<<<<<
dm-2 40258.30 0.00 644127.35 0 2872808 <<<<<<<
dm-0 35988.57 0.00 575817.04 0 2568144
dm-1 34716.37 0.00 555461.88 0 2477360
dm-2 34521.08 0.00 552337.22 0 2463424
dm-0 36909.23 0.00 590545.95 0 2622024
dm-1 34323.20 0.00 549171.17 0 2438320
dm-2 34153.60 0.00 546344.14 0 2425768
dm-0 34906.67 0.00 558506.67 0 2513280
dm-1 33709.56 0.00 539349.33 0 2427072
dm-2 34209.56 0.00 547287.11 0 2462792
dm-0 35394.17 0.00 566410.76 0 2526192
dm-1 33390.36 0.00 534306.73 0 2383008
dm-2 35356.05 0.00 565808.07 0 2523504
dm-0 16723.60 0.00 469194.61 0 2087916 <<<<<<<
dm-1 9435.96 0.00 366801.57 0 1632267 <<<<<<<
dm-2 21639.10 5.39 508268.31 24 2261794 <<<<<<<
dm-0 11498.67 0.00 315091.56 0 1417912 <<<<<<<
dm-1 21372.22 5.33 475187.56 24 2138344 <<<<<<<
dm-2 15014.89 0.00 394926.67 0 1777170 <<<<<<<
dm-0 33084.34 5.37 529331.54 24 2366112 <<<<<<<
dm-1 41148.10 0.00 658367.79 0 2942904 <<<<<<<
dm-2 25574.72 0.00 409111.41 0 1828728 <<<<<<<
dm-0 34765.18 0.00 556241.07 0 2491960
dm-1 34749.78 0.00 555996.43 0 2490864
dm-2 34273.21 0.00 548369.64 0 2456696
dm-0 35772.10 0.00 572341.07 0 2564088
dm-1 33792.86 0.00 540530.36 0 2421576
dm-2 35690.40 0.00 571044.64 0 2558280
dm-0 34965.55 0.00 559429.08 0 2500648
dm-1 35015.21 0.00 560243.40 0 2504288
dm-2 34003.80 0.00 543958.84 0 2431496
dm-0 34828.79 0.00 557366.07 0 2497000
dm-1 35144.42 0.00 562417.86 0 2519632
dm-2 33988.39 0.00 543769.64 0 2436088
dm-0 25764.04 0.00 412182.02 0 1834210 <<<<<<<
dm-1 22946.29 0.00 367083.60 0 1633522 <<<<<<<
dm-2 39979.10 0.00 653973.93 0 2910184 <<<<<<<
dm-0 37178.92 0.00 594832.29 0 2652952 <<<<<<<
dm-1 24420.63 0.00 390550.67 0 1741856 <<<<<<<
dm-2 40920.18 0.00 654695.96 0 2919944 <<<<<<<
dm-0 35388.37 0.00 566144.07 0 2530664
dm-1 35114.09 0.00 561782.55 0 2511168
dm-2 33981.66 0.00 543676.06 0 2430232
dm-0 35202.01 0.00 563228.64 0 2517632
dm-1 34419.69 0.00 550713.20 0 2461688
dm-2 34196.64 0.00 547112.30 0 2445592
dm-0 34176.73 0.00 546731.10 0 2443888
dm-1 32948.99 0.00 526990.60 0 2355648
dm-2 36352.57 0.00 581617.90 0 2599832
dm-0 35192.86 0.00 562916.07 0 2521864
dm-1 34445.31 0.00 551071.43 0 2468800
dm-2 34110.71 0.00 545592.86 0 2444256
dm-0 34828.51 0.00 557222.27 0 2501928
dm-1 35213.81 0.00 563376.39 0 2529560
dm-2 34087.31 0.00 545363.03 0 2448680
dm-0 28085.14 0.00 449459.91 0 1995602 <<<<<<<
dm-1 28701.80 0.00 459330.18 0 2039426 <<<<<<<
dm-2 38110.81 0.00 616873.87 0 2738920 <<<<<<<
dm-0 29772.26 0.00 484220.13 0 2164464 <<<<<<<
dm-1 31771.14 0.00 508089.49 0 2271160 <<<<<<<
dm-2 38417.67 0.00 614672.04 0 2747584 <<<<<<<
dm-0 34103.35 0.00 545508.93 0 2443880
dm-1 34408.48 0.00 550446.43 0 2466000
dm-2 34469.20 0.00 563789.29 0 2525776
dm-0 34283.00 0.00 548506.49 0 2451824
dm-1 35376.06 0.00 565993.74 0 2529992
dm-2 34725.95 0.00 555441.61 0 2482824
dm-0 35205.82 0.00 563291.28 0 2517912
dm-1 35118.79 0.00 561898.88 0 2511688
dm-2 33814.32 0.00 540970.02 0 2418136
dm-0 36445.17 0.00 583122.70 0 2594896
dm-1 33097.98 0.00 529398.65 0 2355824
dm-2 34954.38 0.00 559264.72 0 2488728
dm-0 35248.10 0.00 563795.97 0 2520168
dm-1 34957.49 0.00 559200.00 0 2499624
dm-2 34368.68 0.00 549877.40 0 2457952
dm-0 36696.19 0.00 592000.00 0 2640320 <<<<<<<
dm-1 30397.98 0.00 486409.42 0 2169386 <<<<<<<
dm-2 29955.38 3.59 489987.89 16 2185346 <<<<<<<
dm-0 41788.39 0.00 688048.21 0 3082456 <<<<<<<
dm-1 20116.96 0.00 321871.43 0 1441984 <<<<<<<
dm-2 21677.68 5.36 358007.14 24 1603872 <<<<<<<
dm-0 37506.70 0.00 600107.14 0 2688480 <<<<<<<
dm-1 29063.62 0.00 465016.07 0 2083272 <<<<<<<
dm-2 36207.59 0.00 579308.93 0 2595304 <<<<<<<
dm-0 32667.34 5.37 560195.08 24 2504072
dm-1 32984.79 0.00 527293.06 0 2357000
dm-2 35434.90 0.00 566949.44 0 2534264
dm-0 34265.26 0.00 548238.75 0 2461592
dm-1 32824.28 5.35 541412.92 24 2430944
dm-2 35598.66 0.00 569541.20 0 2557240
dm-0 34853.81 0.00 557653.81 0 2487136
dm-1 34941.70 0.00 559065.47 0 2493432
dm-2 34413.45 0.00 550615.25 0 2455744
dm-0 34828.19 0.00 557251.01 0 2490912
dm-1 35065.32 0.00 561045.19 0 2507872
dm-2 34574.05 0.00 553141.83 0 2472544
dm-0 31283.63 3.59 500597.98 16 2232667
dm-1 32622.20 0.00 521850.00 0 2327451
dm-2 35009.19 0.00 560244.39 0 2498690
dm-0 12572.81 0.00 324825.17 0 1445472 <<<<<<<
dm-1 39278.88 0.00 628296.63 0 2795920 <<<<<<<
dm-2 39375.51 0.00 653073.26 0 2906176 <<<<<<<
dm-0 19262.36 0.00 308197.77 0 1383808 <<<<<<<
dm-1 38768.37 0.00 620293.99 0 2785120 <<<<<<<
dm-2 36592.65 0.00 585464.59 0 2628736 <<<<<<<
dm-0 25140.53 0.00 402246.77 0 1806088 <<<<<<<
dm-1 37817.15 0.00 605074.39 0 2716784 <<<<<<<
dm-2 36809.80 0.00 588898.00 0 2644152 <<<<<<<
dm-0 34272.20 0.00 548353.36 0 2445656
dm-1 34775.78 0.00 556362.33 0 2481376
dm-2 35434.08 0.00 566945.29 0 2528576
dm-0 35166.22 0.00 562657.72 0 2515080
dm-1 33327.96 0.00 538122.60 0 2405408
dm-2 34748.99 0.00 555978.52 0 2485224
dm-0 34530.51 0.00 552488.20 0 2480672
dm-1 34470.16 0.00 551520.71 0 2476328
dm-2 34854.12 0.00 562897.10 0 2527408
dm-0 29927.23 0.00 494966.74 0 2217451
dm-1 33754.24 0.00 568400.00 0 2546432
dm-2 31355.58 0.00 536487.95 0 2403466
dm-0 19824.55 0.00 481598.21 0 2157560 <<<<<<<
dm-1 6611.16 0.00 353630.80 0 1584266 <<<<<<<
dm-2 29875.89 0.00 532652.01 0 2386281 <<<<<<<
dm-0 36691.46 0.00 587052.58 0 2612384
dm-1 25364.94 0.00 513862.47 0 2286688
dm-2 36197.08 0.00 593362.70 0 2640464
dm-0 34072.16 0.00 545147.44 0 2447712
dm-1 34371.49 0.00 549910.02 0 2469096
dm-2 35014.92 0.00 566635.19 0 2544192
dm-0 32494.20 0.00 519842.86 0 2328896
dm-1 33713.17 0.00 549982.14 0 2463920
dm-2 33254.46 0.00 565432.14 0 2533136
dm-0 34066.14 0.00 544938.12 0 2430424
dm-1 35761.43 0.00 572182.96 0 2551936
dm-2 35004.71 0.00 560075.34 0 2497936
dm-0 34033.85 0.00 544541.65 0 2444992
dm-1 33596.21 0.00 537507.35 0 2413408
dm-2 32896.44 0.00 546198.66 0 2452432
dm-0 33809.78 0.00 541038.22 0 2434672
dm-1 33782.00 0.00 540574.22 0 2432584
dm-2 35588.67 0.00 578160.00 0 2601720
dm-0 22325.67 0.00 453068.53 0 2029747 <<<<<<<
dm-1 20525.22 0.00 431493.53 0 1933091 <<<<<<<
dm-2 24213.84 5.36 524191.74 24 2348379 <<<<<<<
dm-0 32452.56 0.00 519068.15 0 2330616
dm-1 32655.90 0.00 527748.78 0 2369592
dm-2 32908.24 0.00 564297.55 0 2533696
dm-0 34977.73 0.00 559643.65 0 2512800
dm-1 34287.31 0.00 548564.81 0 2463056
dm-2 33959.02 0.00 548579.06 0 2463120
dm-0 34490.65 0.00 551816.48 0 2477656
dm-1 34505.57 0.00 552064.14 0 2478768
dm-2 33488.42 0.00 541060.13 0 2429360
dm-0 31625.17 5.35 519837.86 24 2334072 <<<<<<<
dm-1 28918.26 0.00 462214.70 0 2075344 <<<<<<<
dm-2 34211.14 0.00 576848.11 0 2590048 <<<<<<<
dm-0 33163.68 0.00 530378.48 0 2365488
dm-1 34059.42 5.38 558333.63 24 2490168
dm-2 35936.77 0.00 574984.75 0 2564432
dm-0 34723.54 0.00 564419.73 0 2517312
dm-1 34305.38 0.00 557705.83 0 2487368
dm-2 33052.69 0.00 528747.98 0 2358216
dm-0 18900.00 0.00 401263.74 0 1781611 <<<<<<<
dm-1 24734.46 0.00 429701.58 0 1907875 <<<<<<<
dm-2 32280.63 0.00 573387.39 0 2545840 <<<<<<<
dm-0 28238.53 0.00 451561.69 0 2027512 <<<<<<<
dm-1 31791.54 0.00 514277.06 0 2309104 <<<<<<<
dm-2 36535.41 0.00 584552.34 0 2624640 <<<<<<<
dm-0 34163.47 0.00 546478.40 0 2453688
dm-1 32867.04 0.00 525512.69 0 2359552
dm-2 34826.95 0.00 562485.52 0 2525560
dm-0 34127.17 0.00 546032.96 0 2451688
dm-1 34187.75 0.00 546984.41 0 2455960
dm-2 32145.66 0.00 514088.20 0 2308256
dm-0 34534.74 0.00 555887.75 0 2495936
dm-1 29349.00 0.00 469056.57 0 2106064
dm-2 33997.10 0.00 557651.67 0 2503856
dm-0 33370.67 0.00 533662.22 0 2401480
dm-1 32164.22 0.00 514227.56 0 2314024
dm-2 35266.00 0.00 569495.11 0 2562728
After some run time (about 100GB in), 2.6.18 continues
to write at close to 250MB/s per filesystem (as per original
trace). 2.6.20-rc3 with dirty_ratio = 40 does:
$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 7 0 90540 8 15796472 0 0 10 164413 734 6124 0 51 23 25
2 7 0 91060 8 15773864 0 0 0 558089 3780 27825 1 48 12 40
8 8 0 91808 8 15764172 0 0 0 692855 3343 25813 0 63 8 29
4 9 0 86604 8 15770876 0 0 6 670767 3605 32657 1 63 8 28
1 11 0 83840 8 15775184 0 0 0 666833 3398 31701 0 64 3 33
3 9 0 82300 8 15776156 0 0 2 616678 3939 37617 1 58 10 32
11 6 0 90640 8 15771528 0 0 2 696310 3395 38654 0 67 5 28
5 9 0 92412 8 15771872 0 0 2 585962 3453 32651 1 54 14 32
5 9 0 90452 8 15766020 0 0 5 583086 3670 32937 0 55 7 38
2 9 0 87980 8 15763964 0 0 1 681694 3525 30906 1 64 9 26
3 8 0 86664 8 15761044 0 0 2 650998 3677 25797 0 61 6 32
6 8 0 86804 8 15759052 0 0 0 715986 3317 25729 1 69 11 20
3 10 0 84240 8 15765944 0 0 0 673761 3528 25241 0 64 7 29
7 10 0 89916 8 15763700 0 0 5 664583 3506 27555 1 63 7 30
1 8 0 82896 8 15814824 0 0 2 519892 3456 24579 0 49 23 29
3 7 0 88276 8 15809844 0 0 1 415687 3093 15203 0 29 50 20
3 8 0 88324 8 15778956 0 0 0 589647 3370 30624 0 52 13 35
2 8 0 91776 8 15775764 0 0 0 609689 3506 32780 0 56 11 33
3 9 0 85604 8 15778512 0 0 0 656673 3573 37672 0 64 5 31
3 10 0 85676 8 15785708 0 0 5 644915 3746 39503 1 63 6 31
5 8 0 89700 8 15775952 0 0 2 680001 3510 38927 0 65 8 27
0 8 0 82332 8 15814516 0 0 1 638288 3499 36999 0 63 6 30
3 8 0 88748 8 15767756 0 0 0 535037 3797 26778 0 44 9 47
8 9 0 86620 8 15773060 0 0 0 581291 3556 27902 1 54 9 37
3 11 0 93708 8 15759696 0 0 6 647017 3714 33242 0 61 6 32
3 9 0 90172 8 15759952 0 0 0 716961 3345 35311 1 70 7 23
3 9 0 90040 8 15763172 0 0 0 655000 3652 31259 0 61 11 27
4 8 0 87176 8 15764384 0 0 1 643515 3668 32440 0 62 5 32
1 10 0 83812 8 15786040 0 0 0 650601 3605 33809 0 64 8 28
5 9 0 82760 8 15805612 0 0 5 440437 3467 19922 0 35 31 33
$ iostat 5 |grep dm-
dm-0 25998.50 2.57 433094.50 2207 371924231
dm-1 26090.07 2.48 437352.37 2127 375580723
dm-2 26555.31 2.52 451131.47 2167 387413658
dm-0 4710.74 0.00 244624.61 0 1093472
dm-1 10662.86 0.00 293821.92 0 1313384
dm-2 13400.89 3.58 424025.06 16 1895392
dm-0 5820.13 0.00 280354.36 0 1253184
dm-1 31697.99 0.00 506851.01 0 2265624
dm-2 32324.83 0.00 544966.44 0 2436000
dm-0 31331.17 0.00 501296.86 0 2235784
dm-1 34606.50 0.00 553700.45 0 2469504
dm-2 32436.10 0.00 518557.85 0 2312768
dm-0 30406.47 0.00 490957.14 0 2199488
dm-1 29177.46 0.00 466635.71 0 2090528
dm-2 32097.77 0.00 518639.29 0 2323504
dm-0 29419.55 0.00 470106.97 0 2091976
dm-1 32272.36 0.00 526919.55 0 2344792
dm-2 30346.07 0.00 484979.78 0 2158160
dm-0 30287.75 0.00 484119.38 0 2173696
dm-1 25081.74 5.35 412530.96 24 1852264
dm-2 31800.22 0.00 515681.07 0 2315408
dm-0 32531.91 5.39 520604.04 24 2316688
dm-1 32934.83 0.00 527065.17 0 2345440
dm-2 32717.75 0.00 523541.57 0 2329760
dm-0 29717.45 0.00 511937.58 0 2288361
dm-1 16489.26 3.58 408936.69 16 1827947
dm-2 17686.58 0.00 321441.16 0 1436842
dm-0 29951.89 0.00 510305.57 0 2291272
dm-1 28271.27 0.00 451946.55 0 2029240
dm-2 25195.55 0.00 403021.83 0 1809568
dm-0 29194.63 0.00 473558.84 0 2116808
dm-1 31468.68 0.00 506738.26 0 2265120
dm-2 31061.07 0.00 496748.10 0 2220464
dm-0 32711.66 0.00 523320.18 0 2334008
dm-1 29713.90 0.00 478310.31 0 2133264
dm-2 32273.32 0.00 531892.38 0 2372240
dm-0 32079.87 0.00 513091.72 0 2293520
dm-1 33290.38 0.00 532594.18 0 2380696
dm-2 32902.46 0.00 526344.52 0 2352760
dm-0 31768.23 0.00 507946.31 0 2270520
dm-1 27854.14 0.00 445063.09 0 1989432
dm-2 31099.11 0.00 497422.82 0 2223480
dm-0 31845.78 0.00 512867.56 0 2307904
dm-1 30570.67 0.00 495665.78 0 2230496
dm-2 29890.89 0.00 502348.44 0 2260568
dm-0 14096.41 3.59 356843.72 16 1591523
dm-1 13003.14 0.00 336176.46 0 1499347
dm-2 13941.93 0.00 342043.72 0 1525515
dm-0 5363.21 0.00 345241.08 0 1529418
dm-1 1142.89 0.00 260744.02 0 1155096
dm-2 8430.70 0.00 367339.05 0 1627312
dm-0 18464.43 0.00 295423.71 0 1320544
dm-1 27161.97 0.00 462172.71 0 2065912
dm-2 38440.27 0.00 636003.58 0 2842936
dm-0 18842.28 0.00 301259.96 0 1346632
dm-1 30305.59 0.00 487824.61 0 2180576
dm-2 32483.22 0.00 535235.79 0 2392504
dm-0 32819.73 0.00 528437.67 0 2356832
dm-1 30986.10 0.00 495330.94 0 2209176
dm-2 32231.17 0.00 515334.53 0 2298392
dm-0 32100.67 0.00 520234.45 0 2325448
dm-1 27425.28 0.00 437621.48 0 1956168
dm-2 30090.38 0.00 480978.97 0 2149976
dm-0 32054.36 0.00 512816.11 0 2292288
dm-1 32423.49 0.00 518761.52 0 2318864
dm-2 32213.87 5.37 519063.98 24 2320216
dm-0 19114.38 0.00 441338.20 0 1963955
dm-1 22902.70 0.00 381944.27 0 1699652
dm-2 20328.31 0.00 434641.80 0 1934156
dm-0 29357.08 0.00 511746.52 0 2277272
dm-1 9773.71 0.00 299194.61 0 1331416
dm-2 26526.52 0.00 476692.13 0 2121280
dm-0 29961.56 0.00 495352.89 0 2229088
dm-1 14524.67 0.00 304846.22 0 1371808
dm-2 32383.33 0.00 531809.78 0 2393144
dm-0 32353.59 0.00 521090.58 0 2324064
dm-1 30616.82 0.00 489497.76 0 2183160
dm-2 30549.33 0.00 488245.74 0 2177576
dm-0 33246.09 0.00 531937.36 0 2377760
dm-1 33248.77 0.00 531948.10 0 2377808
dm-2 33506.49 0.00 536096.64 0 2396352
dm-0 31178.44 0.00 507873.78 0 2285432
dm-1 24935.56 0.00 397466.67 0 1788600
dm-2 31364.67 0.00 508512.00 0 2288304
dm-0 31783.22 0.00 508452.80 0 2272784
dm-1 28665.10 0.00 458310.51 0 2048648
dm-2 30211.86 0.00 486645.19 0 2175304
dm-0 17848.55 0.00 377156.82 0 1685891
dm-1 24671.14 0.00 430719.24 0 1925315
dm-2 26552.13 0.00 487440.72 0 2178860
dm-0 10159.23 0.00 303722.52 0 1348528
dm-1 17420.05 0.00 478315.32 0 2123720
dm-2 1689.19 0.00 256675.68 0 1139640
dm-0 29412.58 0.00 510968.09 0 2273808
dm-1 33237.08 0.00 548357.75 0 2440192
dm-2 16102.25 0.00 303135.28 0 1348952
dm-0 31949.10 5.38 535790.13 24 2389624
dm-1 27270.85 0.00 434730.04 0 1938896
dm-2 19068.39 0.00 305058.30 0 1360560
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
