Re: [ck] Re: [PATCH] implement nice support across physical cpus on SMP

[mjt@nysv.org (Markus Törnqvist)]

On Mon, May 09, 2005 at 09:47:05PM +1000, Con Kolivas wrote:
>
>Thanks for feedback.

For once I can give something back, it seems; thus it's my pleasure.

>> top - 11:09:24 up 15:30,  2 users,  load average: 4.99, 3.55, 1.63
>>   PID USER      PR  NI  VIRT  SHR S %CPU %MEM    TIME+  #C COMMAND
>>  3917 mjt       25   0  5660  936 R 99.6  0.0   6:11.35  0 load_base.sh
>>  3918 mjt       24   0  5660  936 R 99.6  0.0   6:11.34  3 load_base.sh
>>  3916 mjt       39   0  5660  936 R 65.7  0.0   3:28.95  2 load_base.sh
>>  3919 mjt       39  19  5660  936 R  7.0  0.0   0:23.54  1 load_base.sh
>>  3920 mjt       39  19  5660  936 R  3.0  0.0   0:10.33  2 load_base.sh
>
>These runs don't look absolutely "ideal" as one nice 19 task is bound to cpu1 
>however since you're running hyperthreading it would seem the SMT nice code 
>is keeping that under check anyway (0:23 vs 6:11)

So let no one touch the SMT code as long as it works...

>These runs pretty much confirm what I found to happen. My test machine for 
>this was also 4x. I can't see how the code would behave differently on 2x. 

Who on whichever list one is subscribed to or not would care to
replicate these results on 2x and report?

Thank you.

>Perhaps if I make the prio_bias multiplied instead of added to the cpu load 
>it will be less affected by SCHED_LOAD_SCALE. The attached patch was 
>confirmed during testing to also provide smp distribution according to nice 
>on 4x. Carlos I know your machine is in production so you testing may not be 
>easy for you. Please try this on top if you have time.

I have no idea about SCHED_LOAD_SCALE, I'm afraid, but I will give
this latest patch a run while I'm at it.

The load.sh is the same I posted before

$ ./load.sh 5
top - 19:41:33 up 9 min,  2 users,  load average: 0.40, 0.10, 0.03
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2798 mjt       25   0  5660 1140  936 R 99.9  0.0   0:02.04  2 load_base.sh   
 2799 mjt       25   0  5660 1140  936 R 99.9  0.0   0:02.04  3 load_base.sh   
 2797 mjt       25   0  5660 1140  936 R 51.8  0.0   0:01.16  0 load_base.sh   
 2801 mjt       39  19  5660 1140  936 R  7.0  0.0   0:00.12  1 load_base.sh   
 2800 mjt       39  19  5660 1140  936 R  3.0  0.0   0:00.05  0 load_base.sh   

top - 19:42:20 up 10 min,  2 users,  load average: 2.83, 0.78, 0.26
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2798 mjt       35   0  5660 1140  936 R 99.5  0.0   0:48.55  2 load_base.sh   
 2799 mjt       35   0  5660 1140  936 R 99.5  0.0   0:48.55  3 load_base.sh   
 2797 mjt       34   0  5660 1140  936 R 61.7  0.0   0:27.43  0 load_base.sh   
 2801 mjt       39  19  5660 1140  936 R  6.0  0.0   0:03.11  1 load_base.sh   
 2800 mjt       39  19  5660 1140  936 R  3.0  0.0   0:01.35  0 load_base.sh   

top - 19:43:00 up 10 min,  2 users,  load average: 3.88, 1.31, 0.46
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2798 mjt       24   0  5660 1140  936 R 99.9  0.0   1:29.13  2 load_base.sh   
 2799 mjt       24   0  5660 1140  936 R 99.5  0.0   1:29.12  3 load_base.sh   
 2797 mjt       24   0  5660 1140  936 R 49.8  0.0   0:50.19  0 load_base.sh   
 2801 mjt       39  19  5660 1140  936 R  7.0  0.0   0:05.76  1 load_base.sh   
 2800 mjt       39  19  5660 1140  936 R  3.0  0.0   0:02.48  0 load_base.sh   

$ ./load.sh 7
top - 19:43:49 up 11 min,  2 users,  load average: 4.98, 1.97, 0.73
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2807 mjt       21   0  5660 1140  936 R 99.9  0.0   0:27.73  0 load_base.sh   
 2804 mjt       34   0  5660 1140  936 R 99.6  0.0   0:23.46  1 load_base.sh   
 2808 mjt       20   0  5660 1140  936 R 99.6  0.0   0:25.51  2 load_base.sh   
 2805 mjt       39   0  5660 1140  936 R 39.8  0.0   0:08.12  3 load_base.sh   
 2806 mjt       33   0  5660 1140  936 R 37.9  0.0   0:12.46  3 load_base.sh   
 2788 mjt       20   0  5168 1092  832 R  1.0  0.0   0:00.56  3 top            
 2809 mjt       39  19  5660 1140  936 R  1.0  0.0   0:00.41  3 load_base.sh   
 2810 mjt       39  19  5660 1144  936 R  1.0  0.0   0:00.40  3 load_base.sh   

top - 19:44:20 up 12 min,  2 users,  load average: 5.78, 2.45, 0.92
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2807 mjt       22   0  5660 1140  936 R 99.9  0.0   0:56.56  0 load_base.sh   
 2804 mjt       35   0  5660 1140  936 R 99.5  0.0   0:52.29  1 load_base.sh   
 2808 mjt       21   0  5660 1140  936 R 99.5  0.0   0:54.34  2 load_base.sh   
 2805 mjt       35   0  5660 1140  936 R 33.8  0.0   0:15.99  3 load_base.sh   
 2806 mjt       39   0  5660 1140  936 R 21.9  0.0   0:20.22  3 load_base.sh   
 2788 mjt       20   0  5168 1092  832 R  1.0  0.0   0:00.65  3 top            
 2809 mjt       39  19  5660 1140  936 R  1.0  0.0   0:00.80  3 load_base.sh   
 2810 mjt       39  19  5660 1144  936 R  1.0  0.0   0:00.79  3 load_base.sh   

top - 19:45:00 up 12 min,  2 users,  load average: 6.37, 3.02, 1.18
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2804 mjt       28   0  5660 1140  936 R 99.9  0.0   1:32.02  1 load_base.sh   
 2807 mjt       35   0  5660 1140  936 R 99.9  0.0   1:36.29  0 load_base.sh   
 2808 mjt       33   0  5660 1140  936 R 99.5  0.0   1:34.07  2 load_base.sh   
 2806 mjt       27   0  5660 1140  936 R 30.9  0.0   0:31.09  3 load_base.sh   
 2805 mjt       39   0  5660 1140  936 R 24.9  0.0   0:26.81  3 load_base.sh   
 2809 mjt       39  19  5660 1140  936 R  1.0  0.0   0:01.34  3 load_base.sh   
 2810 mjt       39  19  5660 1144  936 R  1.0  0.0   0:01.33  3 load_base.sh   

Then I decided to do something crazier and renice some pids, to see
what happens...

top - 19:45:45 up 13 min,  2 users,  load average: 6.70, 3.58, 1.45
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2807 mjt       37   0  5660 1140  936 R 99.9  0.0   2:21.19  0 load_base.sh   
 2804 mjt       30   0  5660 1140  936 R 99.5  0.0   2:16.92  1 load_base.sh   
 2806 mjt       36   0  5660 1140  936 R 41.8  0.0   0:43.73  2 load_base.sh   
 2805 mjt       39   0  5660 1140  936 R 21.9  0.0   0:39.13  2 load_base.sh   
 2809 mjt       39  19  5660 1140  936 R  6.0  0.0   0:02.10  3 load_base.sh   
 2808 mjt       39  10  5660 1140  936 R  2.0  0.0   2:16.01  2 load_base.sh   
 2810 mjt       39  19  5660 1144  936 R  2.0  0.0   0:01.96  2 load_base.sh   

top - 19:46:20 up 14 min,  2 users,  load average: 6.83, 3.95, 1.66
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2804 mjt       35   0  5660 1140  936 R 99.6  0.0   2:52.38  1 load_base.sh   
 2805 mjt       37   0  5660 1140  936 R 99.6  0.0   0:54.29  3 load_base.sh   
 2807 mjt       21   0  5660 1140  936 R 99.6  0.0   2:56.65  0 load_base.sh   
 2806 mjt       21   0  5660 1140  936 R 23.9  0.0   0:53.67  2 load_base.sh   
 2808 mjt       39  10  5660 1140  936 R 21.9  0.0   2:34.49  2 load_base.sh   
 2809 mjt       39  19  5660 1140  936 R  2.0  0.0   0:02.66  2 load_base.sh   
 2810 mjt       39  19  5660 1144  936 R  2.0  0.0   0:02.57  2 load_base.sh   

top - 19:47:00 up 14 min,  2 users,  load average: 6.91, 4.33, 1.88
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2805 mjt       32   0  5660 1140  936 R 99.9  0.0   1:34.05  3 load_base.sh   
 2807 mjt       37   0  5660 1140  936 R 99.9  0.0   3:36.42  0 load_base.sh   
 2804 mjt       30   0  5660 1140  936 R 99.6  0.0   3:32.15  1 load_base.sh   
 2806 mjt       36   0  5660 1140  936 R 40.8  0.0   1:07.04  2 load_base.sh   
 2808 mjt       39  10  5660 1140  936 R 21.9  0.0   2:41.09  2 load_base.sh   
 2809 mjt       39  19  5660 1140  936 R  2.0  0.0   0:03.32  2 load_base.sh   
 2810 mjt       39  19  5660 1144  936 R  1.0  0.0   0:03.23  2 load_base.sh   

And sudo renice before I call it a day

top - 19:48:27 up 16 min,  2 users,  load average: 7.21, 5.05, 2.34
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2806 mjt       28   0  5660 1140  936 R 99.9  0.0   1:57.55  2 load_base.sh   
 2807 mjt       27   0  5660 1140  936 R 99.9  0.0   5:00.99  0 load_base.sh   
 2810 mjt       26 -10  5660 1144  936 R 96.3  0.0   0:05.56  3 load_base.sh   
 2804 mjt       39   0  5660 1140  936 R 24.8  0.0   4:57.88  1 load_base.sh   
 2808 mjt       36  10  5660 1140  936 R  9.5  0.0   2:56.90  1 load_base.sh   
 2805 mjt       39   0  5660 1140  936 R  1.0  0.0   2:29.56  1 load_base.sh   

top - 19:49:10 up 16 min,  2 users,  load average: 7.65, 5.46, 2.61
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2806 mjt       24   0  5660 1140  936 R 99.9  0.0   2:39.96  2 load_base.sh   
 2807 mjt       23   0  5660 1140  936 R 99.9  0.0   5:43.40  0 load_base.sh   
 2810 mjt       26 -10  5660 1144  936 R 99.9  0.0   0:45.87  1 load_base.sh   
 2805 mjt       39   0  5660 1140  936 R 17.5  0.0   2:36.59  3 load_base.sh   
 2808 mjt       39  10  5660 1140  936 R  8.7  0.0   2:59.79  3 load_base.sh   
 2804 mjt       27   0  5660 1140  936 R  7.1  0.0   5:03.47  3 load_base.sh   

top - 19:49:45 up 17 min,  2 users,  load average: 7.36, 5.63, 2.77
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  #C COMMAND        
 2806 mjt       31   0  5660 1140  936 R 99.9  0.0   3:15.66  2 load_base.sh   
 2807 mjt       29   0  5660 1140  936 R 99.9  0.0   6:19.10  0 load_base.sh   
 2810 mjt       26 -10  5660 1144  936 R 99.9  0.0   1:16.74  3 load_base.sh   
 2804 mjt       39   0  5660 1140  936 R 17.5  0.0   5:09.32  1 load_base.sh   
 2805 mjt       39   0  5660 1140  936 R 17.5  0.0   2:45.17  1 load_base.sh   
 2808 mjt       39  10  5660 1140  936 R  8.7  0.0   3:02.87  1 load_base.sh   

Seems good enough under this very fabricated stress, hopefully someone
can tell me a good practical application with processes of different
nices going all over the place, so I can try something else.

But these processes are still clinging a bit to cpu 1 here, that's probably
another SMT feature.
Who tests this on SMP without SMT? Anyone? You! Over there!

This box will go into production soon and then I can maybe get some
glimpses of what happens in practice, and that's about it.
And it'll probably run everything with default values, except a light
mysql -5.

Thanks!

-- 
mjt

Attachment: signature.asc
Description: Digital signature