Re: Inconsistent timing results of multithreaded program on an SMP machine.

[Marcel Zalmanovici <MARCEL@il.ibm.com>]

Hi Paul,

I've tried your suggestion.
I tried first adding latestcpu( getpid() ), like you've said, but as you
can see from the log this did not yield anything interesting.
I've then tried with latestcpu( gettid() ). The results there surprised my
a bit:

I would've expected to see low times whenever the 2 threads using a DS are
on the same CPU and high times when they are placed on different CPUs.
Also lower times would be expected when the threads aer divided 4 to CPU 0
and 4 to CPU 1 (I've disabled hyperthreading for now).

If you look at the run that took 14.78sec you'll see that only the threads
of 1 out of 4 DS are on same CPU, but result is low. Conversly, look at the
19.55sec reults; same thing happens there.
Pretty much the same can be seen looking at the 14.62 and 19.57 results,
except there the threads of 2 out of 4 DS match.

As for the 2nd presumption, look at times 14.64 and 18.90 - threads are
divided 4 and 4 on both ocassions.

Here are the log files:

(See attached file: latest.tid.log)(See attached file: latest.pid.log)

Any other ideas are highly welcomed.

Marcel



                                                                                                                                   
                      Paul Jackson                                                                                                 
                      <pj@sgi.com>             To:       Marcel Zalmanovici/Haifa/IBM@IBMIL                                        
                                               cc:       linux-kernel@vger.kernel.org                                              
                      20/11/2005 16:02         Subject:  Re: Inconsistent timing results of multithreaded program on an SMP        
                                                machine.                                                                           
                                                                                                                                   
                                                                                                                                   
                                                                                                                                   



Marcel wrote:
> Instead what I've got was an oscillation where the maximum time was twice
> and more than the minimum!! For a short test results ranged ~7sec to ~16
...

Just for grins, try displaying which cpu each thread runs on.  Display the
return from "latestcpu(getpid())" in the two per-thread printf's, to
display
the thread's cpu at the beginning and end of each compute_thread().
Perhaps
you will notice that the per-thread cpu correlates with the test times.

/*
 * int latestcpu(pid_t pid)
 *
 * Copyright (C) 2005 Silicon Graphics, Inc.
 * This code is subject to the terms and conditions of the
 * GNU General Public License.
 *
 * Return most recent CPU on which task pid executed.
 *
 * The last used CPU is visible for a given pid as field #39
 * (starting with #1) in the file /proc/<pid>/stat.  Currently
 * this file has 41 fields, in which case this is the 3rd to
 * the last field.
 */

#include <stdio.h>
#include <fcntl.h>
#include <sys/types.h>
#include <limits.h>
#include <linux/limits.h>

int latestcpu(pid_t pid)
{
             char buf[PATH_MAX];
             int fd;
             int cpu = -1;

             snprintf(buf, sizeof(buf), "/proc/%d/stat", pid);
             fd = open(buf, O_RDONLY);
             buf[0] = 0;     /* in case fd < 0 and read() is no-op */
             read(fd, buf, sizeof(buf));
             close(fd);
             sscanf(buf, "%*u %*s %*s %*u %*u %*u %*u %*u %*u %*u "
                             "%*u %*u %*u %*u %*u %*u %*u %*u %*u %*u "
                             "%*u %*u %*u %*u %*u %*u %*u %*u %*u %*u "
                             "%*u %*u %*u %*u %*u %*u %*u %*u %u", /* 39th
field */
                             &cpu);
             return cpu;
}

--
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Attachment: latest.tid.log
Description: Binary data

Attachment: latest.pid.log
Description: Binary data