Re: High priority tasks break SMP balancer?

[Micah Dowty <micah@vmware.com>]

Dmitry,

Thank you for the detailed explanation of the scheduler behaviour I've
been seeing.

On Thu, Nov 22, 2007 at 01:53:02PM +0100, Dmitry Adamushko wrote:
> >  - Is there a good way to detect, without any kernel debug flags
> >    set, whether the current machine has any scheduling domains
> >    that are missing the SD_BALANCE_NEWIDLE bit?
> 
> e.g. by reading its config and kernel version. But as a generic way
> (sort of API for user-space), I guess, no.

Would it be reasonable perhaps to look for the numa_* entries in
/proc/zoneinfo, vmstat, or slabinfo?

> > I'm looking for
> >    a good way to work around the problem I'm seeing with VMware's
> >    code. Right now the best I can do is disable all thread priority
> >    elevation when running on an SMP machine with Linux 2.6.20 or
> >    later.
> 
> why are your application depends on the load-balancer's decisions?
> Maybe it's just smth wrong with its logic instead? :-/

The application doesn't really depend on the load-balancer's decisions
per se, it just happens that this behaviour I'm seeing on NUMA systems
is extremely bad for performance.

In this context, the application is a virtual machine runtime which is
executing either an SMP VM or it's executing a guest which has a
virtual GPU. In either case, there are at least three threads:

  - Two virtual CPU/GPU threads, which are nice(0) and often CPU-bound
  - A low-latency event handling thread, at nice(-10)

The event handling thread performs periodic tasks like delivering
timer interrupts and completing I/O operations. It isn't expected to
use much CPU, but it does run very frequently. To get the best latency
performance, we've been running it at nice(-10). The intention is that
this event handling thread should usually be able to preempt either of
the CPU-bound threads.

The behaviour we expect is that on a system which is otherwise lightly
loaded, the two CPU-bound threads each get nearly an entire physical
CPU to themselves. The event handling thread periodically wakes up,
preempts one of the CPU-bound threads briefly, then goes to sleep and
yields the CPU back to one of the CPU-bound threads.

The actual behaviour I see when the load balancer makes this
unfavorable decision is that the two CPU-bound threads share a single
CPU, and the other CPU is mostly idle. Best case, the virtual machine
runs half as fast as it could be running. Worst case it runs much
slower, because the CPU-bound threads often need to wait for each
other to catch up if they don't always run at about the same
rate. This involves a lot of extra synchronization overhead.

My current workaround for this is to avoid ever boosting our thread
priority on a kernel in which this problem could occur. Currently this
is any 2.6.20 or later kernel running with at least two CPUs. I'd like
to narrow this test now to only cover kernels with NUMA enabled.

Hopefully this explains why the scheduler behaviour I've observed is
undesirable for this particular workload. I would be surprised if I'm
the only one who has a workload that is negatively impacted by it, but
it does seem difficult to reconcile this workload's requirements with
the cache properties expected from a NUMA-aware scheduler.

Is there something I'm doing wrong, or is this just a weakness in the
current scheduler implementation? If it's the latter, are there any
planned improvements?

Thank you again,
--Micah
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/