[RFC] Add thread_info flag for "no cpu migration"

[Keith Owens <kaos@sgi.com>]

The thread below talks about disabling preemption in udelay() because
different cpus can have hardware clocks that move at different rates.

http://marc.theaimsgroup.com/?l=linux-ia64&m=113460274218885&w=2

preempt_disable() is overkill for this class of problem.  Code that
uses per cpu variables and/or hardware often only has the requirement
that it not be migrated off the current cpu, it can cope with
preemption as long as it gets scheduled back onto the same cpu.
Disabling preemption just to prevent cpu migration can introduce
additional preemption delays.

The normal way of pinning a task to a cpu is to use set_cpus_allowed(),
but that requires that the caller hold no locks and be running enabled.
Code such as udelay() can be called in any context, it may or may not
be runing atomic.

Could we add a TIF_ flag that says "no migration to another cpu"?  It
would be very light weight, functions cpu_migration_save(flags) and
cpu_migration_restore(flags) plus a couple of tests in the scheduler.

#define cpu_migration_save(flags) do { flags = test_and_set_thread_flag(TIF_NO_CPU_MIGRATION); } while(0)
#define cpu_migration_restore(flags) do { if (!flags) clear_thread_flag(TIF_NO_CPU_MIGRATION); } while(0)

Unlike set_cpus_allowed(), TIF_NO_CPU_MIGRATION can be set in any
context.  This makes it far more useful for lower level code, which may
be called with interrupts or preemption enabled or disabled.

I do not know enough about the scheduler to be completely sure of what
changes are required there, but at a quick glance -

set_cpus_allowed() returns -EBUSY if TIF_NO_CPU_MIGRATION is set and
the new mask does not include the current cpu.

can_migrate_task() returns 0 if TIF_NO_CPU_MIGRATION is set.

sched_fork() and sched_exec() BUG if TIF_NO_CPU_MIGRATION is set, it
makes no sense to fork/exec with migration disabled.  These are sanity
checks, and may not be absolutely required.

If this feature is added, some uses of get_cpu() could be redefined to
use cpu_migration_save(), further reducing the size of code that runs
with preempt disabled.

The debug version of smp_processor_id() would accept
TIF_NO_CPU_MIGRATION being set as one of the criteria for not
complaining.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/