Re: PREEMPT_RT : 2.6.16-rt12 and boot : BUG ?

[Steven Rostedt <rostedt@goodmis.org>]

On Tue, 2006-04-11 at 18:15 +0200, Serge Noiraud wrote:
> Hi,
> 
> 	It now works with all config parameters set to yes on 2.6.16-rt16.
> however, PERCPU_ENOUGH_ROOM is set to 384KB.
> I'm now trying to lower this value of PERCPU_ENOUGH_ROOM.
> With 256K and 2.6.16-rt16 : doesn't work.
> With 320K : OK
> So I need to use 320K for PERCPU_ENOUGH_ROOM to boot correctly.
> 
> I have several questions :
> Is there a problem in my config file ?

Not that I know of. (perhaps debugging options are on. Ingo?)
Or you have too many things as modules (that's our problem, not yours).

> Will this memory freed at end of kernel loading ?

No

> Why do we need such a size ?

It seems that the -rt kernel has increased the size of structures that
are used in modules and are defined per cpu.

> What usage is this for ?

There are variables that are defined per CPU.  The reason for variables
to be defined special for each CPU is that you want the variables in
their own cache line such that modifying a variable that is specific for
a CPU wont cause a write to a cache line that has a variable (say read
only) to all CPUS. Because this would cause strain on the bus and slow
things down as the write to the cache line is causing the other CPUs to
update that line and become coherent.

But to make this easier for developers and to actually save space (don't
want to waste the cache alignment just to space out variables), there is
a lot of linker magic to do all the work for you.  So all a developer
needs to do to declare a variable with DEFINE_PER_CPU(type, name) and
friends and the linker takes care of the rest.

Currently this is implemented by creating a section at compile time to
hold all these variables.  At compile time, only one set is made.  When
the machine boots up, this section is copied (cache aligned) NR_CPUS
times.  And to access these variables, a macro per_cpu(var, cpu) is used
to find the variable in this index.  Note: since the size of
PERCPU_ENOUGH_ROOM is used if it is bigger than the current compile time
section, PERCPU_ENOUGH_ROOM must be a multiple of the cache size or
there can be an overlap in the CPU cache lines. (hmm, this looks like a
patch is needed.)

Now the problem you have is with modules.  Since the variables in the
per_cpu() macro are looked up via an index and cpu, all these variables
must be located in the same section.  Currently, to make this easier,
(and this too probably should change), the per_cpu variables of a module
are put in this same section.  So when a module is loaded, it finds a
block in the per cpu area that is available and makes a copy of its per
cpu variables into each section (per cpu).  But since this is static
memory (the per cpu section cant grow) it must be allocated at boot up
hoping that there's enough room for the modules that will be loaded in
the future.  Don't worry about leaks, when a module is unloaded, it
frees up the space in the per cpu area.

What you have seen, is that the -rt patch grew something that the
modules were using per cpu.  So when it tried to allocate the space in
the per cpu area, there wasn't enough room.  So your module failed to be
loaded.

Hope this helps,

-- Steve


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/