Re: [RFC] MMIO accessors & barriers documentation #2

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

On Sun, 2006-09-17 at 19:24 -0700, Linus Torvalds wrote:

> >  1- {read,write}{b,w,l,q} : MMIO accessors. Those accessors provide
> > all MMIO ordering requirements. They are thus called "fully ordered".
> > That is #1, #2 and #4 for writes and #1 and #3 for reads. 
> 
> Well, it's already not defined to be #4 right now on SGI boxes, and we 
> have that (badly named) mmiowb() thing to enforce #4, so I think we should 
> just accept that write[bwl]() it's _that_ ordered.
>
> And on x86, we _already_ depend on "wmb()" to be a "normal write to MMIO 
> write" barrier, which is technically wrong and bad. Again, thanks to 
> mmiowb(), normal memory accesses and MMIO accesses have already been 
> defined to not be in the same "ordering domain", so "wmb()" is technically 
> wrong and may not order a regular write wrt a MMIO (because it doesn't do 
> so for the other order: MMIO->spin_unlock).
> 
> So I think we should just admit that at least MMIO _stores_ are already 
> not entirely ordered, and not try to strengthen the rules for the current 
> setup (and just try to clarify the currently accepted semantics).

I'm a little bit confused as Alan seemed to imply that memory store vs.
MMIO were fully ordered on x86 and thus no barrier was needed (which was
the start of the discussion in the first place since on PowerPC, they
were not ordered until Paul's latest patch).

I tend to agree with Alan and others though that non-ordered accessors
will be a problem with driver writers who don't understand what it's all
about. David Miller even said that assuming anything else but strongly
ordered accessor was basically shooting ourselves in the foot :)

There have been two main cases biting us so far out of my list of 4,
which is #2 and #4 in my list of ordering requirements: memory store vs.
mmio (read or store) and mmio vs. spinlock. The former was/is handled in
some drivers with wmb() (which I dislike as wmb() should really only be
defined for the coherency domain) and not in others. The later was/is
handled in some drivers with mmiowb() (which I agree is badly named).

Thus we though it might be a more reasonable path to simply define
existing accessors as strongly ordered (and make them so on archs where
they aren't yet) and add some partially relaxed ones with semantics
precise enough to make them useful for drivers in non-arch specific
ways.

Now, you seem to say (unless I misunderstood) that x86 -does- have the
problem of storing to memory followed by an MMIO write not being in
order (wich means that tg3 would have been broken on x86 as well) which
I though wasn't the case.

So what do you recommend we do now ? Go back to step #1 and simply
define readl,writel and friends as not being ordered vs. memory loads
and then define a bunch of explicit barriers, which basically boils down
to using the semantics I defined in my document for class 2, partially
relaxed accessors (__readl/__writel/...) ? Or do we consider that it's
too much of a risk vs. driver writers and do implement my proposal,
making the current accessors slower on some archs but safer, and
starting to convert some drivers hot path to use the relaxed ones ?

Ben.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/