Re: Multiple MSI messages support

["Shawn Jin" <shawnxjin@gmail.com>]

On Oct 30, 2007 2:51 PM, Roland Dreier <rdreier@cisco.com> wrote:
>
>  > > I understand that the current PCI subsystem or linux kernel (x86)
>  > > supports only one message when MSI is enabled even for devices having
>  > > multiple MSI messages. But why? Is this a limitation solely due to the
>  > > OS or due to the x86 APIC?
>  > > I know the current linux kernel (2.6.23) changed MSI message data
>  > > format a little bit to support other architectures. But some older
>  > > version (e.g. 2.6.18) defined a specific format for the MSI msg data
>  > > in a way that 8 bits contain the irq number and the other 8 bits have
>  > > the interrupt attributes, which is x86 specific. Why does the msg data
>  > > need to contain the irq number? Here is my hypothetic explanation. The
>  > > device writes the MSI msg data to the specified MSI msg address. And
>  > > APIC uses the irq number in the msg data to generate appropriate
>  > > interrupt, which of course results in an appropriate ISR invoked. A
>  > > device having multiple MSI messages typically appends some information
>  > > of which MSI message to the msg data field. For example, if the system
>  > > (or OS) configures the MSI msg data as 0x5000, a device having 4 MSI
>  > > messages could write 0x5000, 0x5001, 0x5002, 0x5003 to differentiate
>  > > the MSI messages. However this cannot work with the APIC due to the
>  > > way how APIC asserts interrupts as I described above (if my
>  > > understanding is correct).
>  > > Hence my answer to the question is this is due to the x86 APIC. For
>  > > other architectures such as powerpc this is probably not a problem
>  > > since the interrupt controller is different. Am I correct?
>
>  > IMO it's more like there has never been enough need for anybody to
>  > look into it, I bet...
>
> Actually the original poster's explanation is pretty accurate.  MSI
> imposes a restriction on how a device generates multiple messages,
> since there is only one message data register and the rest of the
> messages must be based on that in a simple way.  And the format of
> "Intel-style" APIC MSI messages does not match up with the way the PCI
> spec generates multiple messages.
>
> In fact at least IBM pSeries boxes seem to be able to use MSI to
> generate multiple interrupts from the same device -- BenH can probably
> give details.
>
> Multiple interrupt messages are supported by Linux via MSI-X (which
> allows each message to have completely different data).

Thanks for confirming this. I found that Windows also allocates only
one MSI message for such device, though I know nobody here is
interested in Windows. :-P This also confirms from another perspective
that supporting only one MSI message is actually due to the APIC
limitation.

>  > The way drivers are written, you are typically must touch a few key
>  > hardware registers _anyway_, so the multiple messages in practice are
>  > not much more useful than the simple fact that your MSI irq handler
>  > function was called (with all that indicates and implies).
>
> For high-performance devices this is not really true.  The InfiniBand
> mthca driver can use MSI to get a single interrupt, or MSI-X to get
> different interrupts for different types of events.  MSI-X allows the
> read of the "interrupt cause register" to be avoided, and this ends up
> making a measurable performance difference for the fast path.

I agree. Accessing HW registers across PCIe is not quite desirable
since it takes relatively long time especially for reads. In some
situation, if multiple MSI messages were supported, even a single
interrupt would handle multiple events and avoid reading some HW
status register to determine the type of event since the message data
has such information. Hence the performance should be improved quite a
bit.

-Shawn.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/