> -----Original Message-----
> From: Eric W. Biederman [mailto:[email protected]]
> Sent: Monday, October 09, 2006 8:46 AM
> To: Arjan van de Ven
> Cc: Linus Torvalds; Muli Ben-Yehuda; Ingo Molnar; Thomas
> Gleixner; Benjamin Herrenschmidt; Rajesh Shah; Andi Kleen;
> Protasevich, Natalie; Luck, Tony; Andrew Morton;
> Linux-Kernel; Badari Pulavarty; Roland Dreier
> Subject: Re: 2.6.19-rc1 genirq causes either boot hang or
> "do_IRQ: cannot handle IRQ -1"
>
> Arjan van de Ven <[email protected]> writes:
>
> >> > So yes, having software say "We want to steer this particular
> >> > interrupt to this L3 cache domain" sounds eminently sane.
> >> >
> >> > Having software specify which L1 cache domain it wants
> to pollute
> >> > is likely just crazy micro-management.
> >>
> >> The current interrupt delivery abstraction in the kernel
> is a set of
> >> cpus an interrupt can be delivered to. Which seem
> sufficient to the
> >> cause of aiming at a cache domain. Frequently the lower levels of
> >> interrupt delivery map this to a single cpu because of hardware
> >> limitations but in certain cases we can honor a multiple
> cpu request.
> >>
> >> I believe the scheduler has knowledge about different locality
> >> domains for NUMA and everything else. So what is wanting
> on our side
> >> is some architecture? work to do the broad steering by default.
> >
> >
> > well normally this is the job of the userspace IRQ balancer to get
> > right; the thing is undergoing a redesign right now to be
> smarter and
> > deal better with dual/quad core, numa etc etc, but as a principle
> > thing this is best done in userspace (simply because there's higher
> > level information there, like "is this interrupt for a network
> > device", so that policy can take that into account)
>
> So far I have seen all of that higher level information in
> the kernel, and it has to export it to user space for the
> user space daemon to do anything about it.
>
> The only time I have seen user space control being useful is
> when there isn't a proper default policy so at least you can
> distribute things between the cache domains properly. So far
> I have been more tempted to turn it off as it will routinely
> change which NUMA node an irq is pointing at, which is not at
> all ideal for performance, and I haven't seen a way (short of
> replacing it) to tell the user space irq balancer that it got
> it's default policy wrong.
>
> It is quite possible I don't have the whole story, but so far
> it just feels like we are making a questionable decision by
> pushing things out to user space. If nothing else it seems
> to make changes more difficult in the irq handling
> infrastructure because we have to maintain stable interfaces.
> Things like per cpu counters for every irq start to look
> really questionable when you scale the system up in size.
I'd like also to question current policies of user space irqbalanced. It
seems to just go round-robin without much heuristics involved. We are
seeing loss of timer interrupts on our systems - and the more processors
the more noticeable it is, but it starts even on 8x partitions; on 48x
system I see about 50% loss, on both ia32 and x86_64 (haven't checked on
ia64 yet). With say 16 threads it is unsettling to see 70% overall idle
time, and still only 40-50% of interrupts go through. System's time is
not affected, so the problem is on the back burner for now :) It's not
clear yet whether this is software or hardware fault, and how much
damage it does (performance wise etc.) I will provide more information
as it becomes available, just wanted to raise red flag, maybe others
also see such phenomena.
Thanks,
--Natalie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
