status of: tasklet_unlock_wait() causes soft lockup with -rt and ieee1394 audio

[Pieter Palmers <pieterp@joow.be>]

Dear all,

What is the status with respect to this problem? I see that in the 
current -rt patch the problematic code piece is different. I personally 
haven't tried to reproduce this myself on a more recent kernel, but I 
just got a report from one of our users who experienced the same problem 
with 2.6.19-rt15 and RT preemption (desktop preemption works fine).

Should the latest -rt patches be fixed with respect to this issue? If so 
I'll try and test them, otherwise I omit the effort.

Thanks,

Pieter

Lee Revell wrote:
> Pieter has found this bug in -rt:
>
> We are experiencing 'soft' deadlocks when running our code (Freebob, 
> i.e. userspace lib for firewire audio) on RT kernels. After a few 
> seconds of system freeze, I get a kernel panic message that signals a soft lockup.
>
> I've uploaded the photo's of the panic here:
> http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
> http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
> both are of suboptimal quality unfortunately, but all info is readable 
> on one or the other.
>
> The problems occur when an ISO stream (receive and/or transmit) is shut 
> down in a SCHED_FIFO thread. More precisely when running the freebob 
> jackd backend in real-time mode. And even more precise: they only seem 
> to occur when jackd is shut down. There are no problems when jackd is 
> started without RT scheduling.
>
> I havent been able to reproduce this with other test programs that are 
> shutting down streams in a SCHED_FIFO thread.
>
> The problem is not reproducible on non-RT kernels, and it only occurs on those configured for 
> PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. The PREEMPT_DESKTOP setting was the only change between the two tests, all other kernel settings (threaded irq's etc...) were unchanged.
>
> My tests are performed on 2.6.17-rt1, but the lockups are confirmed for 
> PREEMPT_RT configured kernels 2.6.14 and 2.6.16.
>
> Some extra information:
>
> Lee Revell wrote:
>
> > <...>
> >
> > It seems that the -rt patch changes tasklet_kill:
> >
> > Unpatched 2.6.17:
> >
> > void tasklet_kill(struct tasklet_struct *t)
> > {
> >         if (in_interrupt())
> >                 printk("Attempt to kill tasklet from interrupt\n");
> >
> >         while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
> >                 do
> >                         yield();
> >                 while (test_bit(TASKLET_STATE_SCHED, &t->state));
> >         }
> >         tasklet_unlock_wait(t);
> >         clear_bit(TASKLET_STATE_SCHED, &t->state);
> > }
> >
> > 2.6.17-rt:
> >
> > void tasklet_kill(struct tasklet_struct *t)
> > {
> >         if (in_interrupt())
> >                 printk("Attempt to kill tasklet from interrupt\n");
> >
> >         while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
> >                 do                              msleep(1);
> >                 while (test_bit(TASKLET_STATE_SCHED, &t->state));
> >         }
> >         tasklet_unlock_wait(t);
> >         clear_bit(TASKLET_STATE_SCHED, &t->state);
> > }
> >
> > You should ask Ingo & the other -rt developers what the intent of this
> > change was.  Obviously it loops forever waiting for the state bit to
> > change.
>
> On Thu, 2006-07-06 at 22:14 +0200, Pieter Palmers wrote:
>
> > > I've put the debugging printk's into tasklet_kill. One interesting 
> > > remark is that now that they are in place, I had to start/stop jackd 
> > > multiple times before deadlock occurs. Without the printk's the machine 
> > > always locked up on the first pass. However I stopped after the first 
> > > lockup, so maybe this is not really significant.
> > >
> > > Anyway, the new tasklet_kill looks like this:
> > >
> > > void tasklet_kill(struct tasklet_struct *t)
> > > {
> > > 	printk("enter tasklet_kill\n");
> > > 	if (in_interrupt())
> > > 		printk("Attempt to kill tasklet from interrupt\n");
> > > 	
> > > 	printk("passed interrupt check\n");
> > >
> > > 	while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
> > > 		do
> > > 			msleep(1);
> > > 		while (test_bit(TASKLET_STATE_SCHED, &t->state));
> > > 	}
> > > 	printk("passed test_and_set_bit\n");
> > > 	
> > > 	tasklet_unlock_wait(t);
> > > 	printk("passed tasklet_unlock_wait\n");
> > > 	
> > > 	clear_bit(TASKLET_STATE_SCHED, &t->state);
> > > }
> > >
> > > And the last line printed before lockup is:
> > > "passed test_and_set_bit"
> >   
> This makes the change in tasklet_unlock_wait() as the prime suspect for this problem.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/