Re: BUG in filp_close() (was: Re: 2.6.19-rc1-mm1)

[Vadim Lobanov <vlobanov@speakeasy.net>]

On Tuesday 10 October 2006 15:07, Dave Kleikamp wrote:
> On Tue, 2006-10-10 at 10:47 -0500, Dave Kleikamp wrote:
> > On Tue, 2006-10-10 at 00:09 -0700, Andrew Morton wrote:
> > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.19-rc
> > >1/2.6.19-rc1-mm1/
> >
> > I'm seeing an exception in filp_close(), called from sys_dup2().  I have
> > only seen it when I try to start up a java application (Lotus
> > Workplace).
> >
> > I suspect that it may be related to the fdtable work, but I haven't
> > investigated it too closely.
>
> Still don't know exactly what's going on here.  In case it helps, this
> is the call to dup2() from strace output:
>
> 1419  open("/dev/null", O_RDWR)         = 7
> 1419  getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
> 1419  dup2(7, 524)                      = 524
> 1419  dup2(7, 525 <unfinished ...>
>
> > > +fdtable-delete-pointless-code-in-dup_fd.patch
> > > +fdtable-make-fdarray-and-fdsets-equal-in-size.patch
> > > +fdtable-remove-the-free_files-field.patch
> > > +fdtable-implement-new-pagesize-based-fdtable-allocator.patch
> > >
> > >  Redo the fdtable code.

D'oh!!! Everybody who hit this bug can feel free to call me a moron now! (And 
Andrew will probably take me up on that offer, for all the residual flak he 
caught. :)) The problem is in the following logic:
+        nr++;
+        nr /= (PAGE_SIZE / 4 / sizeof(struct file *));
+        nr = roundup_pow_of_two(nr);
+        nr *= (PAGE_SIZE / 4 / sizeof(struct file *));
+        if (nr > NR_OPEN)
+                nr = NR_OPEN;
The problem is that roundup_pow_of_two() will not necessarily bring the array 
up to the necessary size, and we get an array overflow. This is clearly 
visible in the example above: dup2(..., 524) with a PAGE_SIZE of 4K. (Thanks 
for sending that in, Dave.) Let me think about the best way to fix this 
computation, and I'll send out a patch for you folks to test to see if it 
fixes your problem, if you'll oblige.

-- Vadim Lobanov, idiot of the day
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/