Re: oops in :snd_pcm_oss:resample_expand+0x19c/0x1f0

[Vivek Goyal <vgoyal@in.ibm.com>]

On Mon, Sep 25, 2006 at 12:03:59AM +0200, Jean-Marc Saffroy wrote:
> On Sun, 24 Sep 2006, Andrew Morton wrote:
> 
> >>I keep a crash dump from kdump,
> >
> >Whoa.  Impressed.  Which distro are you using and how did you go about this
> >and how much fuss was it to set up?
> 
> Well, it sure didn't work right out of the box... But it's not that hard, 
> really: the main problems are lack of proper end-user docs and packaging, 
> as usual.

Hi Jean,

Thanks for trying out kdump. Things are still in state of flux when it
comes to packaging and I am hoping that its a matter of few more months
and things will be in a much better shape.

A lot of kdump functionality is in user space when it comes to configuring
kdump, saving the core dump, filtering the image and then finally doing the
analysis. Complete user space automation is still being implemented by
distros and issues are being resolved. 

One of the issues is  how to bring all the distros on the same platform.
SuSE and RedHat are doing quite some amount of automation for using kdump.
For example, the small script you wrote for saving the dump, is already
part of Fedora and SuSE and will be available in RHEL. I have no idea
what's going on in debian when it comes to packaging.

> A serial console on another PC very much helped in 
> troubleshooting problems (eg. my kdump kernel's initrd was initially too 
> large), but it's not required (though it's always a nice thing to have 
> anyway).
> 
> The basic instructions for setting up kdump are here:
>   http://lse.sourceforge.net/kdump/kdump-test-reports/test-plan.txt
> 

More docs:

-  Documentation/kdump/kdump.txt
-  Distros (SuSE, RedHat) are now also packaging a README file in
   /usr/share/doc.... They are also putting together some man page
   info for kexec sytem call.

> I followed them, and also had to:
>  - reduce the uncompressed size of my initramfs (was 50+ MB initially, now 
> ca. 22MB), by listing only the required modules (lsmod w/ some hand 
> picking), see /etc/initramfs-tools/initramfs.conf

Hmm.. This looks like another packaging issue. Now we are automating
the process, so that a custom initrd is generated for kdump kernel
and dump will be saved from the initrd itself. This custom initrd will be
only big enough contain the functionality to perform the job of bringing
the dump device up and saving the dump. User shall have to just say
"service kdump start" and initrd will be generated and kdump kernel will
be loaded into the memory.

Developers from NEC are also working on providing a capability to filter
the dump so that final core size is small and managable. In fact this
filtering also can be done from initrd itself.

>  - reserve more RAM for the kdump kernel (now I use "crashkernel=96M@16M")

Looks like big initramfs and running the init scripts in second kernel might
be taking some toll. We have been able to take dumps successfully by reserving
64MB on x86_64 boxes. Now with dump capturing capability from initrd, it
should become even better as we don't have to run user space init scripts
reducing memory pressure.

> 
> The distro is the AMD64 Debian etch, with two vanilla 2.6.18 kernels: one 
> "regular" SMP kernel with CONFIG_KEXEC=y, which is what I use, and one UP 
> kernel with CONFIG_PROC_VMCORE=y and a different load address, which is 
> activated on a crash; other than that they are the same.

Now we are also looking at a completely relocatable kernel so that the need
to build a separate kernel for dumping purposes is gone. Few weeks back
Eric posted the patches on LKML for RFC and testing purposes. We have been
testing out those patches and fixing few bugs. In a couple of weeks we
should be able to re-post the patches for another round of review. So
things should become even better once relocatable kernel is in.

> 
> The crude script below is run at boot time and configures kdump properly 
> for me. All oopses invoke panic ("kernel.panic_on_oops = 1" in 
> /etc/sysctl.conf), and thus the kdump kernel is activated. This kernel 
> boots on the very same root fs, runs the script below again, this time to 
> save the dump and reboot to the regular kernel. I did not try to have the 
> kdump kernel reset the vga console, so it stays in graphic mode and shows 
> garbage until the final reboot.

Now latest kexec-tools has one patch to re-initialize frame buffer over
kdump boot but I am not sure what can we do if we are in X at the time
of crash? One suggestion was that at-least just blink the keyboard lights
while dump is being saved. 

So at this point of time, I am not sure how can we reset the display if
user was in graphic mode at the time of crash.  
> 
> Now from an oops to being back to work, I would say it takes about 2-3 
> minutes (the longer part is the cold reboot, then the 30" timeout on the 
> Windows loader ;-)). I would certainly not do kernel development on my 
> desktop (backups? what backups?), but oopses do occur from time to time 
> (oh the joys of SMP on a desktop), and it's great to have a dump the first 
> time I see one.
> 
> >>Oh and I wish I could use gdb on a kdump core. :-)
> >

Currently we can use gdb but only for linearly mapped region. You are right
its just a matter of re-generating the elf headers and remap the vmalloc
areas to enable module debugging in gdb. I can not do it after the crash
so probably the best place would be do it in user space. A program can
read /proc/vmcore and regenerate the headers for enabling module debugging
with gdb.

> >Would be nice, but this is much better than what we usually get, no?
> 
> I'm used to having LKCD for debugging at work (on IA64), it has raised my 
> expectations. :-)
> 
> But using gdb would be so much nicer, and I'm sure we're close to being 
> able to use it, it's probably only a matter of messing a bit with the ELF 
> image provided by kdump (it's really a good old core dump): ie. remapping 
> vmalloc'ed memory pages, so modules can be accessed, and ideally adding 
> processor context for each task (currently there is one per CPU). Crash 
> does all this stuff internally and has other nice commands, but I don't 
> find it quite as comfortable as gdb (with a few good scripts gdb would 
> certainly be on par with crash).
> 

Hmm.. Crash vs gdb is an interesting issue. I have not used
gdb very extensively on core dumps, but with my limited experiece, I 
found "crash" to be more friendly. Crash has got so many in-built commands
tailored for kernel debugging and gdb lacks all those. Yes, we can write
gdb scripts to implement those, but last time Alaxender Nyberg wrote
few gdb scripts to dump all the threads and it was so slow.

Look at Documentation/kdump/gdbmacros.txt  

So what's issue with crash? Is it just a matter of being more familiar
with gdb or gdb has got advantages over "crash" when it comes to kernel
debugging?

Thanks
Vivek
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/