Re: Oops after 30 days of uptime

[Willy Tarreau <w@1wt.eu>]

On Fri, Sep 01, 2006 at 06:52:39PM +0200, Ondrej Zary wrote:
> Hello,
> my home router crashed after about a month. It does this sometimes but this 
> time I was able to capture the oops. Here is the result of running ksymoops 
> on it (took a photo of the screen and then manually converted to plain-text). 
> Does it look like a bug or something other?

I have another problem with your oops. It looks like you used a /proc/ksyms
from another running kernel. The symbol decoding does not match the code.
For instance, in the disassembled code, you'll see that two functions are
indicated for the same sequence of instructions (init_or_cleanup then
ip_conntrack_protocol_register). And the difference does not look like a
small offset, since neither of those functions seem to produce comparable
code here.

You should backup the /proc/ksyms from your currently running kernel, and
reuse it to decode the next oops when it occurs. BTW, could you provide
the full config file and tell us what version of GCC you're using ? Maybe
we can try to find the same code sequence in a module and identify it
without waiting for further oops.

> ksymoops 2.4.11 on i486 2.4.31.  Options used
>      -V (default)
>      -k /proc/ksyms (default)
>      -l /proc/modules (default)
>      -o /lib/modules/2.4.31/ (default)
>      -m System.map (specified)
> 
> Error (regular_file): read_ksyms stat /proc/ksyms failed
> No modules in ksyms, skipping objects
> No ksyms, skipping lsmod
> Unable to handle kernel paging request at virtual address c2000000
> *pde = 00000000
> Oops: 0000
> CPU:    0
> EIP:    0010:[<c01eeb9e>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010a96
> eax: db1cec0a   ebx: 7af4a90b   ecx: fff113e8   edx: 00000008
> esi: c1ffffe8   edi: c17835a4   ebp: c0b4b8b4   esp: c0227cd0
> ds: 0018   es: 0018   ss: 0018
> Process swapper (pid: 0, stackpage=c0227000)
> Stack: fd00a8c0 c17835a4 c01e6587 c0227ce8 00000008 0000a0c5 0000dff0 0000200f
>        c01e8757 0000dff0 0000200f 00005f3a 00000000 00000028 c1783590 c0b4b8b4
>        c01e7162 c1783590 00000028 c0b4b8b4 00000000 00000006 c0b4b810 00000000
> Call Trace:    [<c01e6587>] [<c01e8757>] [<c01e7162>] [<c01e72ea>] [<c017fda8>]
>   [<c01baca0>] [<c01e5ed4>] [<c01baca0>] [<c01e5fba>] [<c01baca0>] [<c01afe00>]
>   [<c01baca0>] [<c01baca0>] [<c01b00d0>] [<c01baca0>] [<c01b8270>] [<c01b9652>]
>   [<c01baca0>] [<c01b8270>] [<c01b82ba>] [<c01b0110>] [<c01b05dc>] [<c01b8214>]
>   [<c01b8270>] [<c01b7220>] [<c01b739b>] [<c01b7220>] [<c01b0110>] [<c01b70a6>]
>   [<c01b7220>] [<c01a87bb>] [<c01a885d>] [<c01a8970>] [<c011427a>] [<c01081cd>]
>   [<c0105250>] [<c010a3b8>] [<c0105250>] [<c0105273>] [<c01052d8>] [<c0105000>]
>   [<c0105027>]
> Code: 8b 5e 18 11 d8 8b 5e 1c 11 d8 8d 76 20 49 75 d3 83 d0 00 89
> 
> 
> >>EIP; c01eeb9e <init_or_cleanup+15e/160>   <=====
> 
> >>esp; c0227cd0 <bdf_prm+30/40>
> 
> Trace; c01e6587 <icmp_timestamp+47/f0>
> Trace; c01e8757 <inet_sock_release+57/80>
> Trace; c01e7162 <inet_rtm_newaddr+42/190>
> Trace; c01e72ea <devinet_ioctl+3a/680>
> Trace; c017fda8 <ei_start_xmit+248/260>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01e5ed4 <icmp_send+84/350>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01e5fba <icmp_send+16a/350>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01afe00 <nf_iterate+0/80>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01b00d0 <nf_hook_slow+80/150>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01b8270 <htb_timer+30/50>
> Trace; c01b9652 <htb_dump_class+22/260>
> Trace; c01baca0 <sfq_dequeue+e0/1c0>
> Trace; c01b8270 <htb_timer+30/50>
> Trace; c01b82ba <htb_rate_timer+2a/120>
> Trace; c01b0110 <nf_hook_slow+c0/150>
> Trace; c01b05dc <eth_header+2c/120>
> Trace; c01b8214 <htb_requeue+114/140>
> Trace; c01b8270 <htb_timer+30/50>
> Trace; c01b7220 <htb_classify+f0/110>
> Trace; c01b739b <htb_debug_dump+15b/300>
> Trace; c01b7220 <htb_classify+f0/110>
> Trace; c01b0110 <nf_hook_slow+c0/150>
> Trace; c01b70a6 <L2T+26/30>
> Trace; c01b7220 <htb_classify+f0/110>
> Trace; c01a87bb <netif_receive_skb+db/140>
> Trace; c01a885d <process_backlog+3d/110>
> Trace; c01a8970 <net_rx_action+40/110>
> Trace; c011427a <do_softirq+5a/b0>
> Trace; c01081cd <do_IRQ+9d/b0>
> Trace; c0105250 <default_idle+0/30>
> Trace; c010a3b8 <call_do_IRQ+5/d>
> Trace; c0105250 <default_idle+0/30>
> Trace; c0105273 <default_idle+23/30>
> Trace; c01052d8 <cpu_idle+38/50>
> Trace; c0105000 <_stext+0/0>
> Trace; c0105027 <rest_init+27/30>
> 
> Code;  c01eeb9e <init_or_cleanup+15e/160>
> 00000000 <_EIP>:
> Code;  c01eeb9e <init_or_cleanup+15e/160>   <=====
>    0:   8b 5e 18                  mov    0x18(%esi),%ebx   <=====
> Code;  c01eeba1 <ip_conntrack_protocol_register+1/70>
>    3:   11 d8                     adc    %ebx,%eax
> Code;  c01eeba3 <ip_conntrack_protocol_register+3/70>
>    5:   8b 5e 1c                  mov    0x1c(%esi),%ebx
> Code;  c01eeba6 <ip_conntrack_protocol_register+6/70>
>    8:   11 d8                     adc    %ebx,%eax
> Code;  c01eeba8 <ip_conntrack_protocol_register+8/70>
>    a:   8d 76 20                  lea    0x20(%esi),%esi
> Code;  c01eebab <ip_conntrack_protocol_register+b/70>
>    d:   49                        dec    %ecx
> Code;  c01eebac <ip_conntrack_protocol_register+c/70>
>    e:   75 d3                     jne    ffffffe3 <_EIP+0xffffffe3>
> Code;  c01eebae <ip_conntrack_protocol_register+e/70>
>   10:   83 d0 00                  adc    $0x0,%eax
> Code;  c01eebb1 <ip_conntrack_protocol_register+11/70>
>   13:   89 00                     mov    %eax,(%eax)
> 
>  <0>Kernel panic: Aiee, killing interrupt handler!
> 
> 1 error issued.  Results may not be reliable.
> 
> 
> -- 
> Ondrej Zary

Regards,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/