2.6.23.1-smp kernel panic (network-related)

[Marek Kierdelewicz <marek@koba.pl>]

Hi there,

My company's (ISP) bussines model requires dynamic resizing of the
client queues. It's achieved by regenerating shaping rules and loading
then using batch mode of a tc binary. On production systems it's done
once every 1 or 2 minutes. Unfortunately this causes smp kernels to
panic. Non-smp kernels don't have such problems. Bug is around a long
time. I first noticed it after migrating to shaping configs that use
IFB, it might have been 2.6.18 "era".

Test scenario:

I've put together a test machine with configuration copied from
production router. I'm feeding the machine with production traffic
by means of port mirroring. Test machine has the same config as
production one (including mac addresses), so it tries to route the
incoming traffic.

Tested kernels were 2.6.31.1 and 2.6.20.6 (config from 2.6.20.6 is in
attachment). Both panicked if compiled with SMP support and work stable
otherwise. Problem occurs only with cyclic "shaping restarts". For the
test, reload operation using "tc -b ..." was executed in an infinite
loop. 

Box's CPU usage was approximately 15%. Panics occur with few hours -
one day intervals.


Below I attach the panic message captured via serial console:
----------------------------------------------------------------------
printk: 63 messages suppressed.
dst cache overflow
SMP
Modules linked in: ipt_LOG xt_hashlimit ipt_MASQUERADE ip_set_macipmap
ip_set_ipmap xt_state w83627hf hwmon_vid eeprom ifb ipt_SET ipt_set
ip_set ipip tunnel4 ip_gre e1000 i2c_i801 i2c_core CPU:    1 EIP:
0060:[<f255c08f>]    Not tainted VLI EFLAGS: 00010202   (2.6.23.1-smp
#2) EIP is at 0xf255c08f
eax: c196a000   ebx: 00000100   ecx: ef2f408c   edx: f3630000
esi: f0332029   edi: f255c08c   ebp: 00000001   esp: f3631ea8
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process tc (pid: 27695, ti=f3630000 task=f26a9560 task.ti=f3630000)
Stack: c01280ad f26a9560 00000001 f3631eb4 c0106f57 ef2f408c f0e8c08c
00000031 c0495308 0000000a c0124eb6 00000046 00000000 f7657740 f3630000
c0124f4c c180f120 c0114e62 00000000 00000000 f7bfd224 f74ed95c c01047e0
f7bfd224 Call Trace:
 [<c01280ad>] run_timer_softirq+0xf5/0x154
 [<c0106f57>] profile_pc+0x21/0x4a
 [<c0124eb6>] __do_softirq+0x5d/0xc1
 [<c0124f4c>] do_softirq+0x32/0x36
 [<c0114e62>] smp_apic_timer_interrupt+0x74/0x80
 [<c01047e0>] apic_timer_interrupt+0x28/0x30
 [<c014c82e>] remove_vma+0x1c/0x36
 [<c014c912>] exit_mmap+0xca/0xe1
 [<c011eedc>] mmput+0x1d/0x75
 [<c0123688>] do_exit+0x1be/0x68a
 [<c0123bc0>] sys_exit_group+0x0/0xd
 [<c0103d12>] sysenter_past_esp+0x5f/0x85
 =======================
Code: 00 52 41 f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 fa 00 00 00 ea 05 00 00 7f 00 00 00 34 a5 96 <c1> 8c
00 fd f3 a5 2b 09 00 d1 d8 32 c0 00 c0 55 f2 00 a0 96 c1 EIP:
[<f255c08f>] 0xf255c08f SS:ESP 0068:f3631ea8 Kernel panic - not
syncing: Fatal exception in interrupt
----------------------------------------------------------------------


-- 
Marek Kierdelewicz
Kierownik Działu Systemów Sieciowych, KoBa
Manager of Network Systems Department, KoBa
tel. (85) 7406466; fax. (85) 7406467
e-mail: admin@koba.pl

Attachment: kernel_config
Description: Binary data