Subject: kernel BUG at net/core/skbuff.c:95!

From: Jim Westfall <[email-masked]>
Date: Tue, 19 Feb 2008 16:20:01 -0800

Hi

I have a reproducible crash that can be triggered remotely at the layer2 
level.  Example crash outputs is as follows

kernel: skb_over_panic: text:c0541fc7 len:1000 put:997 head:c166ac00 data:c166ac2f tail:0xc166b017 end:0xc166ac80 dev:eth0
kernel: ------------[ cut here ]------------
kernel: kernel BUG at net/core/skbuff.c:95!
kernel: invalid opcode: 0000 [#1]
kernel: SMP 
kernel: Modules linked in:
kernel: CPU:    0
kernel: EIP:    0060:[<c052753e>]    Not tainted VLI
kernel: EFLAGS: 00010286   (2.6.23.16 #1)
kernel: EIP is at skb_over_panic+0x5e/0x70
kernel: eax: 00000076   ebx: df600460   ecx: 00000086   edx: 00000000
kernel: esi: 000003e5   edi: 000003e8   ebp: dcf652e0   esp: c087fed0
kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
kernel: Process swapper (pid: 0, ti=c087f000 task=c076e0e0 task.ti=c0810000)
kernel: Stack: c0746150 c0541fc7 000003e8 000003e5 c166ac00 c166ac2f c166b017 c166ac80 
kernel: c1588000 df600460 deb48418 c0541fcc d0000246 5429a9b7 c07f6468 c07f645c 
kernel: dcf652e0 00000000 c054209b dcf652e0 dcf652e0 00000003 deb48033 c0542105 
kernel: Call Trace:
kernel: [<c0541fc7>] llc_station_ac_send_test_r+0x137/0x190
kernel: [<c0541fcc>] llc_station_ac_send_test_r+0x13c/0x190
kernel: [<c054209b>] llc_station_next_state+0x7b/0xc0
kernel: [<c0542105>] llc_station_state_process+0x25/0x40
kernel: [<c053b37a>] llc_rcv+0x24a/0x290
kernel: [<c05308e2>] netif_receive_skb+0x242/0x360
kernel: [<c0358563>] e100_poll+0x193/0x4a0
kernel: [<c0118312>] run_rebalance_domains+0x112/0x410
kernel: [<c052d6f4>] net_rx_action+0x74/0x110
kernel: [<c011ff65>] __do_softirq+0x75/0xf0
kernel: [<c010580b>] do_softirq+0x5b/0xb0
kernel: [<c011c738>] profile_tick+0x38/0x60
kernel: [<c013d680>] handle_fasteoi_irq+0x0/0xd0
kernel: [<c01058da>] do_IRQ+0x7a/0xc0
kernel: [<c0101750>] default_idle+0x0/0x50
kernel: [<c01038c3>] common_interrupt+0x23/0x30
kernel: [<c0101750>] default_idle+0x0/0x50
kernel: [<c0101784>] default_idle+0x34/0x50
kernel: [<c01017d4>] cpu_idle+0x34/0x80
kernel: [<c0817962>] start_kernel+0x272/0x350
kernel: [<c0817390>] unknown_bootoption+0x0/0x1d0
kernel: =======================

skb_over_panic: text:c055be07 len:1000 put:997 head:de32a800 data:de32a82f tail:0xde32ac17 end:0xde32a880 dev:eth0
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:95!
invalid opcode: 0000 [#1] SMP 
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.24.2 #1)
EIP: 0060:[<c054078e>] EFLAGS: 00010282 CPU: 0
EIP is at skb_over_panic+0x5e/0x70
EAX: 00000076 EBX: de32daa0 ECX: 00000092 EDX: 00000000
ESI: 000003e5 EDI: 000003e8 EBP: df3cf200 ESP: c08a3ec0
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=c08a3000 task=c0794180 task.ti=c0833000)
Stack: c076ae10 c055be07 000003e8 000003e5 de32a800 de32a82f de32ac17 de32a880 
       df9f0800 de32daa0 df199418 c055be0c d0000246 5429a9b7 c081a8f4 c081a8e8 
       df3cf200 00000000 c055bedb df3cf200 df3cf200 00000003 df199033 c055bf45 
Call Trace:
 [<c055be07>] llc_station_ac_send_test_r+0x137/0x190
 [<c055be0c>] llc_station_ac_send_test_r+0x13c/0x190
 [<c055bedb>] llc_station_next_state+0x7b/0xc0
 [<c055bf45>] llc_station_state_process+0x25/0x40
 [<c055515a>] llc_rcv+0x24a/0x2c0
 [<c054a0bb>] netif_receive_skb+0x27b/0x3a0
 [<c03673c9>] e100_poll+0x179/0x410
 [<c05470f1>] net_rx_action+0x91/0x150
 [<c0120275>] __do_softirq+0x75/0xf0
 [<c0105a3b>] do_softirq+0x5b/0xb0
 [<c011ca78>] profile_tick+0x38/0x60
 [<c013ea80>] handle_fasteoi_irq+0x0/0xd0
 [<c0105b0a>] do_IRQ+0x7a/0xc0
 [<c0101730>] default_idle+0x0/0x50
 [<c0103903>] common_interrupt+0x23/0x30
 [<c0101730>] default_idle+0x0/0x50
 [<c0101764>] default_idle+0x34/0x50
 [<c01017c8>] cpu_idle+0x48/0xa0
 [<c083a942>] start_kernel+0x272/0x350
 [<c083a370>] unknown_bootoption+0x0/0x1d0
 =======================
Code: 00 00 89 44 24 14 8b 83 98 00 00 00 89 44 24 10 8b 43 50 89 74 24 0c 89 4c
 24 04 89 44 24 08 c7 04 24 10 ae 76 c0 e8 42 ba bd ff <0f> 0b eb fe 8d b4 26 00
 00 00 00 8d bc 27 00 00 00 00 56 53 83 
EIP: [<c054078e>] skb_over_panic+0x5e/0x70 SS:ESP 0068:c08a3ec0
Kernel panic - not syncing: Fatal exception in interrupt

The issue was originally noticed on 2.6.20.21 machines, when we had ~60 
servers all panic at the same time after a switch went crazy.

To reproduce download linkloop

http://freshmeat.net/projects/linkloop/

then run linkloop -d -s 1000 <target mac>

thanks
jim