Subject: Re: Kernel Panics in the network stack

From: Kevin Constantine <[email-masked]>
Date: Fri, 11 Dec 2009 14:16:45 -0800

On 12/11/2009 01:58 PM, Eric Dumazet wrote:
> Le 11/12/2009 22:50, Kevin Constantine a écrit :
>> On 12/11/2009 01:39 PM, Eric Dumazet wrote:
>>> Le 11/12/2009 22:09, Kevin Constantine a écrit :
>>>> Hey Everyone-
>>>>
>>>> I've been playing with an ARM based linuxstamp
>>>> http://opencircuits.com/Linuxstamp, and I've been seeing kernel panics
>>>> with both 2.6.28.3, and 2.6.30 within an hour or so of turning the
>>>> linuxstamp on.  The stack traces always seem to point at functions
>>>> related to networking.  I've pasted a couple of the crash outputs below.
>>>>    The linuxstamp isn't typically doing anything when the crashes occur,
>>>> in fact it'll crash even if I haven't logged in.
>>>>
>>>> If I ifconfig the interface down, the linuxstamp stays up indefinitely.
>>>>    Any pointers in one direction or another would be much appreciated.
>>>>
>>>> I'm not sure if this is the right audience to help out or if the arm
>>>> lists might be better.  But in any event, any help would be really
>>>> appreciated.
>>>>
>>>>
>>>> linuxstamp login: Unable to handle kernel paging request at virtual
>>>> address 183cb7b0
>>>> pgd = c0004000
>>>> [183cb7b0] *pgd=00000000
>>>> Internal error: Oops: 0 [#1] PREEMPT
>>>> Modules linked in:
>>>> CPU: 0    Not tainted  (2.6.30-00002-g0148992 #13)
>>>> PC is at 0x183cb7b0
>>>> LR is at __udp4_lib_rcv+0x43c/0x72c
>>>
>>> Could you disassemble your vmlinux file, __udp4_lib_rcv function
>>> around LR
>>> <c024ff4c>, to see which function was called ? This function then called
>>> a wrong pointer (0x183cb7b0 not a kernel pointer)
>>>
>>> Maybe a kernel stack corruption, or bad ram, ...
>>
>> The vmlinux file I'm using has probably changed a number of times since
>> then.  I'll get a fresh stack trace and disassemble that one.
>>
>> I've has this type of problem with several linuxstamps.  I'm only able
>> to trigger this panic when the linuxstamp is plugged into a cisco
>> catalyst gigabit switch.  Plugging it in at home into a consumer grade
>> 10/100 switch, the linuxstamp stays up indefinitely.
>>
>> Also worth noting, I'm seeing the error counts in ifconfig increase
>> steadily.
>
> Could be an error in NIC driver error path, this is a good point.
>
>>
>> eth0      Link encap:Ethernet  HWaddr ac:de:48:00:00:01
>>            inet addr:172.30.194.255  Bcast:172.30.207.255 Mask:255.255.240.0
>>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>            RX packets:42492 errors:1442 dropped:0 overruns:6 frame:784
>>            TX packets:1169 errors:0 dropped:0 overruns:0 carrier:0
>>            collisions:0 txqueuelen:1000
>>            RX bytes:3804651 (3.6 MiB)  TX bytes:106728 (104.2 KiB)
>>            Interrupt:24 Base address:0xc000
>>
>>
>
> Please give us more information, since this platform is not well known :)
>
> lsmod
> cat /proc/meminfo

debian:~# cat /proc/meminfo
MemTotal:          28732 kB
MemFree:           10260 kB
Buffers:            2304 kB
Cached:             9220 kB
SwapCached:            0 kB
Active:             8964 kB
Inactive:           4796 kB
Active(anon):       2292 kB
Inactive(anon):        0 kB
Active(file):       6672 kB
Inactive(file):     4796 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:          2256 kB
Mapped:             3308 kB
Slab:               3876 kB
SReclaimable:       1508 kB
SUnreclaim:         2368 kB
PageTables:          156 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       14364 kB
Committed_AS:      32372 kB
VmallocTotal:     989184 kB
VmallocUsed:        1136 kB
VmallocChunk:     986108 kB

> cat /proc/cpuinfo

debian:~# cat /proc/cpuinfo
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 89.53
Features        : swp half thumb
CPU implementer : 0x41
CPU architecture: 4T
CPU variant     : 0x1
CPU part        : 0x920
CPU revision    : 0

Hardware        : emQbit's ECB_AT91
Revision        : 0000
Serial          : 0000000000000000

> cat /proc/slabinfo  (after more than 2000 error count in ifconfig eth0)
debian:~# cat /proc/slabinfo 

slabinfo - version: 2.1 

# name            <active_objs> <num_objs> <objsize> <objperslab> 
<pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata 
<active_slabs> <num_slabs> <sharedavail>
rpc_buffers            8      8   2048    2    1 : tunables   24   12 
  0 : slabdata      4      4      0 

rpc_tasks              8     24    160   24    1 : tunables  120   60 
  0 : slabdata      1      1      0 

rpc_inode_cache        0      0    416    9    1 : tunables   54   27 
  0 : slabdata      0      0      0 

flow_cache             0      0     80   48    1 : tunables  120   60 
  0 : slabdata      0      0      0 

nfs_direct_cache       0      0     68   56    1 : tunables  120   60 
  0 : slabdata      0      0      0 

nfs_write_data        36     36    416    9    1 : tunables   54   27 
  0 : slabdata      4      4      0 

nfs_read_data         32     36    416    9    1 : tunables   54   27 
  0 : slabdata      4      4      0 

nfs_inode_cache        0      0    584    7    1 : tunables   54   27 
  0 : slabdata      0      0      0 

nfs_page               0      0     64   59    1 : tunables  120   60 
  0 : slabdata      0      0      0 

journal_handle         0      0     20  169    1 : tunables  120   60 
  0 : slabdata      0      0      0 

journal_head          14    126     60   63    1 : tunables  120   60 
  0 : slabdata      2      2      0 

revoke_table           2    254     12  254    1 : tunables  120   60 
  0 : slabdata      1      1      0 

revoke_record          0      0     16  203    1 : tunables  120   60 
  0 : slabdata      0      0      0 

ext2_inode_cache       0      0    416    9    1 : tunables   54   27 
  0 : slabdata      0      0      0 

ext3_inode_cache     443    450    432    9    1 : tunables   54   27 
  0 : slabdata     50     50      0 

ext3_xattr             0      0     44   84    1 : tunables  120   60 
  0 : slabdata      0      0      0 

reiser_inode_cache      0      0    368   10    1 : tunables   54   27 
   0 : slabdata      0      0      0 

configfs_dir_cache      0      0     52   72    1 : tunables  120   60 
   0 : slabdata      0      0      0 

kioctx                 0      0    160   24    1 : tunables  120   60 
  0 : slabdata      0      0      0 

kiocb                  0      0    160   24    1 : tunables  120   60 
  0 : slabdata      0      0      0 

inotify_event_cache      0      0     28  127    1 : tunables  120   60 
    0 : slabdata      0      0      0 

inotify_watch_cache      2     92     40   92    1 : tunables  120   60 
    0 : slabdata      1      1      0 

dnotify_cache          0      0     20  169    1 : tunables  120   60 
  0 : slabdata      0      0      0 

fasync_cache           0      0     16  203    1 : tunables  120   60 
  0 : slabdata      0      0      0 

shmem_inode_cache   2497   2500    392   10    1 : tunables   54   27 
  0 : slabdata    250    250      0 

nsproxy                0      0     24  145    1 : tunables  120   60 
  0 : slabdata      0      0      0 

posix_timers_cache      0      0    112   35    1 : tunables  120   60 
   0 : slabdata      0      0      0 

uid_cache              0      0     64   59    1 : tunables  120   60 
  0 : slabdata      0      0      0 

UNIX                   5     10    384   10    1 : tunables   54   27 
  0 : slabdata      1      1      0 

UDP-Lite               0      0    480    8    1 : tunables   54   27 
  0 : slabdata      0      0      0 

tcp_bind_bucket        1    113     32  113    1 : tunables  120   60 
  0 : slabdata      1      1      0 

inet_peer_cache        0      0     64   59    1 : tunables  120   60 
  0 : slabdata      0      0      0 

secpath_cache          0      0     32  113    1 : tunables  120   60 
  0 : slabdata      0      0      0 

xfrm_dst_cache         0      0    288   13    1 : tunables   54   27 
  0 : slabdata      0      0      0 

ip_fib_alias           0      0     16  203    1 : tunables  120   60 
  0 : slabdata      0      0      0 

ip_fib_hash            9    101     36  101    1 : tunables  120   60 
  0 : slabdata      1      1      0 

ip_dst_cache         388    435    256   15    1 : tunables  120   60 
  0 : slabdata     29     29      0 

arp_cache              1     30    128   30    1 : tunables  120   60 
  0 : slabdata      1      1      0 

RAW                    2      9    448    9    1 : tunables   54   27 
  0 : slabdata      1      1      0 

UDP                    1      8    480    8    1 : tunables   54   27 
  0 : slabdata      1      1      0 

tw_sock_TCP            0      0     96   40    1 : tunables  120   60 
  0 : slabdata      0      0      0 

request_sock_TCP       0      0     96   40    1 : tunables  120   60 
  0 : slabdata      0      0      0 

TCP                    2      3   1056    3    1 : tunables   24   12 
  0 : slabdata      1      1      0 

eventpoll_pwq          0      0     36  101    1 : tunables  120   60 
  0 : slabdata      0      0      0 

eventpoll_epi          0      0     96   40    1 : tunables  120   60 
  0 : slabdata      0      0      0 

sgpool-128             2      2   2048    2    1 : tunables   24   12 
  0 : slabdata      1      1      0 

sgpool-64              2      4   1024    4    1 : tunables   54   27 
  0 : slabdata      1      1      0 

sgpool-32              2      8    512    8    1 : tunables   54   27 
  0 : slabdata      1      1      0 

sgpool-16              2     15    256   15    1 : tunables  120   60 
  0 : slabdata      1      1      0 

sgpool-8               2     30    128   30    1 : tunables  120   60 
  0 : slabdata      1      1      0
scsi_data_buffer       0      0     20  169    1 : tunables  120   60 
  0 : slabdata      0      0      0
blkdev_queue          10     12   1216    3    1 : tunables   24   12 
  0 : slabdata      4      4      0
blkdev_requests        8     18    216   18    1 : tunables  120   60 
  0 : slabdata      1      1      0
blkdev_ioc            10     84     44   84    1 : tunables  120   60 
  0 : slabdata      1      1      0
bio-0                  2     30    128   30    1 : tunables  120   60 
  0 : slabdata      1      1      0
biovec-256             2      2   3072    1    1 : tunables   24   12 
  0 : slabdata      2      2      0
biovec-128             0      0   1536    2    1 : tunables   24   12 
  0 : slabdata      0      0      0
biovec-64              0      0    768    5    1 : tunables   54   27 
  0 : slabdata      0      0      0
biovec-16              0      0    192   20    1 : tunables  120   60 
  0 : slabdata      0      0      0
sock_inode_cache      18     22    352   11    1 : tunables   54   27 
  0 : slabdata      2      2      0
skbuff_fclone_cache     11     11    352   11    1 : tunables   54   27 
    0 : slabdata      1      1      0
skbuff_head_cache    100    180    192   20    1 : tunables  120   60 
  0 : slabdata      9      9      0
file_lock_cache        1     40     96   40    1 : tunables  120   60 
  0 : slabdata      1      1      0
proc_inode_cache     132    132    320   12    1 : tunables   54   27 
  0 : slabdata     11     11      0
sigqueue               1     27    144   27    1 : tunables  120   60 
  0 : slabdata      1      1      0
radix_tree_node      289    299    288   13    1 : tunables   54   27 
  0 : slabdata     23     23      0
bdev_cache             3      9    416    9    1 : tunables   54   27 
  0 : slabdata      1      1      0
sysfs_dir_cache     3902   3948     44   84    1 : tunables  120   60 
  0 : slabdata     47     47      0
mnt_cache             20     30    128   30    1 : tunables  120   60 
  0 : slabdata      1      1      0
filp                 210    210    128   30    1 : tunables  120   60 
  0 : slabdata      7      7      0
inode_cache         1560   1560    296   13    1 : tunables   54   27 
  0 : slabdata    120    120      0
dentry              4829   4830    128   30    1 : tunables  120   60 
  0 : slabdata    161    161      0
names_cache            1      1   4096    1    1 : tunables   24   12 
  0 : slabdata      1      1      0
buffer_head          614    648     52   72    1 : tunables  120   60 
  0 : slabdata      9      9      0
vm_area_struct       631    644     84   46    1 : tunables  120   60 
  0 : slabdata     14     14      0
mm_struct             20     20    384   10    1 : tunables   54   27 
  0 : slabdata      2      2      0
fs_cache              10    113     32  113    1 : tunables  120   60 
  0 : slabdata      1      1      0
files_cache           11     40    192   20    1 : tunables  120   60 
  0 : slabdata      2      2      0
signal_cache          45     45    448    9    1 : tunables   54   27 
  0 : slabdata      5      5      0
sighand_cache         36     36   1312    3    1 : tunables   24   12 
  0 : slabdata     12     12      0
task_struct           40     40    768    5    1 : tunables   54   27 
  0 : slabdata      8      8      0
cred_jar              56    120     96   40    1 : tunables  120   60 
  0 : slabdata      3      3      0
anon_vma             265    339      8  339    1 : tunables  120   60 
  0 : slabdata      1      1      0
pid                   35     59     64   59    1 : tunables  120   60 
  0 : slabdata      1      1      0
idr_layer_cache      127    130    148   26    1 : tunables  120   60 
  0 : slabdata      5      5      0
size-4194304           0      0 4194304    1 1024 : tunables    1    1 
   0 : slabdata      0      0      0
size-2097152           0      0 2097152    1  512 : tunables    1    1 
   0 : slabdata      0      0      0
size-1048576           0      0 1048576    1  256 : tunables    1    1 
   0 : slabdata      0      0      0
size-524288            0      0 524288    1  128 : tunables    1    1 
  0 : slabdata      0      0      0
size-262144            0      0 262144    1   64 : tunables    1    1 
  0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4 
  0 : slabdata      0      0      0
size-65536             0      0  65536    1   16 : tunables    8    4 
  0 : slabdata      0      0      0
size-32768             0      0  32768    1    8 : tunables    8    4 
  0 : slabdata      0      0      0
size-16384             0      0  16384    1    4 : tunables    8    4 
  0 : slabdata      0      0      0
size-8192              0      0   8192    1    2 : tunables    8    4 
  0 : slabdata      0      0      0
size-4096              4      4   4096    1    1 : tunables   24   12 
  0 : slabdata      4      4      0
size-2048             14     14   2048    2    1 : tunables   24   12 
  0 : slabdata      7      7      0
size-1024             40     40   1024    4    1 : tunables   54   27 
  0 : slabdata     10     10      0
size-512             173    200    512    8    1 : tunables   54   27 
  0 : slabdata     25     25      0
size-256              75     75    256   15    1 : tunables  120   60 
  0 : slabdata      5      5      0
size-192             669    680    192   20    1 : tunables  120   60 
  0 : slabdata     34     34      0
size-128             240    240    128   30    1 : tunables  120   60 
  0 : slabdata      8      8      0
size-96              919    920     96   40    1 : tunables  120   60 
  0 : slabdata     23     23      0
size-64              590    590     64   59    1 : tunables  120   60 
  0 : slabdata     10     10      0
size-32             3374   3390     32  113    1 : tunables  120   60 
  0 : slabdata     30     30      0
kmem_cache           105    120     96   40    1 : tunables  120   60 
  0 : slabdata      3      3      0
> ...
debian:~# dmesg 

Linux version 2.6.30-00002-g0148992 (kconstan@debian) (gcc version 4.3.2 
(Debian 4.3.2-1.1) ) #16 PREEMPT Fri Dec 11 13:48:45 PST 2009 

CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177 

CPU: VIVT data cache, VIVT instruction cache 

Machine: emQbit's ECB_AT91 

Memory policy: ECC disabled, Data cache writeback 

On node 0 totalpages: 8192 

free_area_init_node: node 0, pgdat c03a3340, node_mem_map c03c3000 

   Normal zone: 64 pages used for memmap 

   Normal zone: 0 pages reserved 

   Normal zone: 8128 pages, LIFO batch:0 

Clocks: CPU 179 MHz, master 59 MHz, main 18.432 MHz 

Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 
8128 

Kernel command line: mem=32M root=/dev/mmcblk0p1 ip=192.168.0.51 
console=ttyS0,115200n8 rootdelay=4 

NR_IRQS:192 

AT91: 96 gpio irqs in 3 banks
...
eth0: Link now 100-FullDuplex
eth0: AT91 ethernet at 0xfefbc000 int=24 100-FullDuplex (00:00:00:00:00:5b)
eth0: Micrel KS8721 PHY
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

-kevin