Subject: 2.6.24 BUG: soft lockup - CPU#X

From: Matheos Worku <[email-masked]>
Date: Wed, 26 Mar 2008 09:46:32 -0700

All,

While running iperf (transmit) on NIU driver I get several soft lockup 
errors.  I have seen similar soft lockup errors on kernels 2.6.18 and 
later (with other 10G NIC drivers with LSO disabled) but it is more 
frequent/pronounced on 2.6.24 kernel.  oprofile is not giving  much clue 
as where the cpus are spending time (atleast doesn't indicate any thing 
abnormal in the driver(s)). The lockup location as indicated by the 
error message is not consistent and varies from time time and could be 
outside the driver as well.  I have attached several lockup  error 
traces and corresponding profile data. Any clues?

Regards
matheos

Test system is 8xdual core AMD opteron, with 8 cpus booted.

[root@nsn57-110 ~]# more /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 33
model name      : Dual Core AMD Opteron(tm) Processor 885
stepping        : 2
cpu MHz         : 1000.000
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext 
fxsr_opt lm 3dnowext 3dnow rep_good pni lahf_lm cmp_legacy
bogomips        : 2019.84
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

nsn57-110 login: BUG: soft lockup - CPU#3 stuck for 11s! [iperf:9064]
CPU 3:
Modules linked in: oprofile niu nfs lockd nfs_acl autofs4 hidp rfcomm 
l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm rdma_cm 
iw_cm ib_addr ib_srp scsi_transport_srp  ib_cm ib_ipoib ib_sa ib_uverbs 
ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp parport 
joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core 
dm_snapshotdm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase 
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 9064, comm: iperf Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff80406fb8>]  [<ffffffff80406fb8>] 
pfifo_fast_dequeue+0x3b/0x59
RSP: 0018:ffff8101bece5a20  EFLAGS: 00000286
RAX: 00000000000005ea RBX: ffff8101ced38000 RCX: ffff8102f397a990
RDX: ffff8102facead10 RSI: 000000000000004d RDI: ffff8102faceac00
RBP: ffff8103e10ab700 R08: 0000000000000003 R09: 0400000000000000
R10: 00000003e9d5d700 R11: ffffffff80221995 R12: ffffffff882ee016
R13: ffff8103e9d5d100 R14: 0000000000000006 R15: ffff8103e58a8ed0
FS:  000000004700d940(0063) GS:ffff8101fb072e00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000032e6e95770 CR3: 00000001d051a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff80406d65>] __qdisc_run+0x31/0x174
 [<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
 [<ffffffff8041a17c>] ip_queue_xmit+0x343/0x396
 [<ffffffff8028596d>] add_partial_tail+0x12/0x3f
 [<ffffffff80267f23>] get_pageblock_flags_group+0x3e/0x7f
 [<ffffffff80267f23>] get_pageblock_flags_group+0x3e/0x7f
 [<ffffffff80429176>] tcp_transmit_skb+0x69f/0x6dc
 [<ffffffff8042aee0>] __tcp_push_pending_frames+0x79f/0x878
 [<ffffffff80427d69>] tcp_rcv_established+0x108/0x6f3
 [<ffffffff8042dab0>] tcp_v4_do_rcv+0x30/0x3a1
 [<ffffffff803ece3e>] release_sock+0x60/0xbf
 [<ffffffff8042048d>] tcp_sendmsg+0x968/0xa67
 [<ffffffff803ea812>] sock_aio_write+0xf8/0x110
 [<ffffffff8028c3b0>] do_sync_write+0xc9/0x10c
 [<ffffffff8027302b>] handle_mm_fault+0x668/0x69b
 [<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8028cb6b>] vfs_write+0xd9/0x14f
 [<ffffffff8028d0ae>] sys_write+0x45/0x6e
 [<ffffffff8020b7fc>] tracesys+0xdc/0xe1

BUG: soft lockup - CPU#3 stuck for 11s! [iperf:9064]
CPU 3:
Modules linked in: oprofile niu nfs lockd nfs_acl autofs4 hidp rfcomm 
l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm rdma_cm 
iw_cm ib_addr ib_srp scsi_transport_srp  ib_cm ib_ipoib ib_sa ib_uverbs 
ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp parport 
joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core 
dm_snapshotdm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase 
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 9064, comm: iperf Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff802219c8>]  [<ffffffff802219c8>] 
gart_map_single+0x33/0x70
RSP: 0018:ffff8101bece5968  EFLAGS: 00000246
RAX: 00000002f656e100 RBX: 0000000000000600 RCX: 0000000000000001
RDX: 0000000000000600 RSI: ffff8102f656e100 RDI: ffff8102f656e100
RBP: 0000000000000000 R08: 4000111900000000 R09: 0050000000000000
R10: 0000000000000000 R11: ffffffff80221995 R12: 0000000000000286
R13: ffff8103fba11200 R14: ffff8103e9913100 R15: ffffffff8026fc8b
FS:  000000004700d940(0063) GS:ffff8101fb072e00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000032e6e95770 CR3: 00000001d051a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff802219c1>] gart_map_single+0x2c/0x70
 [<ffffffff882edf65>] :niu:niu_start_xmit+0x339/0x5af
 [<ffffffff80221995>] gart_map_single+0x0/0x70
 [<ffffffff803f5e2b>] dev_hard_start_xmit+0x1d2/0x246
 [<ffffffff80406fa1>] pfifo_fast_dequeue+0x24/0x59
 [<ffffffff80406dab>] __qdisc_run+0x77/0x174
 [<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
 [<ffffffff8041a17c>] ip_queue_xmit+0x343/0x396
 [<ffffffff8028596d>] add_partial_tail+0x12/0x3f
 [<ffffffff80267f23>] get_pageblock_flags_group+0x3e/0x7f
 [<ffffffff80267f23>] get_pageblock_flags_group+0x3e/0x7f
 [<ffffffff80429176>] tcp_transmit_skb+0x69f/0x6dc
 [<ffffffff8042aee0>] __tcp_push_pending_frames+0x79f/0x878
 [<ffffffff80427d69>] tcp_rcv_established+0x108/0x6f3
 [<ffffffff8042dab0>] tcp_v4_do_rcv+0x30/0x3a1
 [<ffffffff803ece3e>] release_sock+0x60/0xbf
 [<ffffffff8042048d>] tcp_sendmsg+0x968/0xa67
 [<ffffffff803ea812>] sock_aio_write+0xf8/0x110
 [<ffffffff8028c3b0>] do_sync_write+0xc9/0x10c
 [<ffffffff8027302b>] handle_mm_fault+0x668/0x69b
 [<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8028cb6b>] vfs_write+0xd9/0x14f
 [<ffffffff8028d0ae>] sys_write+0x45/0x6e
 [<ffffffff8020b7fc>] tracesys+0xdc/0xe1

CPU: AMD64 processors, speed 1000 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a 
unit mask of 0x00 (No unit mask) count 100000
Samples on CPU 0
Samples on CPU 1
Samples on CPU 2
Samples on CPU 3
Samples on CPU 4
Samples on CPU 5
Samples on CPU 6
Samples on CPU 7
samples  %        samples  %        samples  %        samples  %        
samples  %        samples  %        samples  %        samples  %        
image name               app name                 symbol name
35847    10.0121  15879     5.5917  83932     5.8174  176125   19.8328  
339527   17.1392  290339   14.6522  316262   17.6201  391473   17.5484  
vmlinux
                vmlinux                  _raw_spin_lock
33542     9.3683  13947     4.9113  4596      0.3186  54781     6.1687  
5327      0.2689  4976      0.2511  14967     0.8339  13636     0.6113  
niu.ko
                niu                      niu_start_xmit
20350     5.6838  21907     7.7144  54143     3.7527  69738     7.8530  
239428   12.0862  208502   10.5222  285560   15.9096  184698    8.2794  
vmlinux
                vmlinux                  copy_user_generic_string
13818     3.8594  12931     4.5535  9246      0.6408  24130     2.7172  
66874     3.3758  60730     3.0648  52617     2.9315  83159     3.7277  
vmlinux
                vmlinux                  tcp_ack
12375     3.4563  10647     3.7492  6462      0.4479  16099     1.8129  
45097     2.2765  41207     2.0795  33129     1.8457  56178     2.5183  
vmlinux
                vmlinux                  __tcp_select_window
11995     3.3502  8815      3.1041  5267      0.3651  15376     1.7314  
42396     2.1401  39193     1.9779  29124     1.6226  57981     2.5991  
vmlinux
                vmlinux                  __copy_skb_header
10350     2.8908  7941      2.7964  11467     0.7948  27716     3.1210  
64291     3.2454  56789     2.8659  52322     2.9151  80111     3.5911  
vmlinux
                vmlinux                  __tcp_push_pending_frames
9191      2.5671  2121      0.7469  1062      0.0736  13214     1.4880  
1180      0.0596  1195      0.0603  4256      0.2371  3715      0.1665  
vmlinux
                vmlinux                  pfifo_fast_dequeue
9141      2.5531  5952      2.0959  4051      0.2808  10781     1.2140  
30384     1.5338  28521     1.4393  22391     1.2475  41852     1.8761  
vmlinux
                vmlinux                  free_hot_cold_page
8095      2.2609  2488      0.8761  1697      0.1176  17957     2.0221  
1959      0.0989  1626      0.0821  5020      0.2797  4570      0.2049  
vmlinux
                vmlinux                  __qdisc_run
7922      2.2126  5276      1.8579  6511      0.4513  15156     1.7067  
43016     2.1714  39552     1.9960  32309     1.8001  52778     2.3659  
vmlinux
                vmlinux                  kmem_cache_free
7700      2.1506  6904      2.4312  7348      0.5093  17601     1.9820  
48613     2.4540  40715     2.0547  35488     1.9772  57007     2.5554  
vmlinux
                vmlinux                  ip_queue_xmit
7543      2.1068  6779      2.3872  23039     1.5969  50172     5.6497  
132212    6.6740  112382    5.6715  107581    5.9937  161639    7.2457  
vmlinux
                vmlinux                  dev_queue_xmit
7404      2.0679  8373      2.9485  193876   13.4377  2406      0.2709  
7016      0.3542  45116     2.2768  6269      0.3493  7755      0.3476  
vmlinux
                vmlinux                  skb_release_data
7369      2.0582  8834      3.1108  77903     5.3995  19075     2.1480  
47375     2.3915  61162     3.0866  36048     2.0084  57243     2.5660  
niu.ko
                niu                      niu_interrupt
7207      2.0129  8108      2.8552  56756     3.9338  16089     1.8117  
40057     2.0221  49214     2.4836  33659     1.8753  51010     2.2866  
niu.ko
                niu                      niu_poll
6914      1.9311  4099      1.4434  1058      0.0733  3846      0.4331  
11434     0.5772  11337     0.5721  6269      0.3493  19662     0.8814  
vmlinux
                vmlinux                  get_pageblock_flags_group
5883      1.6431  2139      0.7532  793       0.0550  10490     1.1812  
824       0.0416  779       0.0393  2586      0.1441  2272      0.1018  
vmlinux
                vmlinux                  sock_wfree
5855      1.6353  5284      1.8607  4817      0.3339  16055     1.8079  
35248     1.7793  32311     1.6306  25091     1.3979  49224     2.2065  
vmlinux
                vmlinux                  skb_copy_bits
5665      1.5822  6456      2.2734  12491     0.8658  15495     1.7448  
56038     2.8288  49153     2.4806  69654     3.8807  43563     1.9528  
vmlinux
                vmlinux                  tcp_sendmsg
5094      1.4228  3513      1.2371  1318      0.0914  4933      0.5555  
16424     0.8291  15262     0.7702  10637     0.5926  21869     0.9803  
vmlinux
                vmlinux                  tcp_set_skb_tso_segs
4958      1.3848  4813      1.6949  165349   11.4605  6882      0.7750  
16981     0.8572  44073     2.2242  9780      0.5449  20187     0.9049  
vmlinux
                vmlinux                  skb_release_all
4767      1.3314  4185      1.4737  16709     1.1581  36711     4.1339  
69252     3.4958  57613     2.9075  62623     3.4890  101863    4.5662  
vmlinux
                vmlinux                  pfifo_fast_enqueue
4678      1.3066  3174      1.1177  3419      0.2370  10220     1.1508  
27198     1.3729  24120     1.2172  20724     1.1546  37027     1.6598  
vmlinux
                vmlinux                  tcp_transmit_skb
4532      1.2658  3820      1.3452  12096     0.8384  38766     4.3653  
14964     0.7554  13336     0.6730  17124     0.9540  18142     0.8132  
vmlinux
                vmlinux                  _raw_spin_unlock
4440      1.2401  5000      1.7607  144565   10.0199  0              0  
0              0  40207     2.0291  0              0  0              0  
niu.ko
                niu                      niu_pci_unmap_single
4421      1.2348  4625      1.6287  77354     5.3615  4        4.5e-04  
2        1.0e-04  15377     0.7760  5        2.8e-04  1        4.5e-05  
vmlinux
                vmlinux                  kfree_skb
4264      1.1909  4006      1.4107  3971      0.2752  10260     1.1553  
30687     1.5491  27278     1.3766  24134     1.3446  36029     1.6151  
vmlinux
                vmlinux                  kmem_cache_alloc_node
4094      1.1435  3503      1.2336  3712      0.2573  7679      0.8647  
23417     1.1821  21114     1.0655  21857     1.2177  28370     1.2717  
vmlinux
                vmlinux                  kfree
2964      0.8278  2503      0.8814  2183      0.1513  7702      0.8673  
20190     1.0192  18987     0.9582  13373     0.7451  26578     1.1914  
vmlinux
                vmlinux                  tcp_v4_rcv
2822      0.7882  2502      0.8811  1811      0.1255  6020      0.6779  
20457     1.0327  19217     0.9698  15003     0.8359  27615     1.2379  
vmlinux
                vmlinux                  local_bh_enable
2815      0.7862  2004      0.7057  1814      0.1257  3225      0.3632  
14068     0.7101  13196     0.6659  8887      0.4951  17759     0.7961  
vmlinux
                vmlinux                  __slab_free
2731      0.7628  3904      1.3748  236108   16.3649  0              0  
0              0  51620     2.6051  0              0  0              0  
niu.ko
                niu                      release_tx_packet
2693      0.7522  897       0.3159  402       0.0279  3862      0.4349  
859       0.0434  740       0.0373  1872      0.1043  1605      0.0719  
vmlinux
                vmlinux                  dev_hard_start_xmit
2633      0.7354  2183      0.7687  2805      0.1944  5772      0.6500  
15536     0.7843  13160     0.6641  11056     0.6160  15467     0.6933  
vmlinux
                vmlinux                  ip_finish_output
2567      0.7170  1542      0.5430  3680      0.2551  4625      0.5208  
14971     0.7557  13076     0.6599  16538     0.9214  11771     0.5277  
vmlinux
                vmlinux                  get_page_from_freelist
2504      0.6994  1623      0.5715  1093      0.0758  3293      0.3708  
10124     0.5111  9167      0.4626  7378      0.4111  14708     0.6593  
vmlinux
                vmlinux                  skb_clone
2437      0.6807  816       0.2873  374       0.0259  3701      0.4168  
696       0.0351  618       0.0312  1463      0.0815  1226      0.0550  
vmlinux
                vmlinux                  gart_map_single
2424      0.6770  2060      0.7254  84        0.0058  206       0.0232  
543       0.0274  1183      0.0597  593       0.0330  820       0.0368  
vmlinux
                vmlinux                  eth_header
2007      0.5606  2392      0.8423  7641      0.5296  12076     1.3598  
38413     1.9391  33796     1.7055  40611     2.2626  34932     1.5659  
vmlinux
                vmlinux                  __alloc_skb
1958      0.5469  856       0.3014  2011      0.1394  6385      0.7190  
19324     0.9755  16077     0.8113  11571     0.6447  22939     1.0283  
vmlinux
                vmlinux                  __phys_addr
1868      0.5217  1310      0.4613  4119      0.2855  5267      0.5931  
17008     0.8586  15217     0.7679  19030     1.0602  15463     0.6932  
vmlinux
                vmlinux                  memset_c
1834      0.5122  558       0.1965  344       0.0238  4411      0.4967  
572       0.0289  528       0.0266  1401      0.0781  1332      0.0597  
niu.ko
                niu                      niu_set_txd
1702      0.4754  1381      0.4863  148       0.0103  859       0.0967  
2748      0.1387  2943      0.1485  1009      0.0562  5099      0.2286  
vmlinux
                vmlinux                  free_pages_bulk
1402      0.3916  1614      0.5684  1566      0.1085  5598      0.6304  
10886     0.5495  10396     0.5246  8331      0.4642  13281     0.5953  
vmlinux
                vmlinux                  __pskb_pull_tail
1321      0.3690  1058      0.3726  1253      0.0868  2939      0.3310  
9312      0.4701  7869      0.3971  6736      0.3753  10654     0.4776  
vmlinux
                vmlinux                  tcp_rcv_established
1314      0.3670  11301     3.9795  8380      0.5808  2592      0.2919  
11391     0.5750  1007      0.0508  4551      0.2536  2900      0.1300  
oprofiled                oprofiled                (no symbols)
1311      0.3662  1701      0.5990  1919      0.1330  527       0.0593  
5967      0.3012  5769      0.2911  11960     0.6663  1958      0.0878  
vmlinux
                vmlinux                  __rmqueue
1218      0.3402  635       0.2236  287       0.0199  486       0.0547  
1996      0.1008  2162      0.1091  1289      0.0718  2629      0.1178  
vmlinux
                vmlinux                  __mod_zone_page_state
1177      0.3287  1256      0.4423  1969      0.1365  6394      0.7200  
15622     0.7886  14234     0.7183  11927     0.6645  20082     0.9002  
vmlinux
                vmlinux                  netif_receive_skb
1072      0.2994  1093      0.3849  2235      0.1549  5437      0.6122  
13295     0.6711  12614     0.6366  11498     0.6406  17651     0.7912  
vmlinux
                vmlinux                  ip_route_input
1043      0.2913  680       0.2395  737       0.0511  1932      0.2176  
5400      0.2726  5268      0.2659  4528      0.2523  6865      0.3077  
vmlinux
                vmlinux                  put_page
1034      0.2888  447       0.1574  2398      0.1662  4145      0.4668  
26089     1.3170  24902     1.2567  13333     0.7428  14704     0.6591  
vmlinux
                vmlinux                  __delay
996       0.2782  751       0.2645  21850     1.5144  258       0.0291  
809       0.0408  6957      0.3511  557       0.0310  1027      0.0460  
vmlinux
                vmlinux                  gart_unmap_single
953       0.2662  711       0.2504  1331      0.0923  3741      0.4213  
16598     0.8379  14805     0.7471  5559      0.3097  17815     0.7986  
vmlinux
                vmlinux                  ip_local_deliver
947       0.2645  250       0.0880  1315      0.0911  56        0.0063  
876       0.0442  1013      0.0511  1247      0.0695  255       0.0114  
iperf
                iperf                    reporter_condprintstats
930       0.2597  939       0.3307  936       0.0649  3379      0.3805  
6289      0.3175  6103      0.3080  5442      0.3032  9742      0.4367  
niu.ko
                niu                      niu_find_rxpage
885       0.2472  1082      0.3810  76280     5.2870  533       0.0600  
1716      0.0866  21803     1.1003  1219      0.0679  2108      0.0945  
vmlinux
                vmlinux                  __kfree_skb
876       0.2447  775       0.2729  869       0.0602  2878      0.3241  
7079      0.3573  6356      0.3208  3648      0.2032  8924      0.4000  
vmlinux
                vmlinux                  ip_local_deliver_finish
864       0.2413  552       0.1944  3143      0.2178  6539      0.7363  
18190     0.9182  16338     0.8245  12059     0.6719  20170     0.9042  
vmlinux
                vmlinux                  pfn_to_page
831       0.2321  815       0.2870  71        0.0049  167       0.0188  
440       0.0222  698       0.0352  422       0.0235  650       0.0291  
vmlinux
                vmlinux                  _raw_read_lock
779       0.2176  679       0.2391  2057      0.1426  2551      0.2873  
11112     0.5609  10998     0.5550  13292     0.7405  10674     0.4785  
vmlinux
                vmlinux                  __kmalloc_node_track_caller
700       0.1955  358       0.1261  46        0.0032  81        0.0091  
206       0.0104  294       0.0148  271       0.0151  262       0.0117  
vmlinux
                vmlinux                  local_bh_enable_ip
682       0.1905  853       0.3004  29        0.0020  82        0.0092  
117       0.0059  238       0.0120  170       0.0095  236       0.0106  
vmlinux
                vmlinux                  neigh_resolve_output
640       0.1788  398       0.1402  457       0.0317  838       0.0944  
2527      0.1276  2174      0.1097  3696      0.2059  4044      0.1813  
vmlinux
                vmlinux                  tcp_init_tso_segs
639       0.1785  428       0.1507  541       0.0375  348       0.0392  
1888      0.0953  1729      0.0873  3567      0.1987  838       0.0376  
vmlinux
                vmlinux                  __rmqueue_smallest
626       0.1748  485       0.1708  678       0.0470  1686      0.1899  
3472      0.1753  3666      0.1850  2181      0.1215  4042      0.1812  
vmlinux
                vmlinux                  page_to_pfn
596       0.1665  571       0.2011  202       0.0140  434       0.0489  
1472      0.0743  1303      0.0658  1399      0.0779  1285      0.0576  
vmlinux
                vmlinux                  new_slab
535       0.1494  838       0.2951  854       0.0592  2850      0.3209  
7345      0.3708  5461      0.2756  8328      0.4640  7682      0.3444  
vmlinux
                vmlinux                  ip_rcv_finish
519       0.1450  505       0.1778  3538      0.2452  3250      0.3660  
16741     0.8451  16927     0.8542  23509     1.3098  14544     0.6520  
vmlinux
                vmlinux                  __slab_alloc
509       0.1422  438       0.1542  467       0.0324  1647      0.1855  
4471      0.2257  3949      0.1993  2904      0.1618  5552      0.2489  
vmlinux
                vmlinux                  ___pskb_trim
497       0.1388  402       0.1416  422       0.0292  1373      0.1546  
3719      0.1877  3156      0.1593  2957      0.1647  4635      0.2078  
vmlinux
                vmlinux                  tcp_rtt_estimator
487       0.1360  348       0.1225  502       0.0348  160       0.0180  
1626      0.0821  1474      0.0744  2784      0.1551  684       0.0307  
vmlinux
                vmlinux                  rmqueue_bulk
455       0.1271  360       0.1268  355       0.0246  483       0.0544  
3794      0.1915  3716      0.1875  3955      0.2203  5553      0.2489  
vmlinux
                vmlinux                  add_partial_tail

======

nsn57-110 login: BUG: soft lockup - CPU#3 stuck for 11s! [iperf:10566]
CPU 3:
Modules linked in: oprofile niu nfs lockd nfs_acl autofs4 hidp rfcomm 
l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm rdma_cm 
iw_cm ib_addr ib_srp scsi_transport_srp ib_cm ib_ipoib ib_sa ib_uverbs 
ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp parport 
joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core 
dm_snapshot dm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase 
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 10566, comm: iperf Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff80316d29>]  [<ffffffff80316d29>] 
_raw_spin_lock+0x7d/0xf6
RSP: 0018:ffff8101f99bb9f8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 000000000000000e RCX: 00000000a0c5b122
RDX: 0000000000003b06 RSI: 0000000000000033 RDI: 0000000000000001
RBP: ffffffff803eecfd R08: 0000000000000003 R09: 0400000000000000
R10: 00000003e24a9700 R11: ffffffff80221995 R12: ffffffff802219c1
R13: ffff8103db913450 R14: ffff8101ced38000 R15: ffff8103dd5c0700
FS:  000000004580a940(0063) GS:ffff8101fb072e00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b762a36c050 CR3: 00000001c6950000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff80316d3a>] _raw_spin_lock+0x8e/0xf6
 [<ffffffff80406e8a>] __qdisc_run+0x156/0x174
 [<ffffffff80406dd2>] __qdisc_run+0x9e/0x174
 [<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
 [<ffffffff8041a17c>] ip_queue_xmit+0x343/0x396
 [<ffffffff80429176>] tcp_transmit_skb+0x69f/0x6dc
 [<ffffffff8042aee0>] __tcp_push_pending_frames+0x79f/0x878
 [<ffffffff80427d69>] tcp_rcv_established+0x108/0x6f3
 [<ffffffff8042dab0>] tcp_v4_do_rcv+0x30/0x3a1
 [<ffffffff803ece3e>] release_sock+0x60/0xbf
 [<ffffffff8042048d>] tcp_sendmsg+0x968/0xa67
 [<ffffffff8026fc8b>] zone_statistics+0x3f/0x60
 [<ffffffff803ea812>] sock_aio_write+0xf8/0x110
 [<ffffffff8042048d>] tcp_sendmsg+0x968/0xa67
 [<ffffffff8028c3b0>] do_sync_write+0xc9/0x10c
 [<ffffffff8027302b>] handle_mm_fault+0x668/0x69b
 [<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8022b995>] __dequeue_entity+0x1c/0x32
 [<ffffffff8045e4a1>] _spin_unlock_irq+0x9/0xc
 [<ffffffff802314ce>] finish_task_switch+0x37/0x83
 [<ffffffff8028cb6b>] vfs_write+0xd9/0x14f
 [<ffffffff8028d0ae>] sys_write+0x45/0x6e
 [<ffffffff8020b7fc>] tracesys+0xdc/0xe1

BUG: soft lockup - CPU#3 stuck for 11s! [iperf:10557]
CPU 3:
Modules linked in: oprofile niu nfs lockd nfs_acl autofs4 hidp rfcomm 
l2cap bluetooth sunrpc ipv6 cpufreq_ondemand rdma_ucm ib_ucm rdma_cm 
iw_cm ib_addr ib_srp scsi_transport_srp  ib_cm ib_ipoib ib_sa ib_uverbs 
ib_umad ib_mad ib_core dm_multipath battery ac parport_pc lp parport 
joydev sr_mod sg e1000 button i2c_nforce2 pcspkr shpchp i2c_core 
dm_snapshotdm_zero dm_mirror dm_mod usb_storage mptsas mptscsih mptbase 
scsi_transport_sas sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Pid: 10557, comm: iperf Not tainted 2.6.24-mati #3
RIP: 0010:[<ffffffff882edf6c>]  [<ffffffff882edf6c>] 
:niu:niu_start_xmit+0x340/0x5af
RSP: 0018:ffff8101ac1f5998  EFLAGS: 00000293
RAX: 00000003ec4b7100 RBX: ffff8103e54cb610 RCX: 00000003ec4b7100
RDX: 00000fffffffffff RSI: ffff8103ec4b7100 RDI: ffff8103ec4b7100
RBP: ffffffff80221995 R08: ffffffff882edf84 R09: 0050000000000000
R10: 0000000000000000 R11: ffffffff80221995 R12: ffff8102f04d4ed0
R13: 0050000000000000 R14: 0000000000000000 R15: ffffffff80221995
FS:  0000000041001940(0063) GS:ffff8101fb072e00(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b762a36c050 CR3: 00000001c6950000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff882edf65>] :niu:niu_start_xmit+0x339/0x5af
 [<ffffffff803f5e2b>] dev_hard_start_xmit+0x1d2/0x246
 [<ffffffff80406e8a>] __qdisc_run+0x156/0x174
 [<ffffffff80406dab>] __qdisc_run+0x77/0x174
 [<ffffffff803f8139>] dev_queue_xmit+0x141/0x270
 [<ffffffff8041a17c>] ip_queue_xmit+0x343/0x396
 [<ffffffff8028596d>] add_partial_tail+0x12/0x3f
 [<ffffffff80267f23>] get_pageblock_flags_group+0x3e/0x7f
 [<ffffffff802868a5>] __slab_free+0x7a/0x2d5
 [<ffffffff80429176>] tcp_transmit_skb+0x69f/0x6dc
 [<ffffffff8042aee0>] __tcp_push_pending_frames+0x79f/0x878
 [<ffffffff80427d69>] tcp_rcv_established+0x108/0x6f3
 [<ffffffff8042dab0>] tcp_v4_do_rcv+0x30/0x3a1
 [<ffffffff803ece3e>] release_sock+0x60/0xbf
 [<ffffffff8042048d>] tcp_sendmsg+0x968/0xa67
 [<ffffffff802ec44d>] dummy_socket_sendmsg+0x0/0x3
 [<ffffffff803ea812>] sock_aio_write+0xf8/0x110
 [<ffffffff8042048d>] tcp_sendmsg+0x968/0xa67
 [<ffffffff8028c3b0>] do_sync_write+0xc9/0x10c
 [<ffffffff8027302b>] handle_mm_fault+0x668/0x69b
 [<ffffffff802480e0>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8022b995>] __dequeue_entity+0x1c/0x32
 [<ffffffff8028cb6b>] vfs_write+0xd9/0x14f
 [<ffffffff8028d0ae>] sys_write+0x45/0x6e
 [<ffffffff8020b7fc>] tracesys+0xdc/0xe1

samples  %        samples  %        samples  %        samples  %        
samples  %        samples  %        samples  %        samples  %        
image name               app name                 symbol name
7382      5.6251  88        0.0707  39484    17.0818  23        0.0662  
222       0.0663  272       0.0697  483       0.1890  541       0.1645  
vmlinux
                vmlinux                  skb_release_data
5373      4.0942  0              0  29098    12.5885  0              0  
0              0  0              0  0              0  0              0  
niu.ko
                niu                      niu_pci_unmap_single
5143      3.9190  2411      1.9382  1314      0.5685  625       1.8001  
14759     4.4101  7958      2.0390  25960    10.1604  26348     8.0092  
vmlinux
                vmlinux                  _raw_spin_lock
4794      3.6530  4198      3.3748  4016      1.7374  1899      5.4695  
55928    16.7117  29035     7.4392  37442    14.6543  40278    12.2436  
vmlinux
                iperf                    copy_user_generic_string
4660      3.5509  248       0.1994  24570    10.6296  37        0.1066  
767       0.2292  648       0.1660  1125      0.4403  1301      0.3955  
vmlinux
                vmlinux                  skb_release_all
4633      3.5304  1274      1.0242  0              0  137       0.3946  
2946      0.8803  2362      0.6052  4198      1.6430  4657      1.4156  
vmlinux
                vmlinux                  tcp_ack
4620      3.5205  0              0  13861     5.9966  0              0  
0              0  0              0  0              0  0              0  
vmlinux
                vmlinux                  kfree_skb
4557      3.4724  0              0  41087    17.7753  0              0  
0              0  0              0  0              0  0              0  
niu.ko
                niu                      release_tx_packet
4518      3.4427  1410      1.1335  0              0  112       0.3226  
2227      0.6654  1844      0.4725  3439      1.3460  3756      1.1417  
vmlinux
                vmlinux                  __tcp_select_window
4080      3.1090  799       0.6423  0              0  288       0.8295  
6662      1.9907  4099      1.0502  10426     4.0806  11095     3.3726  
vmlinux
                vmlinux                  dev_queue_xmit
4023      3.0655  779       0.6263  66        0.0286  84        0.2419  
1632      0.4877  1249      0.3200  2576      1.0082  3005      0.9135  
vmlinux
                vmlinux                  free_hot_cold_page
3980      3.0328  1254      1.0081  0              0  105       0.3024  
2095      0.6260  1723      0.4415  3342      1.3080  3640      1.1065  
vmlinux
                vmlinux                  __copy_skb_header
3751      2.8583  1133      0.9108  0              0  165       0.4752  
3165      0.9457  2097      0.5373  5288      2.0697  5409      1.6442  
vmlinux
                vmlinux                  __tcp_push_pending_frames
3504      2.6701  611       0.4912  68        0.0294  61        0.1757  
1193      0.3565  917       0.2349  1990      0.7789  2300      0.6991  
vmlinux
                vmlinux                  get_pageblock_flags_group
3274      2.4948  488       0.3923  12477     5.3979  61        0.1757  
1828      0.5462  1409      0.3610  4512      1.7659  5579      1.6959  
niu.ko
                niu                      niu_interrupt
3197      2.4361  8745      7.0303  1519      0.6572  5717     16.4660  
33696    10.0686  51587    13.2174  15227     5.9596  25442     7.7338  
vmlinux
                iperf                    _raw_spin_lock
3172      2.4171  953       0.7661  66        0.0286  103       0.2967  
2313      0.6911  1719      0.4404  3461      1.3546  3618      1.0998  
vmlinux
                vmlinux                  kmem_cache_free
3168      2.4140  493       0.3963  9766      4.2250  67        0.1930  
1712      0.5116  1153      0.2954  3655      1.4305  4368      1.3278  
niu.ko
                niu                      niu_poll
2693      2.0521  436       0.3505  1        4.3e-04  191       0.5501  
3350      1.0010  2245      0.5752  6262      2.4509  7050      2.1430  
vmlinux
                vmlinux                  pfifo_fast_enqueue
2402      1.8303  676       0.5434  0              0  78        0.2247  
1472      0.4398  1137      0.2913  2471      0.9671  3098      0.9417  
vmlinux
                vmlinux                  skb_copy_bits
2297      1.7503  904       0.7267  1        4.3e-04  111       0.3197  
2129      0.6362  1700      0.4356  3091      1.2098  4294      1.3053  
vmlinux
                vmlinux                  ip_queue_xmit
1944      1.4813  2137      1.7180  0              0  0              0  
0              0  0              0  32        0.0125  353       0.1073  
niu.ko
                niu                      niu_start_xmit
1809      1.3785  624       0.5016  0              0  48        0.1382  
1044      0.3120  783       0.2006  1766      0.6912  1912      0.5812  
vmlinux
                vmlinux                  tcp_set_skb_tso_segs
1779      1.3556  644       0.5177  2        8.7e-04  48        0.1382  
1207      0.3607  736       0.1886  1831      0.7166  2233      0.6788  
vmlinux
                vmlinux                  tcp_transmit_skb
1534      1.1689  356       0.2862  6         0.0026  41        0.1181  
933       0.2788  604       0.1548  1465      0.5734  1527      0.4642  
vmlinux
                vmlinux                  kfree
1368      1.0424  1213      0.9752  1054      0.4560  511       1.4718  
13895     4.1519  6859      1.7574  10097     3.9518  10450     3.1766  
vmlinux
                iperf                    tcp_sendmsg
1341      1.0218  320       0.2573  0              0  43        0.1238  
893       0.2668  763       0.1955  2021      0.7910  2460      0.7478  
vmlinux
                vmlinux                  kmem_cache_alloc_node
1260      0.9601  30        0.0241  13743     5.9456  6         0.0173  
65        0.0194  60        0.0154  103       0.0403  124       0.0377  
vmlinux
                vmlinux                  __kfree_skb
1187      0.9045  365       0.2934  0              0  33        0.0950  
670       0.2002  472       0.1209  967       0.3785  1199      0.3645  
vmlinux
                vmlinux                  skb_clone
1185      0.9030  231       0.1857  0              0  29        0.0835  
843       0.2519  624       0.1599  1343      0.5256  1783      0.5420  
vmlinux
                vmlinux                  local_bh_enable
1107      0.8435  288       0.2315  0              0  40        0.1152  
878       0.2624  792       0.2029  1299      0.5084  1456      0.4426  
vmlinux
                vmlinux                  tcp_v4_rcv
1094      0.8336  296       0.2380  0              0  39        0.1123  
480       0.1434  412       0.1056  947       0.3706  1777      0.5402  
vmlinux
                vmlinux                  ip_finish_output
1051      0.8009  217       0.1744  13        0.0056  28        0.0806  
438       0.1309  312       0.0799  769       0.3010  818       0.2487  
vmlinux
                vmlinux                  free_pages_bulk
1016      0.7742  7593      6.1041  203       0.0878  2511      7.2321  
518       0.1548  2476      0.6344  388       0.1519  369       0.1122  
niu.ko
                iperf                    niu_start_xmit
935       0.7125  202       0.1624  32        0.0138  23        0.0662  
495       0.1479  355       0.0910  899       0.3519  742       0.2256  
vmlinux
                vmlinux                  __slab_free
902       0.6873  537       0.4317  0              0  0              0  
0              0  0              0  10        0.0039  99        0.0301  
vmlinux
                vmlinux                  __qdisc_run
787       0.5997  0              0  0              0  0              0  
462       0.1380  1080      0.2767  0              0  0              0  
libperl.so               perl                     (no symbols)
708       0.5395  8         0.0064  4378      1.8940  0              0  
21        0.0063  25        0.0064  60        0.0235  42        0.0128  
vmlinux
                vmlinux                  gart_unmap_single
660       0.5029  153       0.1230  0              0  27        0.0778  
554       0.1655  477       0.1222  914       0.3577  1055      0.3207  
vmlinux
                vmlinux                  ip_route_input