Subject: Re: Problem with ixgbe and TX locked on one cpu

From: Paweł Staszewski <[email-masked]>
Date: Wed, 19 Oct 2011 11:21:16 +0200

W dniu 2011-10-18 20:57, Jesse Brandeburg pisze:
> CC: e1000-devel
>
> 2011/10/14 Paweł Staszewski<[email-masked]>:
>> Hello
>>
>> I have weird problem with ixgbe and irq affinity / rx-tx queue assignment
> what application are you running, how are you using ixgbe? looks like
> a router.  is something changing the skb->rx_queue entry (like
> netfilter) or is there a layered device above ixgbe (bonding or ...) ?
Yes it is a router. - Quagga - BGP

There is few (10rules) iptables rules in INPUT - and nothing more

There are also namespaces running - and traffic is processed in namespaces.
no bonding

> why do your interrupts move after a long period? did you do it by
> hand? we recommend disabling irqbalance and hand tuning interrupts
> possibly with the set_irq_affinity.sh script.
Yes it was changed by hand to check why TX is processed on one interrupt.
Also now it is impossible to move this TX traffic to other interrupt.

All configuration for affinity is made by hand - no irqbalance running.

>> Statistics for my ethernet - ixgbe driver:
>> ethtool -S eth4
>> NIC statistics:
>>      rx_packets: 5815535848808
>>      tx_packets: 5811202378421
>>      rx_bytes: 4791001750842200
>>      tx_bytes: 4781190419358301
>>      rx_pkts_nic: 5815535848827
>>      tx_pkts_nic: 5811202378510
>>      rx_bytes_nic: 4837563124411799
>>      tx_bytes_nic: 4829987507084013
>>      lsc_int: 8
>>      tx_busy: 0
>>      non_eop_descs: 0
>>      rx_errors: 0
>>      tx_errors: 0
>>      rx_dropped: 0
>>      tx_dropped: 0
>>      multicast: 92494273
>>      broadcast: 268718206
>>      rx_no_buffer_count: 28829
>>      collisions: 0
>>      rx_over_errors: 0
>>      rx_crc_errors: 0
>>      rx_frame_errors: 0
>>      hw_rsc_aggregated: 0
>>      hw_rsc_flushed: 0
>>      fdir_match: 0
>>      fdir_miss: 0
>>      rx_fifo_errors: 0
>>      rx_missed_errors: 307051074
>>      tx_aborted_errors: 0
>>      tx_carrier_errors: 0
>>      tx_fifo_errors: 0
>>      tx_heartbeat_errors: 0
>>      tx_timeout_count: 0
>>      tx_restart_queue: 15926219
>>      rx_long_length_errors: 298
>>      rx_short_length_errors: 0
>>      tx_flow_control_xon: 0
>>      rx_flow_control_xon: 0
>>      tx_flow_control_xoff: 0
>>      rx_flow_control_xoff: 0
>>      rx_csum_offload_errors: 54173917
>>      alloc_rx_page_failed: 0
>>      alloc_rx_buff_failed: 0
>>      rx_no_dma_resources: 0
>>      tx_queue_0_packets: 68694825
>>      tx_queue_0_bytes: 9443750332
>>      tx_queue_1_packets: 8410961
>>      tx_queue_1_bytes: 2527763233
>>      tx_queue_2_packets: 14411252
>>      tx_queue_2_bytes: 1317132394
>>      tx_queue_3_packets: 15013508147
>>      tx_queue_3_bytes: 17364767277348
>>      tx_queue_4_packets: 62779891
>>      tx_queue_4_bytes: 63476596221
>>      tx_queue_5_packets: 11176001
>>      tx_queue_5_bytes: 2763600253
>>      tx_queue_6_packets: 4416357
>>      tx_queue_6_bytes: 611874984
>>      tx_queue_7_packets: 8933405
>>      tx_queue_7_bytes: 1837198524
>>      tx_queue_8_packets: 13292669
>>      tx_queue_8_bytes: 3241333510
>>      tx_queue_9_packets: 10747236
>>      tx_queue_9_bytes: 1805109931
>>      tx_queue_10_packets: 5795935258380
>>      tx_queue_10_bytes: 4763725304722245
>>      tx_queue_11_packets: 12073934
>>      tx_queue_11_bytes: 2982743045
>>      tx_queue_12_packets: 10523764
>>      tx_queue_12_bytes: 2637451199
>>      tx_queue_13_packets: 12480552
>>      tx_queue_13_bytes: 2434827407
>>      tx_queue_14_packets: 7401777
>>      tx_queue_14_bytes: 2413618099
>>      tx_queue_15_packets: 8269270
>>      tx_queue_15_bytes: 2854359576
>>      rx_queue_0_packets: 361373769507
>>      rx_queue_0_bytes: 298565751248279
>>      rx_queue_1_packets: 369901571908
>>      rx_queue_1_bytes: 303414679798160
>>      rx_queue_2_packets: 362508961738
>>      rx_queue_2_bytes: 299852439447157
>>      rx_queue_3_packets: 363449272013
>>      rx_queue_3_bytes: 299738390792515
>>      rx_queue_4_packets: 361876234461
>>      rx_queue_4_bytes: 297483366939732
>>      rx_queue_5_packets: 361402926316
>>      rx_queue_5_bytes: 297633876486533
>>      rx_queue_6_packets: 362261522767
>>      rx_queue_6_bytes: 298026696344647
>>      rx_queue_7_packets: 361248593301
>>      rx_queue_7_bytes: 296756459279986
>>      rx_queue_8_packets: 361654143416
>>      rx_queue_8_bytes: 298272433659520
>>      rx_queue_9_packets: 362781764710
>>      rx_queue_9_bytes: 298804803191595
>>      rx_queue_10_packets: 361386593064
>>      rx_queue_10_bytes: 297434987797644
>>      rx_queue_11_packets: 369886597895
>>      rx_queue_11_bytes: 302353350171712
>>      rx_queue_12_packets: 361582732276
>>      rx_queue_12_bytes: 298670408005971
>>      rx_queue_13_packets: 365248093536
>>      rx_queue_13_bytes: 302573023878287
>>      rx_queue_14_packets: 366571142073
>>      rx_queue_14_bytes: 302396739276514
>>      rx_queue_15_packets: 362401929830
>>      rx_queue_15_bytes: 299024344526029
>>
>> The problem is with queue 10
>>      tx_queue_10_packets: 5795935258380
>>      tx_queue_10_bytes: 4763725304722245
>>
>> as you can see most of the queue processing is used in queue 10
>> Average difference is 1,854271229903958e-6  - compared to other queues
>>
>> and the problem is that almost all TX packet processing is on one CPU
>> cat /proc/interrupts - in attached file
>>
>> Is this driver or kernel problem ?
>>
>> Kernel is: 2.6.38.2
>>
>> ixgbe driver is:
>> ethtool -i eth4
>> driver: ixgbe
>> version: 3.2.9-k2
>> firmware-version: 1.12-2
>> bus-info: 0000:04:00.0
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

--