Subject: Re: ixgbe question

From: Eric Dumazet <[email-masked]>
Date: Mon, 23 Nov 2009 11:37:20 +0100

Waskiewicz Jr, Peter P a écrit :
> On Mon, 23 Nov 2009, Eric Dumazet wrote:
> 
>> Hi Peter
>>
>> I tried a pktgen stress on 82599EB card and could not split RX load on multiple cpus.
>>
>> Setup is :
>>
>> One 82599 card with fiber0 looped to fiber1, 10Gb link mode.
>> machine is a HPDL380 G6 with dual quadcore E5530 @2.4GHz (16 logical cpus)
> 
> Can you specify kernel version and driver version?

Well, I forgot to mention I am only working with net-next-2.6 tree.

Ubuntu 9.10 kernel (Fedora Core 12 installer was not able to recognize disks on this machine :( )

ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 2.0.44-k2

> 
>> I use one pktgen thread sending to fiber0 one many dst IP, and checked that fiber1
>> was using many RX queues :
>>
>> grep fiber1 /proc/interrupts 
>> 117:       1301      13060          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-0
>> 118:        601       1402          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-1
>> 119:        634        832          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-2
>> 120:        601       1303          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-3
>> 121:        620       1246          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-4
>> 122:       1287      13088          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-5
>> 123:        606       1354          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-6
>> 124:        653        827          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-7
>> 125:        639        825          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-8
>> 126:        596       1199          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-9
>> 127:       2013      24800          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-10
>> 128:        648       1353          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-11
>> 129:        601       1123          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-12
>> 130:        625        834          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-13
>> 131:        665       1409          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-14
>> 132:       2637      31699          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1-TxRx-15
>> 133:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      fiber1:lsc
>>
>>
>>
>> But only one CPU (CPU1) had a softirq running, 100%, and many frames were dropped
>>
>> root@demodl380g6:/usr/src# ifconfig fiber0
>> fiber0    Link encap:Ethernet  HWaddr 00:1b:21:4a:fe:54  
>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>           Packets reçus:4 erreurs:0 :0 overruns:0 frame:0
>>           TX packets:309291576 errors:0 dropped:0 overruns:0 carrier:0
>>           collisions:0 lg file transmission:1000 
>>           Octets reçus:1368 (1.3 KB) Octets transmis:18557495682 (18.5 GB)
>>
>> root@demodl380g6:/usr/src# ifconfig fiber1
>> fiber1    Link encap:Ethernet  HWaddr 00:1b:21:4a:fe:55  
>>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>           Packets reçus:55122164 erreurs:0 :254169411 overruns:0 frame:0
>>           TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
>>           collisions:0 lg file transmission:1000 
>>           Octets reçus:3307330968 (3.3 GB) Octets transmis:1368 (1.3 KB)
> 
> I stay in the states too much.  I love seeing net stats in French.  :-)

Ok :)

> 
>>
>> How and when multi queue rx can really start to use several cpus ?
> 
> If you're sending one flow to many consumers, it's still one flow.  Even 
> using RSS won't help, since it requires differing flows to spread load  
> (5-tuple matches for flow distribution).

Hm... I can try varying both src and dst on my pktgen test.

Thanks