Subject: Re: s2io: packet reordering with 2.6.25.4

From: Tom Quetchenbach <[email-masked]>
Date: Thu, 26 Jun 2008 17:05:35 -0700

Jarek Poplawski wrote:
> 
>> Out of curiosity: is there affinity set for eths' irqs or they are
>> balanced between cpus? (/proc/interrupts?)

No affinity set, but they're not exactly balanced either:

           CPU0       CPU1
  0:         73          0   IO-APIC-edge      timer
  1:          0          2   IO-APIC-edge      i8042
  2:          0          0    XT-PIC-XT        cascade
  8:          0          1   IO-APIC-edge      rtc
 12:          0          5   IO-APIC-edge      i8042
 25:         47      53074   IO-APIC-fasteoi   eth0
 26:         21        334   IO-APIC-fasteoi   eth1
 28:        358    1280440   IO-APIC-fasteoi   eth2 Neterion 10 Gigabit
Ethernet-SR PCI-X 2.0 DDR Adapter
NMI:          0          0   Non-maskable interrupts
LOC:     227793     227721   Local timer interrupts
RES:       6483       3213   Rescheduling interrupts
CAL:         40         31   function call interrupts
TLB:       1119        105   TLB shootdowns
TRM:          0          0   Thermal event interrupts
THR:          0          0   Threshold APIC interrupts
SPU:          0          0   Spurious interrupts
ERR:          0

Anyhow, it looks like setting CPU affinity for the card on the receiver
(echo 1 > /proc/irq/28/smp_affinity) does eliminate the reordering, at
least with UDP at up to 2.5 Gbit/s (which is as fast as iperf can spit
out packets on this machine). So, it looks like the "mystery" is solved :-)

> 
> ...and maybe ifconfig on TX side?

eth2      Link encap:Ethernet  HWaddr 00:0C:FC:00:0D:3F
          inet addr:10.4.72.3  Bcast:10.4.72.15  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:496 (496.0 b)  TX bytes:532 (532.0 b)
          Interrupt:28

Thanks
-Tom