Subject: Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0

From: Stefan Priebe - Profihost AG <[email-masked]>
Date: Wed, 20 Jun 2012 11:50:52 +0200

Am 20.06.2012 11:47, schrieb Eric Dumazet:
> On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:
>
>> Sure. In that case i get 4Gbit/s in both variants. I also tried two
>> other different machines same result.
>>
>
> So 3.5 on receiver is the problem, it seems ?
Yes.

> And you checked all the stuff about irq affinities, i presume, since a
> lot of things might have changed between 2.6.32 and 3.5 ?

It is a single core E5 Xeon - i've set the affinity like this:

eth2 mask=1 for /proc/irq/83/smp_affinity
eth2 mask=2 for /proc/irq/84/smp_affinity
eth2 mask=4 for /proc/irq/85/smp_affinity
eth2 mask=8 for /proc/irq/86/smp_affinity
eth2 mask=10 for /proc/irq/87/smp_affinity
eth2 mask=20 for /proc/irq/88/smp_affinity
eth2 mask=40 for /proc/irq/89/smp_affinity
eth2 mask=80 for /proc/irq/90/smp_affinity

> cat /proc/interrupts

# cat /proc/interrupts
             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5 
       CPU6       CPU7
    0:        141          0          0          0          0          0 
          0          0   IO-APIC-edge      timer
    1:          1          8          0          0          0          0 
          0          0   IO-APIC-edge      i8042
    9:          0          0          0          0          0          0 
          0          0   IO-APIC-fasteoi   acpi
   12:          0          3          0          0          0          0 
          0          0   IO-APIC-edge      i8042
   14:          0          0          0          0          0          0 
          0          0   IO-APIC-edge      ide0
   15:          0          0          0          0          0          0 
          0          0   IO-APIC-edge      ide1
   16:          0          0         26          0          0          0 
          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
   23:          0          0         30          0          0          0 
          0          0   IO-APIC-fasteoi   ehci_hcd:usb2
   64:          0          0          0      81979          0          0 
          0          0   PCI-MSI-edge      ahci
   65:          0          0          0          1          0          0 
          0          0   PCI-MSI-edge      eth0
   66:          0          0          0          0       1090          0 
          0          0   PCI-MSI-edge      eth0-TxRx-0
   67:          0          0          0          0        411          0 
          0          0   PCI-MSI-edge      eth0-TxRx-1
   68:          0          0          0          0        592          0 
          0          0   PCI-MSI-edge      eth0-TxRx-2
   69:          0          0          0          0        472          0 
          0          0   PCI-MSI-edge      eth0-TxRx-3
   70:          0          0          0          0          0       1196 
          0          0   PCI-MSI-edge      eth0-TxRx-4
   71:          0          0          0          0          0        374 
          0          0   PCI-MSI-edge      eth0-TxRx-5
   72:          0          0          0          0          0        405 
          0          0   PCI-MSI-edge      eth0-TxRx-6
   73:          0          0          0          0          0        468 
          0          0   PCI-MSI-edge      eth0-TxRx-7
   83:      31278          0          0         65          0          0 
          0          0   PCI-MSI-edge      eth2-TxRx-0
   84:          0      36311          0          0         61          0 
          0          0   PCI-MSI-edge      eth2-TxRx-1
   85:          0          0      46189          0         61          0 
          0          0   PCI-MSI-edge      eth2-TxRx-2
   86:          0          0          0      28712         67          0 
          0          0   PCI-MSI-edge      eth2-TxRx-3
   87:          0          0          0          0      28089          0 
          0          0   PCI-MSI-edge      eth2-TxRx-4
   88:          0          0          0          0          0      34982 
          0          0   PCI-MSI-edge      eth2-TxRx-5
   89:          0          0          0          0          0         61 
      32420          0   PCI-MSI-edge      eth2-TxRx-6
   90:          0          0          0          0          0         61 
          0      25922   PCI-MSI-edge      eth2-TxRx-7
   91:          0          0          0          0          0          3 
          0          0   PCI-MSI-edge      eth2
  NMI:         13         12         15         22          5          5 
          5          5   Non-maskable interrupts
  LOC:      58919      61420      65519      82647      35519      40489 
      27141      30228   Local timer interrupts
  SPU:          0          0          0          0          0          0 
          0          0   Spurious interrupts
  PMI:         13         12         15         22          5          5 
          5          5   Performance monitoring interrupts
  IWI:          0          0          0          0          0          0 
          0          0   IRQ work interrupts
  RTR:          6          0          0          0          0          0 
          0          0   APIC ICR read retries
  RES:      15116       4521       2418       1814       2375       1615 
       1488       1367   Rescheduling interrupts
  CAL:        134        148        100        162        170        172 
        172        172   Function call interrupts
  TLB:        422        486        415        483        460        460 
        476        398   TLB shootdowns
  TRM:          0          0          0          0          0          0 
          0          0   Thermal event interrupts
  THR:          0          0          0          0          0          0 
          0          0   Threshold APIC interrupts
  MCE:          0          0          0          0          0          0 
          0          0   Machine check exceptions
  MCP:          4          4          4          4          4          4 
          4          4   Machine check polls

> what kind of NIC it is ?

# lspci | grep 10-Giga
06:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit 
SFI/SFP+ Network Connection (rev 01)

Stefan