Subject: Re: atl1e: TSO is broken

From: Alejandro Riveira Fernández<[email-masked]>
Date: Fri, 15 Jan 2010 16:54:47 +0100

El Fri, 15 Jan 2010 13:56:13 +0100 (CET)
Anders Boström <[email-masked]> escribió:

CCing netdev

> Short desription
> ----------------
> 
> TCP Segmentation Offload (TSO) result in broken IPv4-packets sent out
> from Atheros AR8121/AR8113/AR8114 with the atl1e driver.
> 
> 
> Work around
> -----------
> 
> Turn off TSO. Requires 2.6.33-rc1 or later.
> 
> 
> Long desription
> ----------------
> 
> When I run NFS over TCP (default options) and read large files from a
> server with Atheros AR8121/AR8113/AR8114 Ethernet chip, I only get
> ~25Mbyte/s performance. I get ~5000 retransmitted packets per GByte
> data, according to RetransSegs in /proc/net/snmp . wireshark in the
> client show that the server send out a sequence of frames. All but the
> last one are 1500 bytes IP-packets. The last one is shorter, but the
> IP-header still say 1500 byte. The client then requests retransmit,
> and the retransmitted frame arrives with correct IP-header.
> 
> If I mount NFS using UDP instead, performance is ~110Mbyte/s.
> 
> TCP Segmentation Offload (TSO) is default enabled in the atl1e
> Ethernet-driver. When I run a patched 2.6.30.10, enabling ethtool to
> turn off TSO (using ac936929092dc6a5409b627c4c67305ab9b626b3 by Ben
> Hutchings), and turn off TSO, the problem disappears. Performance is
> ~110Mbyte/s and no broken IP-packets arrive.
> 
> 
> Capture of 146-byte Ethernet frame with bad IP-header:
> 
> No.     Time        Source                Destination           Protocol Info
>   98329 11.034129   flash.netinsight.se   sid.netinsight.se     RPC      Continuation
> 
> Frame 98329 (146 bytes on wire, 146 bytes captured)
>     Arrival Time: Jan 15, 2010 13:35:16.224491000
>     [Time delta from previous captured frame: 0.000009000 seconds]
>     [Time delta from previous displayed frame: 0.000009000 seconds]
>     [Time since reference or first frame: 11.034129000 seconds]
>     Frame Number: 98329
>     Frame Length: 146 bytes
>     Capture Length: 146 bytes
>     [Frame is marked: False]
>     [Protocols in frame: eth:ip:tcp:rpc]
>     [Coloring Rule Name: TCP]
>     [Coloring Rule String: tcp]
> Ethernet II, Src: AsustekC_ae:69:6d (00:26:18:ae:69:6d), Dst: sid.netinsight.se (00:18:f3:52:22:3f)
> Internet Protocol, Src: flash.netinsight.se (10.100.0.88), Dst: sid.netinsight.se (10.100.1.25)
>     Version: 4
>     Header length: 20 bytes
>     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
>     Total Length: 1500
>     Identification: 0x331e (13086)
>     Flags: 0x02 (Don't Fragment)
>     Fragment offset: 0
>     Time to live: 64
>     Protocol: TCP (0x06)
>     Header checksum: 0xebc5 [correct]
>     Source: flash.netinsight.se (10.100.0.88)
>     Destination: sid.netinsight.se (10.100.1.25)
> Transmission Control Protocol, Src Port: nfs (2049), Dst Port: accessbuilder (888), Seq: 93989617, Ack: 516997, Len: 80
> Remote Procedure Call
> 
> 0000  00 18 f3 52 22 3f 00 26 18 ae 69 6d 08 00 45 00   ...R"?.&..im..E.
> 0010  05 dc 33 1e 40 00 40 06 eb c5 0a 64 00 58 0a 64   ..3.@.@....d.X.d
> 0020  01 19 08 01 03 78 a4 07 57 23 e6 50 1f 1b 80 10   .....x..W#.P....
> 0030  01 f5 dd 8d 00 00 01 01 08 0a 05 28 ca 7e 38 93   ...........(.~8.
> 0040  67 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00   g...............
> 0050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
> 0060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
> 0070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
> 0080  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
> 0090  00 00                                             ..
> 
> 
> 
> Software info:
> 
> I've tested with Debian 2.6.26 (stable) and 2.6.30 (testing), as well
> as 2.6.30.10 from kernel.org. Same result.
> 
> Hardware info:
> 
> lspci -vvv:
> 
> 03:00.0 Ethernet controller: Attansic Technology Corp. Atheros AR8121/AR8113/AR8114 PCI-E Ethernet Controller (rev b0)
>         Subsystem: ASUSTeK Computer Inc. Device 831c
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 27
>         Region 0: Memory at fbfc0000 (64-bit, non-prefetchable) [size=256K]
>         Region 2: I/O ports at ec00 [size=128]
>         Capabilities: [40] Power Management version 2
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
>                 Address: 00000000fee0f00c  Data: 4189
>         Capabilities: [58] Express (v1) Endpoint, MSI 00
>                 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
>                         ExtTag- AttnBtn+ AttnInd+ PwrInd+ RBE- FLReset-
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>                 DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 unlimited, L1 unlimited
>                         ClockPM- Suprise- LLActRep- BwNot-
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>         Capabilities: [100] Advanced Error Reporting <?>
>         Capabilities: [180] Device Serial Number ff-18-26-00-6d-69-ae-ff
>         Kernel driver in use: ATL1E
>         Kernel modules: atl1e
> 
> ethtool -i eth0:
> 
> driver: ATL1E
> version: 1.0.0.7-NAPI
> firmware-version: L1e
> bus-info: 0000:03:00.0
> 
> 
> / Anders
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/