Subject: Re: Network performance - iperf

From: "Steve Magnani" <[email-masked]>
Date: Tue, 30 Mar 2010 07:11:51 -0500

Looks like a lot of time spent in tcp_collapse(). I think there is assumption that the memcpys 
involved in collapsing lead to better throughput than packet retransmit, which on our platforms 
may not be the case. If so is there a way to tune when tcp_collapse gets invoked? I haven't 
seen one.

Steve

-----Original Message-----
From: Michal Simek <[email-masked]>
To: Eric Dumazet <[email-masked]>
Cc: LKML <[email-masked]>,  John Williams <[email-masked]>, 
[email-masked], Grant Likely <[email-masked]>,  John Linn 
<[email-masked]>, "Steven J. Magnani" <[email-masked]>,  Arnd Bergmann 
<[email-masked]>, [email-masked]
Date: Tue, 30 Mar 2010 11:34:29 +0200
Subject: Re: Network performance - iperf

> Eric Dumazet wrote:
> > Le lundi 29 mars 2010 à 17:27 +0200, Michal Simek a écrit :
> >> Michal Simek wrote:
> >>> Eric Dumazet wrote:
> >>>> Le lundi 29 mars 2010 à 13:33 +0200, Michal Simek a écrit :
> >>>>
> >>>>> Do you have any idea howto improve TCP/UDP performance in
> general?
> >>>>> Or tests which can point me on weak places.
> >>>> Could you post "netstat -s" on your receiver, after fresh boot and
> your
> >>>> iperf session, for 32 MB and 256 MB ram case ?
> >>>>
> >>> I am not sure if is helpful but look below.
> >>>
> >> Sorry I forget to c&p that second part. :-(
> >>
> > 
> > Sorry, your netstat is not up2date.
> 
> I am afraid that is up2date.
> 
> > 
> > If you cannot correct it to last version
> > [ net-tools 1.60 , netstat 1.42 ], please send
> > 
> > cat /proc/net/snmp
> > cat /proc/net/netstat
> 
> There is small buffer for parsing /proc/net/netstat.
> There is necessary to extend buffer size because one line is greater 
> than 1024 chars.
> 
> ~ # head -n 1 /proc/net/netstat
> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts 
> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps 
> ArpFilter TW TWRecycled TWKilled PAWSPassive PAWSActive PAWSEstab 
> DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops
> TCPPrequeued TCPDirectCopyFromBacklog TCPDirectCopyFromPrequeue 
> TCPPrequeueDropped TCPHPHits TCPHPHitsToUser TCPPureAcks TCPHPAcks 
> TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPFACKReorder 
> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo 
> TCPDSACKUndo TCPLossUndo TCPLoss TCPLostRetransmit TCPRenoFailures 
> TCPSackFailures TCPLossFailures TCPFastRetrans TCPForwardRetrans 
> TCPSlowStartRetrans TCPTimeouts TCPRenoRecoveryFail TCPSackRecoveryFail
> TCPSchedulerFailed TCPRcvCollapsed TCPDSACKOldSent TCPDSACKOfoSent 
> TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnSyn TCPAbortOnData 
> TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger 
> TCPAbortFailed TCPMemoryPressures TCPSACKDiscard TCPDSACKIgnoredOld 
> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected 
> TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop 
> TCPMinTTLDrop
> 
> Look at attached patch.
> And updated results are below.
> 
> Thanks,
> Michal
> 
> 
> 
> 
> 
> 256M
> 
> ~ # iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port
> 33261
> [ ID] Interval       Transfer     Bandwidth
> [  6]  0.0-50.2 sec  22.9 MBytes  3.83 Mbits/sec
> ~ # ./netstat -s
> Ip:
>      16618 total packets received
>      0 forwarded
>      0 incoming packets discarded
>      16618 incoming packets delivered
>      6490 requests sent out
> Icmp:
>      0 ICMP messages received
>      0 input ICMP message failed.
>      ICMP input histogram:
>      0 ICMP messages sent
>      0 ICMP messages failed
>      ICMP output histogram:
> Tcp:
>      0 active connections openings
>      1 passive connection openings
>      0 failed connection attempts
>      0 connection resets received
>      0 connections established
>      16618 segments received
>      6490 segments send out
>      0 segments retransmited
>      0 bad segments received.
>      0 resets sent
> Udp:
>      0 packets received
>      0 packets to unknown port received.
>      0 packet receive errors
>      0 packets sent
>      RcvbufErrors: 0
>      SndbufErrors: 0
> UdpLite:
>      InDatagrams: 0
>      NoPorts: 0
>      InErrors: 0
>      OutDatagrams: 0
>      RcvbufErrors: 0
>      SndbufErrors: 0
> TcpExt:
>      2233 packets pruned from receive queue because of socket buffer
> overrun
>      ArpFilter: 0
>      1 delayed acks sent
>      5519 packets header predicted
>      TCPPureAcks: 2
>      TCPHPAcks: 0
>      TCPRenoRecovery: 0
>      TCPSackRecovery: 0
>      TCPSACKReneging: 0
>      TCPFACKReorder: 0
>      TCPSACKReorder: 0
>      TCPRenoReorder: 0
>      TCPTSReorder: 0
>      TCPFullUndo: 0
>      TCPPartialUndo: 0
>      TCPDSACKUndo: 0
>      TCPLossUndo: 0
>      TCPLoss: 0
>      TCPLostRetransmit: 0
>      TCPRenoFailures: 0
>      TCPSackFailures: 0
>      TCPLossFailures: 0
>      TCPFastRetrans: 0
>      TCPForwardRetrans: 0
>      TCPSlowStartRetrans: 0
>      TCPTimeouts: 0
>      TCPRenoRecoveryFail: 0
>      TCPSackRecoveryFail: 0
>      TCPSchedulerFailed: 0
>      TCPRcvCollapsed: 207654
>      TCPDSACKOldSent: 0
>      TCPDSACKOfoSent: 0
>      TCPDSACKRecv: 0
>      TCPDSACKOfoRecv: 0
>      TCPAbortOnSyn: 0
>      TCPAbortOnData: 0
>      TCPAbortOnClose: 0
>      TCPAbortOnMemory: 0
>      TCPAbortOnTimeout: 0
>      TCPAbortOnLinger: 0
>      TCPAbortFailed: 0
>      TCPMemoryPressures: 0
>      TCPSACKDiscard: 0
>      TCPDSACKIgnoredOld: 0
>      TCPDSACKIgnoredNoUndo: 0
>      TCPSpuriousRTOs: 0
>      TCPMD5NotFound: 0
>      TCPMD5Unexpected: 0
>      TCPSackShifted: 0
>      TCPSackMerged: 0
>      TCPSackShiftFallback: 0
>      TCPBacklogDrop: 0
>      TCPMinTTLDrop: 0
> IpExt:
>      InNoRoutes: 0
>      InTruncatedPkts: 0
>      InMcastPkts: 0
>      OutMcastPkts: 0
>      InBcastPkts: 0
>      OutBcastPkts: 0
>      InOctets: 24915880
>      OutOctets: 337488
>      InMcastOctets: 0
>      OutMcastOctets: 0
>      InBcastOctets: 0
>      OutBcastOctets: 0
> ~ # ./netstat --version
> net-tools 1.60
> netstat 1.42 (2001-04-15)
> Fred Baumgarten, Alan Cox, Bernd Eckenfels, Phil Blundell, Tuan Hoang 
> and others
> +NEW_ADDRT +RTF_IRTT +RTF_REJECT +FW_MASQUERADE -I18N
> AF: (inet) +UNIX +INET -INET6 -IPX -AX25 -NETROM -X25 -ATALK -ECONET
> -ROSE
> HW:  +ETHER -ARC +SLIP +PPP -TUNNEL -TR -AX25 -NETROM -X25 -FR -ROSE 
> -ASH -SIT -FDDI -HIPPI -HDLC/LAPB
> ~ # head -n 1 /proc/meminfo
> MemTotal:         257108 kB
> 
> 
> 
> 32MB
> 
> ~ # head -n 1 /proc/meminfo
> MemTotal:          29920 kB
> ~ # iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port
> 50088
> [ ID] Interval       Transfer     Bandwidth
> [  6]  0.0-50.0 sec    109 MBytes  18.3 Mbits/sec
> ~ # ./netstat -s
> Ip:
>      79040 total packets received
>      0 forwarded
>      0 incoming packets discarded
>      79040 incoming packets delivered
>      29655 requests sent out
> Icmp:
>      0 ICMP messages received
>      0 input ICMP message failed.
>      ICMP input histogram:
>      0 ICMP messages sent
>      0 ICMP messages failed
>      ICMP output histogram:
> Tcp:
>      0 active connections openings
>      1 passive connection openings
>      0 failed connection attempts
>      0 connection resets received
>      0 connections established
>      79040 segments received
>      29655 segments send out
>      0 segments retransmited
>      0 bad segments received.
>      0 resets sent
> Udp:
>      0 packets received
>      0 packets to unknown port received.
>      0 packet receive errors
>      0 packets sent
>      RcvbufErrors: 0
>      SndbufErrors: 0
> UdpLite:
>      InDatagrams: 0
>      NoPorts: 0
>      InErrors: 0
>      OutDatagrams: 0
>      RcvbufErrors: 0
>      SndbufErrors: 0
> TcpExt:
>      9773 packets pruned from receive queue because of socket buffer
> overrun
>      ArpFilter: 0
>      1 delayed acks sent
>      101 packets directly queued to recvmsg prequeue.
>      558928 packets directly received from prequeue
>      33274 packets header predicted
>      378 packets header predicted and directly queued to user
>      TCPPureAcks: 2
>      TCPHPAcks: 0
>      TCPRenoRecovery: 0
>      TCPSackRecovery: 0
>      TCPSACKReneging: 0
>      TCPFACKReorder: 0
>      TCPSACKReorder: 0
>      TCPRenoReorder: 0
>      TCPTSReorder: 0
>      TCPFullUndo: 0
>      TCPPartialUndo: 0
>      TCPDSACKUndo: 0
>      TCPLossUndo: 0
>      TCPLoss: 0
>      TCPLostRetransmit: 0
>      TCPRenoFailures: 0
>      TCPSackFailures: 0
>      TCPLossFailures: 0
>      TCPFastRetrans: 0
>      TCPForwardRetrans: 0
>      TCPSlowStartRetrans: 0
>      TCPTimeouts: 0
>      TCPRenoRecoveryFail: 0
>      TCPSackRecoveryFail: 0
>      TCPSchedulerFailed: 0
>      TCPRcvCollapsed: 120195
>      TCPDSACKOldSent: 0
>      TCPDSACKOfoSent: 0
>      TCPDSACKRecv: 0
>      TCPDSACKOfoRecv: 0
>      TCPAbortOnSyn: 0
>      TCPAbortOnData: 0
>      TCPAbortOnClose: 0
>      TCPAbortOnMemory: 0
>      TCPAbortOnTimeout: 0
>      TCPAbortOnLinger: 0
>      TCPAbortFailed: 0
>      TCPMemoryPressures: 0
>      TCPSACKDiscard: 0
>      TCPDSACKIgnoredOld: 0
>      TCPDSACKIgnoredNoUndo: 0
>      TCPSpuriousRTOs: 0
>      TCPMD5NotFound: 0
>      TCPMD5Unexpected: 0
>      TCPSackShifted: 0
>      TCPSackMerged: 0
>      TCPSackShiftFallback: 0
>      TCPBacklogDrop: 0
>      TCPMinTTLDrop: 0
> IpExt:
>      InNoRoutes: 0
>      InTruncatedPkts: 0
>      InMcastPkts: 0
>      OutMcastPkts: 0
>      InBcastPkts: 0
>      OutBcastPkts: 0
>      InOctets: 118232864
>      OutOctets: 1542068
>      InMcastOctets: 0
>      OutMcastOctets: 0
>      InBcastOctets: 0
>      OutBcastOctets: 0
> ~ #
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> Michal Simek, Ing. (M.Eng)
> PetaLogix - Linux Solutions for a Reconfigurable World
> w: www.petalogix.com p: +61-7-30090663,+42-0-721842854 f:
> +61-7-30090663