Subject: Re: TCP_STREAM performance regression on commit b3613118

From: Feng Tang <[email-masked]>
Date: Tue, 6 Mar 2012 16:11:17 +0800

On Thu, Mar 01, 2012 at 10:07:43PM -0500, David Miller wrote:
> From: Alex Shi <[email-masked]>
> Date: Fri, 02 Mar 2012 10:45:17 +0800
> 
> > Add CC to tang feng, He is working on this issue. 
> 
> Is he?  I'm pretty sure this is due to the TCP receive window growing
> issue Eric Dumazet, Neal Cardwell and I are discussing in the thread
> starting at:
> 
> http://marc.info/?l=linux-netdev&m=132916352815286&w=2

Yes, probably, as we did find some clue related with the tcp_r/wmem.

Here is the regression we found:
On some machines, we found there is about 10% resgression of netperf
TCP-64K loopback test between 3.2 and 3.3-rc1. The exact test is:
./netperf -t TCP_STREAM -l 60 -H 127.0.0.1 -i 50,3 -I 99,5 -- -s 32768 -S 32768 -m 4096

The test machine is a 2 socket Quad Core Core 2 Duo server(2.66GHz) with
8 GB RAM. Following are the debug info (ifconfig/netstat -s/tcp_rwmem)
before and after the test: 

The most obvious differences I can see are:
1) 311 GB vs 241 GB from ifconfig
2) the difference of the tcp_r/wmem

Thanks,
Feng

----------------------------------------------------------
Before the test (v3.2)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:947 errors:0 dropped:0 overruns:0 frame:0
          TX packets:253 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:76573 (74.7 KiB)  TX bytes:31034 (30.3 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:197 errors:0 dropped:0 overruns:0 frame:0
          TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:21827 (21.3 KiB)  TX bytes:21827 (21.3 KiB)

netstat info
Ip:
    593 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    590 incoming packets delivered
    431 requests sent out
Icmp:
    7 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7
    7 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 7
IcmpMsg:
        InType3: 7
        OutType3: 7
Tcp:
    7 active connections openings
    10 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    371 segments received
    328 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    87 packets received
    7 packets to unknown port received.
    0 packet receive errors
    94 packets sent
UdpLite:
tcp_rmem
4096	87380	4194304
tcp_wmem
4096	16384	4194304
=========================

After the test (v3.2)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2278 errors:0 dropped:0 overruns:0 frame:0
          TX packets:690 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:174366 (170.2 KiB)  TX bytes:88711 (86.6 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:54845546 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54845546 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:334960558399 (311.9 GiB)  TX bytes:334960558399 (311.9 GiB)

netstat info
Ip:
    54846593 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    54846590 incoming packets delivered
    54846212 requests sent out
Icmp:
    7 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7
    7 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 7
IcmpMsg:
        InType3: 7
        OutType3: 7
Tcp:
    21 active connections openings
    22 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    54846182 segments received
    54846067 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    126 packets received
    7 packets to unknown port received.
    0 packet receive errors
    134 packets sent
UdpLite:
tcp_rmem
4096	87380	4194304
tcp_wmem
4096	16384	4194304
=========================

Before the test (v3.3-rc1)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:205 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:50045 (48.8 KiB)  TX bytes:27018 (26.3 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:88 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9613 (9.3 KiB)  TX bytes:9613 (9.3 KiB)

netstat info
Ip:
    365 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    362 incoming packets delivered
    278 requests sent out
Icmp:
    3 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 3
    3 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 3
IcmpMsg:
        InType3: 3
        OutType3: 3
Tcp:
    3 active connections openings
    6 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    226 segments received
    206 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    64 packets received
    3 packets to unknown port received.
    0 packet receive errors
    67 packets sent
UdpLite:
tcp_rmem
4096	87380	87380
tcp_wmem
4096	16384	65536
=========================

After the test (v3.3-rc1)
===========================
ifconfig info
eth1      Link encap:Ethernet  HWaddr 00:30:48:7C:CF:72  
          inet addr:192.168.1.58  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe7c:cf72/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1724 errors:0 dropped:0 overruns:0 frame:0
          TX packets:608 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:134666 (131.5 KiB)  TX bytes:79003 (77.1 KiB)
          Memory:d8420000-d8440000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:45143339 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45143339 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:259440765641 (241.6 GiB)  TX bytes:259440765641 (241.6 GiB)

netstat info
Ip:
    45144169 total packets received
    3 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    45144166 incoming packets delivered
    45143928 requests sent out
Icmp:
    3 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 3
    3 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 3
IcmpMsg:
        InType3: 3
        OutType3: 3
Tcp:
    16 active connections openings
    17 passive connection openings
    0 failed connection attempts
    0 connection resets received
    3 connections established
    45143859 segments received
    45143814 segments send out
    0 segments retransmited
    0 bad segments received.
    3 resets sent
Udp:
    103 packets received
    3 packets to unknown port received.
    0 packet receive errors
    107 packets sent
UdpLite:
tcp_rmem
4096	87380	87380
tcp_wmem
4096	16384	65536
=========================