Subject: Re: TCP funny-ness when over-driving a 1Gbps link.

From: Rick Jones <[email-masked]>
Date: Thu, 19 May 2011 17:24:39 -0700

> >> [root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
> >> tcp        0      0 8.1.1.1:33038               0.0.0.0:*                   LISTEN
> >> tcp        0      0 8.1.1.1:33040               0.0.0.0:*                   LISTEN
> >> tcp        0      0 8.1.1.1:33042               0.0.0.0:*                   LISTEN
> >> tcp        0 9328612 8.1.1.2:33039               8.1.1.1:33040               ESTABLISHED
> >> tcp        0 17083176 8.1.1.1:33038               8.1.1.2:33037               ESTABLISHED
> >> tcp        0 9437340 8.1.1.2:33037               8.1.1.1:33038               ESTABLISHED
> >> tcp        0 17024620 8.1.1.1:33040               8.1.1.2:33039               ESTABLISHED
> >> tcp        0 19557040 8.1.1.1:33042               8.1.1.2:33041               ESTABLISHED
> >> tcp        0 9416600 8.1.1.2:33041               8.1.1.1:33042               ESTABLISHED
> >
> > I take it your system has higher values for the tcp_wmem value:
> >
> > net.ipv4.tcp_wmem = 4096 16384 4194304
> 
> Yes:
> [root@i7-965-1 igb]# cat /proc/sys/net/ipv4/tcp_wmem
> 4096	16384	50000000

Why?!?  Are you trying to get link-rate to Mars or something?  (I assume
tcp_rmem is similarly set...)  If you are indeed doing one 1 GbE, and no
more than 100ms then the default (?) of 4194304 should have been more
than sufficient.

> > and whatever is creating the TCP connections is not making explicit
> > setsockopt() calls to set SO_*BUF.
> 
> It is configured not to, but if you know of an independent way to verify
> that, I'm interested.

You could always strace the code.

rick