Subject: Re: After a while of system running no incoming UDP any more?

From: Paolo Abeni <[email-masked]>
Date: Fri, 28 Jul 2017 10:15:02 +0200

Hi,

On Fri, 2017-07-28 at 01:05 -0700, Eric Dumazet wrote:
> On Fri, 2017-07-28 at 08:26 +0200, Marc Haber wrote:
> > On Mon, Jul 24, 2017 at 04:19:10PM +0200, Paolo Abeni wrote:
> > > Once that a system enter the buggy status, do the packets reach the
> > > relevant socket's queue?
> > > 
> > > ss -u
> > > nstat |grep -e Udp -e Ip
> > > 
> > > will help checking that.
> > 
> > I now have the issue on one machine, a Xen guest acting as authoritative
> > nameserver for my domains. Here are the outputs during normal use, with
> > artificial queries coming in on eth0:
> > 
> > [9/1075]mh@impetus:~ $ ss -u
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port                
> > 0      0                                              127.0.0.1:56547                                                        127.0.0.1:domain               
> > 0      0                                         216.231.132.60:27667                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:44121                                                          8.8.8.8:domain               
> > 0      0                                         216.231.132.60:29814                                                       198.41.0.4:domain               
> > [10/1076]mh@impetus:~ $ ss -u
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port                
> > [11/1076]mh@impetus:~ $ ss -u
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port                
> > [12/1076]mh@impetus:~ $ ss -u
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port                
> > [13/1076]mh@impetus:~ $ ss -u
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port                
> > [14/1076]mh@impetus:~ $ nstat  | grep -e Udp -e Ip
> > IpInReceives                    400688             0.0
> > IpInAddrErrors                  18567              0.0
> > IpInUnknownProtos               3                  0.0
> > IpInDelivers                    330634             0.0
> > IpOutRequests                   283637             0.0
> > UdpInDatagrams                  145860             0.0
> > UdpNoPorts                      1313               0.0
> > UdpInErrors                     9356               0.0
> > UdpOutDatagrams                 153093             0.0
> > UdpIgnoredMulti                 34148              0.0
> > Ip6InReceives                   161178             0.0
> > Ip6InNoRoutes                   8                  0.0
> > Ip6InDelivers                   73841              0.0
> > Ip6OutRequests                  77575              0.0
> > Ip6InMcastPkts                  87332              0.0
> > Ip6OutMcastPkts                 109                0.0
> > Ip6InOctets                     21880674           0.0
> > Ip6OutOctets                    9633059            0.0
> > Ip6InMcastOctets                9371483            0.0
> > Ip6OutMcastOctets               6636               0.0
> > Ip6InNoECTPkts                  161202             0.0
> > Ip6InECT1Pkts                   15                 0.0
> > Ip6InECT0Pkts                   11                 0.0
> > Ip6InCEPkts                     4                  0.0
> > Udp6InDatagrams                 11725              0.0
> > Udp6NoPorts                     2                  0.0
> > Udp6InErrors                    1989               0.0
> > Udp6OutDatagrams                14483              0.0
> > IpExtInBcastPkts                34148              0.0
> > IpExtInOctets                   47462716           0.0
> > IpExtOutOctets                  31262696           0.0
> > IpExtInBcastOctets              7476059            0.0
> > IpExtInNoECTPkts                400178             0.0
> > IpExtInECT1Pkts                 22                 0.0
> > IpExtInECT0Pkts                 481                0.0
> > IpExtInCEPkts                   14                 0.0
> > [15/1077]mh@impetus:~ $ nstat  | grep -e Udp -e Ip
> > IpInReceives                    25                 0.0
> > IpInDelivers                    25                 0.0
> > IpOutRequests                   16                 0.0
> > UdpInDatagrams                  1                  0.0
> > UdpInErrors                     24                 0.0
> > UdpOutDatagrams                 16                 0.0
> > Ip6InReceives                   15                 0.0
> > Ip6InDelivers                   14                 0.0
> > Ip6OutRequests                  12                 0.0
> > Ip6InMcastPkts                  1                  0.0
> > Ip6InOctets                     1219               0.0
> > Ip6OutOctets                    4384               0.0
> > Ip6InMcastOctets                131                0.0
> > Ip6InNoECTPkts                  15                 0.0
> > IpExtInOctets                   11779              0.0
> > IpExtOutOctets                  1023               0.0
> > IpExtInNoECTPkts                25                 0.0
> > [16/1077]mh@impetus:~ $ nstat  | grep -e Udp -e Ip
> > IpInReceives                    24                 0.0
> > IpInDelivers                    24                 0.0
> > IpOutRequests                   18                 0.0
> > UdpInErrors                     22                 0.0
> > UdpOutDatagrams                 16                 0.0
> > Ip6InReceives                   15                 0.0
> > Ip6InDelivers                   12                 0.0
> > Ip6OutRequests                  10                 0.0
> > Ip6InMcastPkts                  3                  0.0
> > Ip6InOctets                     1160               0.0
> > Ip6OutOctets                    2456               0.0
> > Ip6InMcastOctets                216                0.0
> > Ip6InNoECTPkts                  15                 0.0
> > IpExtInOctets                   8612               0.0
> > IpExtOutOctets                  1127               0.0
> > IpExtInNoECTPkts                24                 0.0
> > [17/1077]mh@impetus:~ $ nstat  | grep -e Udp -e Ip
> > IpInReceives                    5                  0.0
> > IpInDelivers                    4                  0.0
> > IpOutRequests                   3                  0.0
> > UdpNoPorts                      1                  0.0
> > UdpInErrors                     2                  0.0
> > UdpOutDatagrams                 1                  0.0
> > Ip6InReceives                   12                 0.0
> > Ip6InDelivers                   12                 0.0
> > Ip6OutRequests                  10                 0.0
> > Ip6InOctets                     944                0.0
> > Ip6OutOctets                    2364               0.0
> > Ip6InNoECTPkts                  12                 0.0
> > IpExtInOctets                   429                0.0
> > IpExtOutOctets                  226                0.0
> > IpExtInNoECTPkts                5                  0.0
> > [18/1077]mh@impetus:~ $
> > 
> > And here, hopefully a bit more helpful:
> > 
> > [19/1078]mh@impetus:~ $ ss -u ; nstat  | grep -e Udp -e Ip ; dig +time=2 @8.8.8.8 zugschlus.de mx ; ss -u ; nstat  | grep -e Udp -e Ip 
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port                
> > 0      0                                         216.231.132.60:27333                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:38101                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:15836                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:50655                                                          8.8.8.8:domain               
> > 0      0                                         216.231.132.60:41953                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:6888                                                        198.41.0.4:domain               
> > 0      0                                         216.231.132.60:51441                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:42503                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:12575                                                       198.41.0.4:domain               
> > 0      0                                         216.231.132.60:13857                                                       198.41.0.4:domain
> > 0      0                                         216.231.132.60:16419                                                    192.36.148.17:domain
> > 0      0                                         216.231.132.60:39227                                                       198.41.0.4:domain
> > 0      0                                              127.0.0.1:54608                                                        127.0.0.1:domain
> > 0      0                                         216.231.132.60:20818                                                       198.41.0.4:domain
> > 0      0                                         216.231.132.60:56662                                                       198.41.0.4:domain
> > 0      0                                         216.231.132.60:48259                                                    192.36.148.17:domain
> > 0      0                                         216.231.132.60:37803                                                       198.41.0.4:domain
> > IpInReceives                    59                 0.0
> > IpInAddrErrors                  1                  0.0
> > IpInDelivers                    56                 0.0
> > IpOutRequests                   50                 0.0
> > UdpInDatagrams                  1                  0.0
> > UdpInErrors                     50                 0.0
> > UdpOutDatagrams                 47                 0.0
> > UdpIgnoredMulti                 1                  0.0
> > Ip6InReceives                   75                 0.0
> > Ip6InDelivers                   73                 0.0
> > Ip6OutRequests                  64                 0.0
> > Ip6InMcastPkts                  2                  0.0
> > Ip6InOctets                     7837               0.0
> > Ip6OutOctets                    11876              0.0
> > Ip6InMcastOctets                279                0.0
> > Ip6InNoECTPkts                  75                 0.0
> > Udp6InErrors                    3                  0.0
> > IpExtInBcastPkts                1                  0.0
> > IpExtInOctets                   18447              0.0
> > IpExtOutOctets                  3478               0.0
> > IpExtInBcastOctets              183                0.0
> > IpExtInNoECTPkts                59                 0.0
> > 
> > ; <<>> DiG 9.10.3-P4-Debian <<>> +time=2 @8.8.8.8 zugschlus.de mx
> > ; (1 server found)
> > ;; global options: +cmd
> > ;; connection timed out; no servers could be reached
> > Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port
> > 0      0                                         216.231.132.60:7879                                                      202.12.27.33:domain
> > 0      0                                         216.231.132.60:32711                                                     202.12.27.33:domain
> > 0      0                                         216.231.132.60:54238                                                     202.12.27.33:domain
> > 0      0                                         216.231.132.60:30948                                                   192.228.79.201:domain
> > 0      0                                         216.231.132.60:4106                                                      202.12.27.33:domain
> > 0      0                                         216.231.132.60:6667                                                      202.12.27.33:domain
> > 0      0                                         216.231.132.60:2090                                                    192.228.79.201:domain
> > 0      0                                         216.231.132.60:60459                                                   192.228.79.201:domain
> > 0      0                                         216.231.132.60:16427                                                     202.12.27.33:domain
> > 0      0                                         216.231.132.60:9019                                                      202.12.27.33:domain
> > 0      0                                         216.231.132.60:2113                                                      202.12.27.33:domain
> > 0      0                                         216.231.132.60:34907                                                     202.12.27.33:domain
> > 0      0                                         216.231.132.60:34654                                                     202.12.27.33:domain
> > 0      0                                         216.231.132.60:47725                                                     202.12.27.33:domain
> > 0      0                                         216.231.132.60:35774                                                     202.12.27.33:domain
> > IpInReceives                    38                 0.0
> > IpInDelivers                    38                 0.0
> > IpOutRequests                   38                 0.0
> > UdpInDatagrams                  2                  0.0
> > UdpInErrors                     34                 0.0
> > UdpOutDatagrams                 36                 0.0
> > Ip6InReceives                   14                 0.0
> > Ip6InDelivers                   13                 0.0
> > Ip6OutRequests                  13                 0.0
> > Ip6InMcastPkts                  1                  0.0
> > Ip6InOctets                     1046               0.0
> > Ip6OutOctets                    6277               0.0
> > Ip6InMcastOctets                133                0.0
> > Ip6InNoECTPkts                  13                 0.0
> > Ip6InECT0Pkts                   1                  0.0
> > Udp6InDatagrams                 1                  0.0
> > Udp6OutDatagrams                1                  0.0
> > IpExtInOctets                   15963              0.0
> > IpExtOutOctets                  2397               0.0
> > IpExtInNoECTPkts                37                 0.0
> > IpExtInECT0Pkts                 1                  0.0
> > [20/1079]mh@impetus:~ $
> > 
> > I am afraid I cannot keep this state for much longer than a few
> > additional hours as this is an authoritative name server...
> > 
> > Greetings
> > Marc
> > 
> 
> To confirm the refcount issue, you might send us :
> 
> cat /proc/net/udp6
> 
> Normally, the refcount column should have a value of 2

I think that the leaked sockets are still unhashed on close() (via
udp_lib_close() -> sk_common_release() ->sk_proto->unhash()), so they
should not be listed there ?!? (Rush answer, subject to lack of coffee
issues)

Thanks,

Paolo