Subject: Re: IPv6 path discovery oddities - flushing the routing cache resolves

From: Valentijn Sessink <[email-masked]>
Date: Thu, 17 Oct 2013 12:53:56 +0200

Hi,

Thanks for your fast response, please see below. Sorry for the rather 
long mail, due to the routing tables. Even though they're really short, 
the output of ipv6_route is extensive...

On 16-10-13 17:48, Hannes Frederic Sowa wrote:
> I do think these two issues are connected. Could you send me a the
> corresponding ip route output and /proc/net/ipv6_route output for when it
> works and mtus are correctly handled and when it does not work?

When it works (please note that the routing cache at first doesn't show 
an MTU, that only happens after a "too big" has been received, hence the 
difference before and after a "ps uaxww"), I have the following - this 
is from last night:

# ip -6 ro li
2a01:4f8:200:546b::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev vnet0  proto kernel  metric 256
fe80::/64 dev vnet1  proto kernel  metric 256
default via fe80::1 dev br0  metric 1024

# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0  metric 0
     cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
     cache

# ps uaxww
[... output cut for brevity ...]

# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0  metric 0
     cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
     cache  expires 577sec mtu 1280

# cat /proc/net/ipv6_route
200107b815290000cd7a7681b261af09 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000001 000062b2 01000023
   br0
20011af8ff030003021966fffe2606dd 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000001 00000002 01400023
   br0
2a0104f80000a0a10000000000020001 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000000 00000001 01000003
   br0
2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet1
00000000000000000000000000000000 00 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000400 00000000 00000000 00000003
   br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00
00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200
    lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000037 80200001
    lo
2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000003 000e6bc4 80200001
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 0000003b 80200001
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
    lo
fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
    lo
fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
    lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
   br0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet1
00000000000000000000000000000000 00 00000000000000000000000000000000 00
00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200
    lo

(Please note, while trying, I found a rather interesting output, showing 
negative time; if it's normal, please ignore:
# ip -statistics -6 ro li cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
     cache  expires -23sec users 2 used 2 age 637sec mtu 1280)

Anyway, today, suddenly it doesn't work anymore:

# ip -6 ro li
2a01:4f8:200:546b::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev vnet0  proto kernel  metric 256
fe80::/64 dev vnet1  proto kernel  metric 256
fe80::/64 dev vnet2  proto kernel  metric 256
default via fe80::1 dev br0  metric 1024

# ip -6 ro li cache
2001:7b8:2ff:37a::2 via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529::f via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
     cache
2a01:4f8:200:546b::5 via 2a01:4f8:200:546b::5 dev br0  metric 0
     cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
     cache

# ip -statistics -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
     cache  users 4 used 9448
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
     cache  users 4 used 260467 age 0sec
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
     cache  users 1 used 987 age 205sec
# ip -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
     cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
     cache
(The second time suddenly a couple of routes were gone - I don't know 
why). After "cat /proc/net/ipv6_route > /tmp/route-bad" I typed a "route 
-6 flush cache" and TCP started flowing again. Here's ipv6_route:

200107b802ff037a0000000000000002 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000000 00000006 01000003 
   br0
200107b815290000000000000000000f 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000000 00000006 01000003 
   br0
200107b81529000002138ffffeda140c 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000004 000024aa 01000023 
   br0
200107b815290000b845bb255d992a07 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000004 0003f79f 01000023 
   br0
2a0104f80200546b0000000000000005 80 00000000000000000000000000000000 00 
2a0104f80200546b0000000000000005 00000000 00000000 00000018 01000001 
   br0
2a0104f80200546b0000000000002eef 80 00000000000000000000000000000000 00 
2a0104f80200546b0000000000002eef 00000000 00000001 000003db 01000001 
   br0
2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
  eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet1
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet2
00000000000000000000000000000000 00 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000400 00000000 00000000 00000003 
   br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 
00000000000000000000000000000000 ffffffff 00000001 00064566 00200200 
    lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000167 80200001 
    lo
2a0104f80200546b0000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 0000000b 002a4bfe 80200001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 000001ed 80200001 
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
fe80000000000000fc5400fffedd5a81 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
   br0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
  eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet1
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet2
00000000000000000000000000000000 00 00000000000000000000000000000000 00 
00000000000000000000000000000000 ffffffff 00000001 00064566 00200200 
    lo

I hope this helps.

Best regards,

Valentijn