Subject: Re: [PATCH net v4 1/2] gre: Fix IPv6 link-local address generation.

From: Simon Horman <[email-masked]>
Date: Thu, 20 Mar 2025 16:26:46 +0000

On Mon, Mar 17, 2025 at 10:10:45PM +0100, Guillaume Nault wrote:
> On Sun, Mar 16, 2025 at 03:08:48PM +0200, Ido Schimmel wrote:
> > On Fri, Mar 14, 2025 at 01:18:21PM -0700, Stanislav Fomichev wrote:
> > > On 03/14, Guillaume Nault wrote:
> > > > On Fri, Mar 14, 2025 at 08:18:32AM -0700, Stanislav Fomichev wrote:
> > > > > 
> > > > > Could you please double check net/forwarding/ip6gre_custom_multipath_hash.sh ?
> > > > > It seems like it started falling after this series has been pulled:
> > > > > https://netdev-3.bots.linux.dev/vmksft-forwarding-dbg/results/31301/2-ip6gre-custom-multipath-hash-sh/stdout
> > > > 
> > > > Hum, net/forwarding/ip6gre_custom_multipath_hash.sh works for me on the
> > > > current net tree (I'm at commit 4003c9e78778). I have only one failure,
> > > > but it already happened before 183185a18ff9 ("gre: Fix IPv6 link-local
> > > > address generation.") was applied.
> > > 
> > > On my side I see the following (ignore ping6 FAILs):
> > > 
> > > bfc6c67ec2d6 - (net-next/main, net-next/HEAD) net/smc: use the correct ndev to find pnetid by pnetid table (7 hours ago) <Guangguan Wang>
> > > 
> > > TAP version 13
> > > 1..1
> > > # timeout set to 0
> > > # selftests: net/forwarding: ip6gre_custom_multipath_hash.sh
> > > [    9.275735][  T167] ip (167) used greatest stack depth: 23536 bytes left
> > > [   13.769300][  T255] gre: GRE over IPv4 demultiplexor driver
> > > [   13.838185][  T255] ip6_gre: GRE over IPv6 tunneling driver
> > > [   13.951780][   T12] ip6_tunnel: g1 xmit: Local address not yet configured!
> > > [   14.038101][   T12] ip6_tunnel: g1 xmit: Local address not yet configured!
> > > [   15.148469][  T281] 8021q: 802.1Q VLAN Support v1.8
> > > [   17.559477][  T321] GACT probability NOT on
> > > [   18.551876][   T12] ip6_tunnel: g2 xmit: Local address not yet configured!
> > > [   18.633656][   T12] ip6_tunnel: g2 xmit: Local address not yet configured!
> > > # TEST: ping                                                          [ OK ]
> > > # TEST: ping6                                                         [FAIL]
> > > # INFO: Running IPv4 overlay custom multipath hash tests
> > > # TEST: Multipath hash field: Inner source IP (balanced)              [FAIL]
> > > #       Expected traffic to be balanced, but it is not
> > > # INFO: Packets sent on path1 / path2: 1 / 12602
> > > # TEST: Multipath hash field: Inner source IP (unbalanced)            [ OK ]
> > > # INFO: Packets sent on path1 / path2: 0 / 12601
> > > # TEST: Multipath hash field: Inner destination IP (balanced)         [FAIL]
> > > #       Expected traffic to be balanced, but it is not
> > > # INFO: Packets sent on path1 / path2: 1 / 12600
> > > # TEST: Multipath hash field: Inner destination IP (unbalanced)       [ OK ]
> > > # INFO: Packets sent on path1 / path2: 0 / 12600
> > > ...
> > > 
> > > 8ecea691e844 - (HEAD -> upstream/net-next/main) Revert "gre: Fix IPv6 link-local address generation." (2 minutes ago) <Stanislav Fomichev>
> > > 
> > > TAP version 13
> > > 1..1
> > > # timeout set to 0
> > > # selftests: net/forwarding: ip6gre_custom_multipath_hash.sh
> > > [   13.863060][  T252] gre: GRE over IPv4 demultiplexor driver
> > > [   13.911551][  T252] ip6_gre: GRE over IPv6 tunneling driver
> > > [   15.226124][  T277] 8021q: 802.1Q VLAN Support v1.8
> > > [   17.629460][  T317] GACT probability NOT on
> > > [   17.645781][  T315] tc (315) used greatest stack depth: 23040 bytes left
> > > # TEST: ping                                                          [ OK ]
> > > # TEST: ping6                                                         [FAIL]
> > > # INFO: Running IPv4 overlay custom multipath hash tests
> > > # TEST: Multipath hash field: Inner source IP (balanced)              [ OK ]
> > > # INFO: Packets sent on path1 / path2: 5552 / 7052
> > > # TEST: Multipath hash field: Inner source IP (unbalanced)            [ OK ]
> > > # INFO: Packets sent on path1 / path2: 12600 / 2
> > > [   36.278056][    C2] clocksource: Long readout interval, skipping watchdog check: cs_nsec: 1078005296 wd_nsec: 1078004682
> > > # TEST: Multipath hash field: Inner destination IP (balanced)         [ OK ]
> > > # INFO: Packets sent on path1 / path2: 6650 / 5950
> > > # TEST: Multipath hash field: Inner destination IP (unbalanced)       [ OK ]
> > > # INFO: Packets sent on path1 / path2: 0 / 12600
> > > ...
> > > 
> > > And I also see the failures on 4003c9e78778. Not sure why we see
> > > different results. And the NIPAs fails as well:
> > > 
> > > https://netdev-3.bots.linux.dev/vmksft-forwarding-dbg/results/32922/1-ip6gre-custom-multipath-hash-sh/stdout
> > 
> > I can reproduce this locally and I'm getting the exact same result as
> > the CI. All the balanced tests fail because the traffic is forwarded via
> > a single nexthop. No failures after reverting 183185a18ff9.
> > 
> > I'm still not sure what happens, but for some reason a neighbour is not
> > created on one of the nexthop devices which causes rt6_check_neigh() to
> > skip over this path (returning RT6_NUD_FAIL_DO_RR). Enabling
> > CONFIG_IPV6_ROUTER_PREF fixes the issue because then RT6_NUD_SUCCEED is
> > returned.
> > 
> > I can continue looking into this on Tuesday (mostly AFK tomorrow).
> 
> I finally managed to reproduce the problem using vng. Still no problem
> on my regular VM, no matter if I enable CONFIG_IPV6_ROUTER_PREF or not.
> I'll continue investigating this problem...

FWIIW, I have tried much, but am unable to _reliably_ reproduce this problem.