Subject: r8169 link up but no traffic, and watchdog error

From: Martin Kjær Jørgensen <[email-masked]>
Date: Wed, 09 Aug 2023 13:50:31 +0200

Hello netdev,

My machine have 2 x PCI-E card both having 2 x "Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE
Controller" devices giving me 4 ethernet ports in addition to the onboard
ethernet controller.

When a cable is plugged to on of the ports and the port in the other end
suddenly goes down and up again, like if power cycled, all traffic on the wire
stops even though the port LEDs light up, and kernel produces klog
entry, indicating a link.

I need to issue a commands like:

ip link set enp3s0 down
ip link set enp3s0 up

to have the link running again. This sometimes produces the error seen below,
but not all the time. After commands are issued the traffic flows again like
normal, until the remote port goes down and up again.

Have you guys got any ideas about this?

/Martin

[1]:

[   36.324381] logitech-hidpp-device 0003:046D:4055.0009: HID++ 4.5 device connected.
[  107.354641] r8169 0000:03:00.0 enp3s0: Link is Down
[  169.142879] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
[  180.198458] ------------[ cut here ]------------
[  180.198469] NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out 9028 ms
[  180.198496] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x232/0x240
[  180.198509] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq nf_conntrack_netlink xt_addrtype br_netfilter xt_policy jitterentropy_rng drbg ansi_cprng authenc echainiv esp4 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat xfrm_interface xfrm6_tunnel tunnel6 tunnel4 xfrm_user xfrm_algo twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cmac cast5_avx_x86_64 nls_utf8 cast5_generic cast_common cifs ctr ecb des_generic libdes algif_skcipher camellia_generic cifs_arc4 cifs_md4 dns_resolver camellia_aesni_avx2 fscache netfs camellia_aesni_avx_x86_64 camellia_x86_64 xcbc md4 algif_hash af_alg nvme_fabrics overlay sunrpc binfmt_misc nft_fib_ipv6 nft_nat nft_fib_ipv4 nft_fib nls_ascii nls_cp437 vfat fat intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp bridge coretemp stp llc kvm_intel cfg80211 kvm rfkill
[  180.198634]  snd_usb_audio uvcvideo videobuf2_vmalloc uvc videobuf2_memops videobuf2_v4l2 snd_usbmidi_lib irqbypass rtsx_usb_ms snd_hwdep iTCO_wdt intel_cstate memstick mei_hdcp videodev mei_wdt mei_pxp intel_pmc_bxt snd_rawmidi snd_seq_device iTCO_vendor_support snd_pcm ftdi_sio videobuf2_common intel_uncore usbserial snd_timer mei_me intel_wmi_thunderbolt snd mc think_lmi watchdog ee1004 firmware_attributes_class wmi_bmof mei soundcore joydev intel_pmc_core int3400_thermal acpi_thermal_rel acpi_tad acpi_pad button evdev nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables msr parport_pc ppdev lp nfnetlink parport fuse loop efi_pstore configfs ip_tables x_tables autofs4 btrfs blake2b_generic hid_logitech_hidpp hid_logitech_dj rtsx_usb_sdmmc mmc_core rtsx_usb hid_jabra hid_generic dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear md_mod usbhid hid ext4 crc16 mbcache jbd2 crc32c_generic
[  180.198772]  i915 i2c_algo_bit drm_buddy crc32_pclmul drm_display_helper crc32c_intel nvme drm_kms_helper nvme_core cec ghash_clmulni_intel sha512_ssse3 rc_core r8169 sha512_generic ahci t10_pi xhci_pci libahci ttm realtek crc64_rocksoft_generic libata xhci_hcd crc64_rocksoft crc_t10dif drm mdio_devres aesni_intel e1000e crct10dif_generic usbcore scsi_mod libphy crypto_simd cryptd crct10dif_pclmul i2c_i801 crc64 i2c_smbus crct10dif_common scsi_common usb_common fan video wmi
[  180.198837] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.4.0-1-amd64 #1  Debian 6.4.4-2
[  180.198843] Hardware name: LENOVO 30E30051UK/1052, BIOS S0AKT3AA 04/25/2023
[  180.198846] RIP: 0010:dev_watchdog+0x232/0x240
[  180.198852] Code: ff ff ff 48 89 df c6 05 36 03 05 01 01 e8 d6 3d fa ff 45 89 f8 44 89 f1 48 89 de 48 89 c2 48 c7 c7 20 85 cf a9 e8 3e 17 70 ff <0f> 0b e9 2d ff ff ff 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90
[  180.198857] RSP: 0018:ffffa264c026ce70 EFLAGS: 00010286
[  180.198862] RAX: 0000000000000000 RBX: ffff948781f58000 RCX: 0000000000000000
[  180.198866] RDX: 0000000000000104 RSI: 0000000000000027 RDI: 00000000ffffffff
[  180.198870] RBP: ffff948781f584c8 R08: 0000000000000000 R09: ffffa264c026cd00
[  180.198873] R10: 0000000000000003 R11: ffffffffaa2d26a8 R12: ffff948781f4f400
[  180.198875] R13: ffff948781f5841c R14: 0000000000000000 R15: 0000000000002344
[  180.198879] FS:  0000000000000000(0000) GS:ffff9496b5680000(0000) knlGS:0000000000000000
[  180.198883] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  180.198886] CR2: 0000561606a0b170 CR3: 0000000146c82002 CR4: 0000000000770ee0
[  180.198890] PKRU: 55555554
[  180.198892] Call Trace:
[  180.198896]  <IRQ>
[  180.198899]  ? dev_watchdog+0x232/0x240
[  180.198904]  ? __warn+0x81/0x130
[  180.198916]  ? dev_watchdog+0x232/0x240
[  180.198920]  ? report_bug+0x191/0x1c0
[  180.198929]  ? prb_read_valid+0x1b/0x30
[  180.198938]  ? handle_bug+0x3c/0x80
[  180.198945]  ? exc_invalid_op+0x17/0x70
[  180.198951]  ? asm_exc_invalid_op+0x1a/0x20
[  180.198960]  ? dev_watchdog+0x232/0x240
[  180.198965]  ? __pfx_dev_watchdog+0x10/0x10
[  180.198969]  call_timer_fn+0x24/0x130
[  180.198979]  ? __pfx_dev_watchdog+0x10/0x10
[  180.198982]  __run_timers+0x222/0x2c0
[  180.198992]  run_timer_softirq+0x2f/0x50
[  180.199000]  __do_softirq+0xf1/0x301
[  180.199007]  __irq_exit_rcu+0xb5/0x130
[  180.199016]  sysvec_apic_timer_interrupt+0xa2/0xd0
[  180.199027]  </IRQ>
[  180.199029]  <TASK>
[  180.199031]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  180.199037] RIP: 0010:cpuidle_enter_state+0xcc/0x440
[  180.199044] Code: da fa 58 ff e8 b5 f0 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 73 06 58 ff 45 84 ff 0f 85 56 02 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 85 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d
[  180.199048] RSP: 0018:ffffa264c0193e90 EFLAGS: 00000246
[  180.199052] RAX: ffff9496b5680000 RBX: ffffc264bfaa8e00 RCX: 0000000000000000
[  180.199055] RDX: 0000000000000002 RSI: ffffffffa9c3fd02 RDI: ffffffffa9c2c5ed
[  180.199058] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000033483483
[  180.199061] R10: ffff9496b56b1d84 R11: 00000000000006b1 R12: ffffffffaa398300
[  180.199064] R13: 00000029f4a9f550 R14: 0000000000000002 R15: 0000000000000000
[  180.199071]  cpuidle_enter+0x2d/0x40
[  180.199081]  do_idle+0x217/0x270
[  180.199089]  cpu_startup_entry+0x1d/0x20
[  180.199095]  start_secondary+0x134/0x160
[  180.199104]  secondary_startup_64_no_verify+0x10b/0x10b
[  180.199115]  </TASK>
[  180.199117] ---[ end trace 0000000000000000 ]---
[  226.051135] r8169 0000:03:00.0 enp3s0: Link is Down
[  275.843212] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx