Subject: Transmit timeouts with forcedeth card in 2.6.30.4

From: Chris Webb <[email-masked]>
Date: Thu, 17 Sep 2009 16:47:24 +0100

Hi. I'm occasionally seeing the following transmit time-out from a forcedeth
ethernet interface with linux 2.6.30.4. We have a cluster of identical machines
with Supermicro H8DMT motherboards on which there are two such interfaces
integrated, and these errors appear on random machines across the cluster,
about once every two or three weeks per machine.

Following this error, it is impossible to ping in or out, but it is restored
back to life by a simple

  ip link set eth0 down
  ip link set eth0 up

The eth0 is enslaved to a bridge and has the mtu set at the default 1500. A
second forcedeth nic in the same machine with mtu set at 9000 and not joined to
bridge doesn't seem to suffer from this problem, although it may have much
lower traffic in any case.

  forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64.
  forcedeth 0000:00:08.0: setting latency timer to 64
  nv_probe: set workaround bit for reversed mac addr
  forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:67:7d:7c
  forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3
  [...]

  WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x130/0x1da()
  Hardware name: H8DMT
  NETDEV WATCHDOG: eth0 (forcedeth): transmit timed out
  Modules linked in:
  Pid: 0, comm: swapper Tainted: G   M       2.6.30.4-elastic-lon-b #2
  Call Trace:
  <IRQ>  [<ffffffff80257fa5>] warn_slowpath_common+0x77/0x8f
  [<ffffffff8025800a>] warn_slowpath_fmt+0x3c/0x3e
  [<ffffffff80260a27>] ? lock_timer_base+0x27/0x4d
  [<ffffffff8058a872>] dev_watchdog+0x130/0x1da
  [<ffffffff80260068>] ? ptrace_detach+0x86/0x95
  [<ffffffff802607fe>] run_timer_softirq+0x164/0x1dc
  [<ffffffff8058a742>] ? dev_watchdog+0x0/0x1da
  [<ffffffff8025ca61>] __do_softirq+0xdc/0x1e3
  [<ffffffff8022bf6c>] call_softirq+0x1c/0x28
  [<ffffffff8022d7c8>] do_softirq+0x34/0x72
  [<ffffffff8025c6ed>] irq_exit+0x3f/0x8b
  [<ffffffff8023ab59>] smp_apic_timer_interrupt+0x89/0x97
  [<ffffffff8022b983>] apic_timer_interrupt+0x13/0x20
  <EOI>  [<ffffffff80231919>] ? default_idle+0x9d/0x118
  [<ffffffff80231c41>] ? c1e_idle+0xe3/0xea
  [<ffffffff8026efb6>] ? atomic_notifier_call_chain+0xf/0x11
  [<ffffffff8022a25c>] ? cpu_idle+0x52/0xb6
  [<ffffffff808ce222>] ? start_secondary+0x1a4/0x1a8
  ---[ end trace 97f305223326b99b ]---
  eth0: Got tx_timeout. irq: 00000036
  eth0: Ring at 42596c000
  eth0: Dumping tx registers
   0: 00002036 00000000 00000003 0006000d 00000000 00000000 00000000 00000000
  20: 00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000
  40: 0420e20e 0000a855 00002e20 00000000 00000000 00000000 00000000 00000000
  60: 00000000 00000000 00000000 0000ffff 0000ffff 0000ffff 0000ffff 00000000
  80: 003b0f3c 00040001 00000000 007f0080 0000061c 00000001 00200000 80007f30
  a0: 0016070f 00000016 67483000 00007c7d 00000001 00000000 00000000 00000000
  c0: 10000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
  e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001
  100: 2596e000 2596c000 01ff00ff 00008000 00010064 00000000 0000004f 2596e150
  120: 2596cb50 204308c0 a000fff2 00000000 00000000 2596e15c 2596cb50 01e08000
  140: 00304120 80c02600 00000004 00000004 00000008 00000000 00000000 00000000
  160: 00000004 00000004 00000004 00000004 01ff0080 0000c000 00000000 00000000
  180: 00000006 00000008 0294796d 00008103 0000004a 00004000 00000591 0000c183
  1a0: 00000006 00000008 0294796d 00008103 0000004a 00004000 0000059e 0000c583
  1c0: 00000006 00000008 0294796d 00008103 0000004a 00004000 00000596 0000c583
  1e0: 00000006 00000008 0294796d 00008103 0000004a 00004000 0000059b 0000c583
  200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  260: 00000000 00000000 fe027001 00000100 00000011 000000a3 fe027011 000001a3
  280: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  2a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  2c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  2e0: 00000000 00000000 00000000 00000000 00000000 00000001 00000001 00000001
  300: 80212000 00000000 00000000 00000000 00000000 00002000 00000000 00000000
  320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  340: 00000000 00000000 00000000 00000000 00000000 00000020 d104438f 00000000
  360: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  380: 00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000000
  3a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  3c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  3e0: 06255300 00701365 00000000 00000000 00000032 00000000 00000000 00000000
  400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  4a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  4c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  4e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  5a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  5c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  5e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  600: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  eth0: Dumping tx ring
  000: 00000005 d4ea5802 20000029 // 00000006 92398802 20000029 // 00000004 423a0c02 20000029 // 00000
  004: 00000005 dc2bf422 20000040 // 00000005 dc2b8422 20000062 // 00000005 dc2b8c22 2000005c // 00000
  008: 00000008 20437c22 20000095 // 00000008 20432022 20000095 // 00000008 20435c22 20000095 // 00000
  00c: 00000006 0aa3d822 2000006e // 00000004 fa8d3022 200005c6 // 00000005 7806f022 2000059e // 00000
  010: 00000008 20435022 20000095 // 00000008 20437022 20000095 // 00000008 20435822 20000095 // 00000
  014: 00000008 20430822 20000095 // 00000006 0aa38822 20000069 // 00000005 61eb3022 200005e9 // 00000
  018: 00000005 7806d822 20000599 // 00000008 20436c22 20000090 // 00000008 20432c22 20000090 // 00000
  01c: 00000008 20436822 20000090 // 00000005 7806f822 200005b7 // 00000008 206c8c22 2000003c // 00000
  020: 00000008 206cc822 20000090 // 00000008 206ce022 20000090 // 00000008 206cc422 20000090 // 00000
  024: 00000008 206c9022 20000090 // 00000008 206c8422 20000090 // 00000008 206cb422 2000003d // 00000
  028: 00000005 0c077022 200005d5 // 00000005 7806a822 20000599 // 00000008 206ca822 20000090 // 00000
  02c: 00000008 206cbc22 20000090 // 00000008 206c9c22 20000090 // 00000006 0aa3ac02 20000029 // 00000
  030: 00000006 0aa3dc22 20000090 // 00000006 0aa3cc22 20000090 // 00000006 0aa3d422 20000090 // 00000
  034: 00000006 0aa3bc22 20000090 // 00000006 0aa3a022 20000090 // 00000006 0aa39402 20000029 // 00000
  038: 00000001 c34fe402 20000029 // 00000004 3683e022 200005e1 // 00000006 0aa3e422 200000a0 // 00000
  03c: 00000006 0aa3c422 20000090 // 00000006 0aa3f822 20000090 // 00000006 0aa3b422 20000090 // 00000
  040: 00000001 c34fdc22 20000069 // 00000001 c34fd402 20000029 // 00000008 25736022 200000a0 // 00000
  044: 00000008 25731022 20000090 // 00000008 25734422 20000090 // 00000008 25735c22 20000090 // 00000
  048: 00000008 25732422 20000090 // 00000005 dc2b9402 2c000059 // 00000008 25735002 20000029 // 00000
  04c: 00000006 000fbc02 2c000059 // 00000008 25abc002 20000029 // 00000008 25abb002 20000029 // 00000
  050: 00000005 dc2bd002 20000029 // 00000008 25ab8c02 20000029 // 00000006 249f5402 20000029 // 00000
  054: 00000008 25abfc02 20000029 // 00000006 249f5802 20000029 // 00000005 dc2ba002 20000029 // 00000
  058: 00000006 249f0402 20000029 // 00000008 254de402 20000029 // 00000008 254df802 20000029 // 00000
  05c: 00000005 dc2ba822 20000041 // 00000006 25091c02 20000029 // 00000006 25096422 20000041 // 00000
  060: 00000005 dc2b9002 20000029 // 00000006 25091402 20000029 // 00000006 249f6c22 20000041 // 00000
  064: 00000008 2022b002 20000029 // 00000006 25090c02 20000029 // 00000005 e597f802 20000029 // 00000
  068: 00000006 25093402 20000029 // 00000006 25093002 20000029 // 00000006 000f9802 20000029 // 00000
  06c: 00000006 249f2422 2000003f // 00000006 000fcc02 20000029 // 00000006 249f1802 20000029 // 00000
  070: 00000005 dc2ba402 20000029 // 00000005 dc2bc002 20000029 // 00000005 e597f402 20000029 // 00000
  074: 00000008 206cd802 20000029 // 00000005 e580d402 20000029 // 00000006 000ff002 20000029 // 00000
  078: 00000006 249f2c02 20000029 // 00000006 249f7802 20000029 // 00000008 2022c802 20000029 // 00000
  07c: 00000006 000f9c02 20000029 // 00000006 249f3802 20000029 // 00000005 dc2bec02 20000029 // 00000
  080: 00000005 e597b802 20000029 // 00000003 9a19cc02 20000029 // 00000003 9a19f802 20000029 // 00000
  084: 00000004 25992c02 20000029 // 00000006 249f2802 20000029 // 00000004 25994c02 20000029 // 00000
  088: 00000006 000f8402 20000029 // 00000005 dc2b9c02 20000029 // 00000006 25091002 20000029 // 00000
  08c: 00000006 000f9422 2000003f // 00000006 000fa822 2000003f // 00000006 9239ac02 20000029 // 00000
  090: 00000006 0aa38402 20000029 // 00000001 c34fc002 20000029 // 00000008 2022ac02 20000029 // 00000
  094: 00000001 c34f9802 20000029 // 00000008 20433c02 20000029 // 00000005 e580d802 20000029 // 00000
  098: 00000005 e5978002 20000029 // 00000001 c34fbc22 2000003f // 00000001 c34fcc02 20000029 // 00000
  09c: 00000006 25096802 20000029 // 00000001 c34ff802 20000029 // 00000005 e580f402 20000029 // 00000
  0a0: 00000005 e580b802 20000029 // 00000006 000fe802 20000029 // 00000001 c34fa402 20000029 // 00000
  0a4: 00000005 e580b402 20000029 // 00000001 c34ffc02 20000029 // 00000001 c34f9002 20000029 // 00000
  0a8: 00000001 c34fe002 20000029 // 00000005 dc2b9802 20000029 // 00000001 c34ff002 20000029 // 00000
  0ac: 00000006 9239dc02 20000029 // 00000001 c34fe802 20000029 // 00000006 9239c802 20000029 // 00000
  0b0: 00000001 c34f9402 20000029 // 00000006 9239d002 20000029 // 00000005 dc2bc402 20000029 // 00000
  0b4: 00000006 249f1c02 20000029 // 00000006 000fc402 20000029 // 00000001 c34fd802 20000029 // 00000
  0b8: 00000006 249f5c02 20000029 // 00000001 c34f8002 20000029 // 00000006 92398c02 20000029 // 00000
  0bc: 00000001 c34f8402 20000029 // 00000001 c34ff402 20000029 // 00000008 206cb802 20000029 // 00000
  0c0: 00000005 e5978c02 20000029 // 00000005 e5978802 20000029 // 00000001 c34fc402 20000029 // 00000
  0c4: 00000005 e597e002 20000029 // 00000003 9a199c02 20000029 // 00000005 dc2bb402 20000029 // 00000
  0c8: 00000005 e597fc02 20000029 // 00000003 9a19d002 20000029 // 00000005 e597d402 20000029 // 00000
  0cc: 00000005 e580fc02 20000029 // 00000003 9a198c02 20000029 // 00000005 e580a402 20000029 // 00000
  0d0: 00000003 9a19b002 20000029 // 00000004 25652002 20000029 // 00000004 25656002 20000029 // 00000
  0d4: 00000005 e580dc02 20000029 // 00000004 25650802 20000029 // 00000005 e580ec02 20000029 // 00000
  0d8: 00000006 000fe402 20000029 // 00000004 23942802 20000029 // 00000004 25997c02 20000029 // 00000
  0dc: 00000004 25994802 20000029 // 00000004 25995802 20000029 // 00000006 92399802 20000029 // 00000
  0e0: 00000006 92398002 20000029 // 00000004 23940802 20000029 // 00000006 9239d402 20000029 // 00000
  0e4: 00000004 25993802 20000029 // 00000004 25995c02 20000029 // 00000004 25996002 20000029 // 00000
  0e8: 00000004 25997402 20000029 // 00000003 9a19dc02 20000029 // 00000004 25917c02 20000029 // 00000
  0ec: 00000004 25914c02 20000029 // 00000004 25914802 20000029 // 00000004 25911402 20000029 // 00000
  0f0: 00000006 9239b402 20000029 // 00000001 c34fac02 20000029 // 00000006 92399002 20000029 // 00000
  0f4: 00000006 92398402 20000029 // 00000001 c34fb002 20000029 // 00000005 d4ea4802 20000029 // 00000
  0f8: 00000003 f5818c02 20000029 // 00000008 206ccc02 20000029 // 00000003 f581ec02 20000029 // 00000
  0fc: 00000008 206cd402 20000029 // 00000003 f581c002 20000029 // 00000006 9239f802 20000029 // 00000

Best wishes,

Chris.