Subject: Re: e1000e interface hang on 82574L

From: Chris Boot <[email-masked]>
Date: Sat, 17 Mar 2012 15:59:30 +0000

On 16/01/2012 16:04, Chris Boot wrote:
> On 16/01/2012 15:56, Wyborny, Carolyn wrote:
>>
>>
>>> -----Original Message-----
>>> From: Chris Boot [mailto:[email-masked]]
>>> Sent: Sunday, January 15, 2012 3:11 AM
>>> To: Wyborny, Carolyn
>>> Cc: netdev; lkml; [email-masked]
>>> Subject: Re: e1000e interface hang on 82574L
>>>
>>> On 04/01/2012 17:12, Chris Boot wrote:
>>>> On 03/01/2012 00:02, Wyborny, Carolyn wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: [email-masked] [mailto:netdev-
>>> [email-masked]]
>>>>>> On Behalf Of Chris Boot
>>>>>> Sent: Saturday, December 31, 2011 1:32 AM
>>>>>> To: netdev; lkml; [email-masked]
>>>>>> Subject: Re: e1000e interface hang on 82574L
>>>>>>
>>>>>> On 27 Dec 2011, at 22:01, Chris Boot wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Another networking issue I've run into, this time with e1000e
>>> (Intel
>>>>>> Corporation 82574L Gigabit). My new VM cluster appears to drop a NIC
>>> -
>>>>>> the port stops responding within Linux and shows the link as being
>>> down
>>>>>> with ethtool. My ISP says 'Ports running Half Duplex or reduced
>>> speed'
>>>>>> on the port.
>>>>>>>
>>>>>>> When the port stops working I see this in dmesg:
>>>>>>>
>>>>>>> [35481.659629] ------------[ cut here ]------------
>>>>>>> [35481.667837] WARNING: at net/sched/sch_generic.c:255
>>>>>> dev_watchdog+0xe9/0x148()
>>>>>>> [35481.676370] Hardware name: X9SCL/X9SCM
>>>>>>> [35481.684793] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0
>>> timed
>>>>>> out
>>>>>>> [35481.684795] Modules linked in: hmac sha256_generic dlm configfs
>>>>>> ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats
>>>>>> cpufreq_conservative cpufreq_userspace cpufreq_powersave microcode
>>>>>> xt_NOTRACK ip_set_hash_net act_police cls_basic cls_flow cls_fw
>>> cls_u32
>>>>>> sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_connlimit
>>>>>> xt_realm xt_addrtype ip_set_hash_ip iptable_raw xt_comment xt_recent
>>>>>> ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN
>>>>>> ipt_ecn ipt_CLUSTERIP ipt_ah nf_nat_tftp nf_nat_snmp_basic
>>>>>> nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc
>>>>>> nf_nat_h323 nf_nat_ftp ip6_queue nf_nat_amanda xt_set ip_set
>>>>>> nf_conntrack_tftp nf_conntrack_sip nf_conntrack_sane
>>>>>> nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp
>>>>>> nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns
>>>>>> nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323
>>>>>> nf_conntrack_ftp ts_kmp nf_conntrack_amanda xt_TPROXY xt_NFLOG
>>>>>> nfnetlink_log nf_tproxy_core xt_time xt_TCPMSS xt_tcpmss xt_sctp
>>>>>> xt_policy xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_multiport
>>> xt_mark
>>>>>> xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP
>>>>>> xt_dscp xt_dccp xt_connmark xt_CLASSIFY xt_AUDIT ip6t_LOG
>>> ip6t_REJECT
>>>>>> nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_raw ipt_LOG
>>>>>> xt_tcpudp ip6table_mangle xt_state iptable_nat nf_nat
>>> nf_conntrack_ipv4
>>>>>> nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter
>>>>>> ip_tables ip6table_filter ip6_tables x_tables bridge stp bonding
>>>>>> w83627ehf hwmon_vid coretemp sha1_ssse3 sha1_generic crc32c_intel
>>>>>> aesni_intel cryptd aes_x86_64 aes_generic ipmi_poweroff ipmi_devintf
>>>>>> ipmi_si ipmi_msghandler vhost_net macvtap macvlan tun drbd lru_cache
>>> cn
>>>>>> loop kvm_intel kvm snd_pcm snd_timer snd iTCO_wdt soundcore psmouse
>>>>>> snd_page_alloc i2c_i801 i2c_core cdc_acm iTCO_vendor_support joydev
>>>>>> evdev serio_raw processor button pcspkr thermal_sys ext4 mbcache
>>> jbd2
>>>>>> crc16 dm_mod raid1 md_mod sd_mod crc_t10dif usb_storage uas usbhid
>>> hid
>>>>>> ahci libahci libata igb ehci_hcd scsi_mod usbcore e1000e dca
>>> usb_common
>>>>>> [last unloaded: scsi_wait_scan]
>>>>>>> [35481.685740] Pid: 0, comm: swapper/4 Not tainted 3.2.0-rc6+ #4
>>>>>>> [35481.685744] Call Trace:
>>>>>>> [35481.685746]<IRQ>  [<ffffffff810467ed>] ?
>>>>>> warn_slowpath_common+0x78/0x8c
>>>>>>> [35481.685849] [<ffffffff81046899>] ? warn_slowpath_fmt+0x45/0x4a
>>>>>>> [35481.685875] [<ffffffff810aeaa0>] ?
>>>>>> perf_event_task_tick+0x166/0x1ab
>>>>>>> [35481.686018] [<ffffffff81294219>] ? netif_tx_lock+0x40/0x72
>>>>>>> [35481.686090] [<ffffffff8129437a>] ? dev_watchdog+0xe9/0x148
>>>>>>> [35481.686136] [<ffffffff81051e58>] ? run_timer_softirq+0x19a/0x261
>>>>>>> [35481.686176] [<ffffffff81294291>] ? netif_tx_unlock+0x46/0x46
>>>>>>> [35481.686215] [<ffffffff810659bb>] ? timekeeping_get_ns+0xd/0x2a
>>>>>>> [35481.686286] [<ffffffff8104bdd4>] ? __do_softirq+0xb9/0x177
>>>>>>> [35481.686365] [<ffffffff81341d6c>] ? call_softirq+0x1c/0x30
>>>>>>> [35481.686530] [<ffffffff8100f841>] ? do_softirq+0x3c/0x7b
>>>>>>> [35481.686580] [<ffffffff8104c03c>] ? irq_exit+0x3c/0x9a
>>>>>>> [35481.686742] [<ffffffff81023e58>] ?
>>>>>> smp_apic_timer_interrupt+0x74/0x82
>>>>>>> [35481.686820] [<ffffffff813405de>] ?
>>> apic_timer_interrupt+0x6e/0x80
>>>>>>> [35481.686826]<EOI>  [<ffffffff811ddf49>] ? intel_idle+0xea/0x119
>>>>>>> [35481.686991] [<ffffffff811ddf28>] ? intel_idle+0xc9/0x119
>>>>>>> [35481.687051] [<ffffffff8125dce3>] ? cpuidle_idle_call+0xec/0x179
>>>>>>> [35481.687089] [<ffffffff8100d255>] ? cpu_idle+0xa1/0xe8
>>>>>>> [35481.687143] [<ffffffff810706ee>] ?
>>> arch_local_irq_restore+0x2/0x8
>>>>>>> [35481.687189] [<ffffffff8132d191>] ? start_secondary+0x1d5/0x1db
>>>>>>> [35481.687234] ---[ end trace 01e9907674757948 ]---
>>>>>>> [35481.687817] e1000e 0000:05:00.0: eth2: Reset adapter
>>>>>>>
>>>>>>> To try to regain connectivity I bring down the bond and the
>>> interface
>>>>>> (eth2), then unload e1000e. Upon loading the module again:
>>>>>>>
>>>>>>> [36021.888962] e1000e: Intel(R) PRO/1000 Network Driver - 1.5.1-k
>>>>>>> [36021.900258] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
>>>>>>> [36021.911446] e1000e 0000:00:19.0: PCI INT A ->  GSI 20 (level,
>>> low) -
>>>>>>> IRQ 20
>>>>>>> [36021.923204] e1000e 0000:00:19.0: setting latency timer to 64
>>>>>>> [36021.923372] e1000e 0000:00:19.0: irq 45 for MSI/MSI-X
>>>>>>> [36022.202737] e1000e 0000:00:19.0: eth2: (PCI
>>> Express:2.5GT/s:Width
>>>>>> x1) 00:25:90:56:ac:75
>>>>>>> [36022.214480] e1000e 0000:00:19.0: eth3: Intel(R) PRO/1000 Network
>>>>>> Connection
>>>>>>> [36022.227506] e1000e 0000:00:19.0: eth3: MAC: 10, PHY: 11, PBA No:
>>>>>> FFFFFF-0FF
>>>>>>> [36022.239789] e1000e 0000:05:00.0: Disabling ASPM L0s
>>>>>>> [36022.239805] e1000e 0000:05:00.0: enabling device (0000 ->  0002)
>>>>>>> [36022.239829] e1000e 0000:05:00.0: PCI INT A ->  GSI 16 (level,
>>> low) -
>>>>>>> IRQ 16
>>>>>>> [36022.239921] e1000e 0000:05:00.0: setting latency timer to 64
>>>>>>> [36022.240963] e1000e 0000:05:00.0: irq 64 for MSI/MSI-X
>>>>>>> [36022.240995] e1000e 0000:05:00.0: irq 65 for MSI/MSI-X
>>>>>>> [36022.241028] e1000e 0000:05:00.0: irq 66 for MSI/MSI-X
>>>>>>> [36022.241596] e1000e 0000:05:00.0: PCI INT A disabled
>>>>>>> [36022.241606] e1000e: probe of 0000:05:00.0 failed with error -2
>>>>>>> [36022.304706] udevd[3634]: renamed network interface eth2 to eth3
>>>>>>>
>>>>>>> I then don't get an eth2 interface. Only a reboot brings the
>>> interface
>>>>>> back. This has happened twice so far on this server in the past
>>> week,
>>>>>> both times using v3.2-rc7-3-g4962516.
>>>>>>>
>>>>>>> lspci -vnn shows:
>>>>>>>
>>>>>>> 05:00.0 Ethernet controller [0200]: Intel Corporation 82574L
>>> Gigabit
>>>>>> Network Connection [8086:10d3]
>>>>>>> Subsystem: Super Micro Computer Inc Device [15d9:0000]
>>>>>>> Flags: bus master, fast devsel, latency 0, IRQ 16
>>>>>>> Memory at fbd00000 (32-bit, non-prefetchable) [size=128K]
>>>>>>> I/O ports at e000 [size=32]
>>>>>>> Memory at fbd20000 (32-bit, non-prefetchable) [size=16K]
>>>>>>> Capabilities: [c8] Power Management version 2
>>>>>>> Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>>>>>>> Capabilities: [e0] Express Endpoint, MSI 00
>>>>>>> Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>>>>>>> Capabilities: [100] Advanced Error Reporting
>>>>>>> Capabilities: [140] Device Serial Number 00-25-90-ff-ff-56-ac-
>>>>>> 74
>>>>>>> Kernel driver in use: e1000e
>>>>>>
>>>>>> I've just had this happen on my other (identical) server with a
>>> nearly
>>>>>> identical trace. Is there anything I can do do avoid this at all or
>>> at
>>>>>> least help narrow down the problem?
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>>
>>>>>> -- 
>>>>>> Chris Boot
>>>>>> [email-masked]
>>>>>>
>>>>>> -- 
>>>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>>>> the body of a message to [email-masked]
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> Hello,
>>>>>
>>>>> Sorry for the delay in responding. We have seen some hang issues
>>> using
>>>>> MSI-X on 82574 parts. Can you try reloading the driver the IntMode
>>>>> module parameter. IntMode=1 (you'll need a setting for each device in
>>>>> the system so two adapters would be IntMode=1,1) See if that changes
>>>>> the symptom you are seeing with this part. That setting will make
>>> sure
>>>>> the adapter uses MSI interrupts instead of MSI-X.
>>>>
>>>> Carolyn,
>>>>
>>>> I'll give this a go next time I reproduce it. I built a new kernel
>>> with
>>>> more debugging and so far it hasn't yet triggered again...
>>>
>>> Upgrading to a more recent 3.2-rc snapshot seems to have cured the
>>> problem - I haven't had an interface stop responding since. Must have
>>> been some seemingly unrelated patch that I can't seem to locate.
>>>
>>> Cheers,
>>> Chris
>>>
>>> -- 
>>> Chris Boot
>>> [email-masked]
>> Thanks for letting me know Chris.  For my own edification, are you
>> still configured with MSI-X?
>
> Carolyn,
>
> I have made no changes to my configuration to change the interrupt
> format. I see the following in dmesg at boot:
>
> [    3.276819] e1000e: Intel(R) PRO/1000 Network Driver - 1.5.1-k
> [    3.288193] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
>
> [    3.299842] e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low)
> -> IRQ 20
> [    3.299909] e1000e 0000:00:19.0: setting latency timer to 64
> [    3.352929] e1000e 0000:00:19.0: irq 45 for MSI/MSI-X
> [    3.710080] e1000e 0000:00:19.0: eth2: (PCI Express:2.5GT/s:Width
> x1) 00:25:90:56:ac:75
> [    3.710082] e1000e 0000:00:19.0: eth2: Intel(R) PRO/1000 Network
> Connection
> [    3.710670] e1000e 0000:00:19.0: eth2: MAC: 10, PHY: 11, PBA No:
> FFFFFF-0FF
>
> [    3.710678] e1000e 0000:05:00.0: Disabling ASPM L0s
> [    3.710850] e1000e 0000:05:00.0: PCI INT A -> GSI 16 (level, low)
> -> IRQ 16
> [    3.710951] e1000e 0000:05:00.0: setting latency timer to 64
> [    3.712757] e1000e 0000:05:00.0: irq 64 for MSI/MSI-X
> [    3.712787] e1000e 0000:05:00.0: irq 65 for MSI/MSI-X
> [    3.712805] e1000e 0000:05:00.0: irq 66 for MSI/MSI-X
> [    3.830364] e1000e 0000:05:00.0: eth3: (PCI Express:2.5GT/s:Width
> x1) 00:25:90:56:ac:74
> [    3.830366] e1000e 0000:05:00.0: eth3: Intel(R) PRO/1000 Network
> Connection
> [    3.830510] e1000e 0000:05:00.0: eth3: MAC: 3, PHY: 8, PBA No:
> FFFFFF-0FF
>
> /proc/interrupts shows:
>
>  45:     615958          0          0          0          0          0
>          0          0  IR-PCI-MSI-edge      eth3
>  64:   65126106          0          0          0          0          0
>          0          0  IR-PCI-MSI-edge      eth2-rx-0
>  65:   52700392          0          0          0          0          0
>          0          0  IR-PCI-MSI-edge      eth2-tx-0
>  66:          2          0          0          0          0          0
>          0          0  IR-PCI-MSI-edge      eth2

Carolyn,

I've just had the opportunity to upgrade to a 3.2.9 kernel on these
systems and have made sure e1000e is loaded with IntMode=1,1. One of the
servers was only up 5.5 hours before the NIC has crashed/stopped working
again.

Here is the latest dmesg after the failure:

[    3.254553] e1000e: Intel(R) PRO/1000 Network Driver - 1.5.1-k
[    3.265852] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[    3.266034] e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) ->
IRQ 20
[    3.266067] e1000e 0000:00:19.0: setting latency timer to 64
[    3.266460] e1000e 0000:00:19.0: (unregistered net_device): Interrupt
Mode set to 1
[    3.266800] e1000e 0000:00:19.0: irq 45 for MSI/MSI-X
[    3.611840] e1000e 0000:00:19.0: eth2: (PCI Express:2.5GT/s:Width x1)
00:25:90:56:ac:75
[    3.611855] e1000e 0000:00:19.0: eth2: Intel(R) PRO/1000 Network
Connection
[    3.612303] e1000e 0000:00:19.0: eth2: MAC: 10, PHY: 11, PBA No:
FFFFFF-0FF
[    3.612350] e1000e 0000:05:00.0: Disabling ASPM L0s
[    3.612594] e1000e 0000:05:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[    3.612812] e1000e 0000:05:00.0: setting latency timer to 64
[    3.613582] e1000e 0000:05:00.0: (unregistered net_device): Interrupt
Mode set to 1
[    3.614156] e1000e 0000:05:00.0: irq 64 for MSI/MSI-X
[    3.734442] e1000e 0000:05:00.0: eth3: (PCI Express:2.5GT/s:Width x1)
00:25:90:56:ac:74
[    3.734465] e1000e 0000:05:00.0: eth3: Intel(R) PRO/1000 Network
Connection
[    3.734689] e1000e 0000:05:00.0: eth3: MAC: 3, PHY: 8, PBA No: FFFFFF-0FF
[   13.799848] e1000e 0000:05:00.0: irq 64 for MSI/MSI-X
[   13.855646] e1000e 0000:05:00.0: irq 64 for MSI/MSI-X
[   14.031739] e1000e 0000:00:19.0: irq 45 for MSI/MSI-X
[   14.087566] e1000e 0000:00:19.0: irq 45 for MSI/MSI-X
[   16.112504] e1000e: eth2 NIC Link is Up 100 Mbps Full Duplex, Flow
Control: None
[   16.124129] e1000e 0000:05:00.0: eth2: 10/100 speed: disabling TSO

And here is the output just as it hangs:

[19745.327241] ------------[ cut here ]------------
[19745.334501] WARNING: at
/build/buildd-linux-2.6_3.2.9-1-amd64-KTPapN/linux-2.6-3.2.9/debian/build/source_amd64_none/net/sched/sch_generic.c:255
dev_watchdog+0xe9/0x148()
[19745.350441] Hardware name: X9SCL/X9SCM
[19745.358859] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out
[19745.367287] Modules linked in: hmac sha256_generic dlm configfs
ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats
cpufreq_conservative cpufreq_userspace cpufreq_powersave microcode
ip6_queue xt_TCPMSS xt_sctp ip6t_LOG ip6t_REJECT nf_conntrack_ipv6
ip6table_raw ip6table_mangle ip6table_filter xt_NOTRACK ip_set_hash_net
act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb
sch_hfsc sch_ingress sch_sfq xt_statistic xt_CT xt_time xt_connlimit
xt_realm xt_addrtype ip_set_hash_ip iptable_raw xt_comment xt_recent
xt_policy ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE
ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat_tftp
nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp
nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip
nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp
nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns
nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323
nf_conntrack_ftp xt_TPROXY nf_tproxy_core ip6_tables nf_defrag_ipv6
xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG
nfnetlink_log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange
xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark
xt_CLASSIFY xt_AUDIT ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink
iptable_filter ip_tables x_tables kvm_intel kvm bridge stp bonding
w83627ehf hwmon_vid coretemp sha1_ssse3 sha1_generic crc32c_intel
aesni_intel cryptd aes_x86_64 aes_generic ipmi_poweroff ipmi_devintf
ipmi_si ipmi_msghandler vhost_net macvtap macvlan tun drbd lru_cache cn
loop snd_pcm snd_timer snd soundcore snd_page_alloc iTCO_wdt i2c_i801
psmouse cdc_acm processor i2c_core iTCO_vendor_support serio_raw pcspkr
thermal_sys button evdev joydev ext4 mbcache jbd2 crc16 dm_mod raid1
md_mod sd_mod crc_t10dif usb_storage uas usbhid hid ahci libahci libata
ehci_hcd usbcore igb scsi_mod e1000e usb_common dca [last unloaded:
scsi_wait_scan]
[19745.502559] Pid: 0, comm: swapper/0 Not tainted 3.2.0-2-amd64 #1
[19745.502561] Call Trace:
[19745.502562]  <IRQ>  [<ffffffff81046879>] ? warn_slowpath_common+0x78/0x8c
[19745.502570]  [<ffffffff81046925>] ? warn_slowpath_fmt+0x45/0x4a
[19745.502574]  [<ffffffff8129aa11>] ? netif_tx_lock+0x40/0x72
[19745.502588]  [<ffffffff8129ab72>] ? dev_watchdog+0xe9/0x148
[19745.502601]  [<ffffffff81051f38>] ? run_timer_softirq+0x19a/0x261
[19745.502603]  [<ffffffff8129aa89>] ? netif_tx_unlock+0x46/0x46
[19745.502606]  [<ffffffff81065a73>] ? timekeeping_get_ns+0xd/0x2a
[19745.502609]  [<ffffffff8104be98>] ? __do_softirq+0xb9/0x177
[19745.502612]  [<ffffffff8134892c>] ? call_softirq+0x1c/0x30
[19745.502615]  [<ffffffff8100f8e5>] ? do_softirq+0x3c/0x7b
[19745.502617]  [<ffffffff8104c100>] ? irq_exit+0x3c/0x9a
[19745.502621]  [<ffffffff81023f18>] ? smp_apic_timer_interrupt+0x74/0x82
[19745.502624]  [<ffffffff8134719e>] ? apic_timer_interrupt+0x6e/0x80
[19745.502625]  <EOI>  [<ffffffff81070761>] ? arch_local_irq_save+0x11/0x17
[19745.502631]  [<ffffffff811e45d9>] ? intel_idle+0xea/0x119
[19745.502633]  [<ffffffff811e45b8>] ? intel_idle+0xc9/0x119
[19745.502637]  [<ffffffff812643f7>] ? cpuidle_idle_call+0xec/0x179
[19745.502639]  [<ffffffff8100d248>] ? cpu_idle+0xa5/0xf2
[19745.502641]  [<ffffffff816aab3d>] ? start_kernel+0x3bd/0x3c8
[19745.502643]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
[19745.502645]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
[19745.502646] ---[ end trace 10e791a6f31603fa ]---
[19745.503125] e1000e 0000:05:00.0: eth2: Reset adapter

Once again, rmmod e1000e followed by modprobe e1000e does not fix the
problem:

[20508.158919] e1000e 0000:05:00.0: PCI INT A disabled
[20508.194927] e1000e 0000:00:19.0: PCI INT A disabled
[20511.119765] e1000e: Intel(R) PRO/1000 Network Driver - 1.5.1-k
[20511.130711] e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
[20511.141206] e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) ->
IRQ 20
[20511.151797] e1000e 0000:00:19.0: setting latency timer to 64
[20511.151921] e1000e 0000:00:19.0: (unregistered net_device): Interrupt
Mode set to 1
[20511.162853] e1000e 0000:00:19.0: irq 45 for MSI/MSI-X
[20511.528436] e1000e 0000:00:19.0: eth2: (PCI Express:2.5GT/s:Width x1)
00:25:90:56:ac:75
[20511.539261] e1000e 0000:00:19.0: eth3: Intel(R) PRO/1000 Network
Connection
[20511.550066] e1000e 0000:00:19.0: eth3: MAC: 10, PHY: 11, PBA No:
FFFFFF-0FF
[20511.561027] e1000e 0000:05:00.0: Disabling ASPM L0s
[20511.571883] e1000e 0000:05:00.0: enabling device (0000 -> 0002)
[20511.575224] udevd[5449]: renamed network interface eth2 to eth3
[20511.594234] e1000e 0000:05:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[20511.605703] e1000e 0000:05:00.0: setting latency timer to 64
[20511.605871] e1000e 0000:05:00.0: (unregistered net_device): Interrupt
Mode set to 1
[20511.617706] e1000e 0000:05:00.0: irq 64 for MSI/MSI-X
[20511.617828] e1000e 0000:05:00.0: PCI INT A disabled
[20511.629565] e1000e: probe of 0000:05:00.0 failed with error -2

Please let me know if/how I can debug this further.

Many thanks,
Chris

-- 
Chris Boot
[email-masked].