From: Andrew Morton <akpm@linux-foundation•org>
To: Stephen Hemminger <shemminger@linux-foundation•org>
Cc: netdev@vger•kernel.org, kolo@albatani•cz,
bugzilla-daemon@bugzilla•kernel.org
Subject: Re: Fw: [Bug 14470] New: freez in TCP stack
Date: Wed, 28 Oct 2009 15:13:13 -0700 [thread overview]
Message-ID: <20091028151313.ba4a4d23.akpm@linux-foundation.org> (raw)
In-Reply-To: <20091026084132.57bc3d07@nehalam>
On Mon, 26 Oct 2009 08:41:32 -0700
Stephen Hemminger <shemminger@linux-foundation•org> wrote:
>
>
> Begin forwarded message:
>
> Date: Mon, 26 Oct 2009 12:47:22 GMT
> From: bugzilla-daemon@bugzilla•kernel.org
> To: shemminger@linux-foundation•org
> Subject: [Bug 14470] New: freez in TCP stack
>
Stephen, please retain the bugzilla and reporter email cc's when
forwarding a report to a mailing list.
> http://bugzilla.kernel.org/show_bug.cgi?id=14470
>
> Summary: freez in TCP stack
> Product: Networking
> Version: 2.5
> Kernel Version: 2.6.31
> Platform: All
> OS/Version: Linux
> Tree: Mainline
> Status: NEW
> Severity: high
> Priority: P1
> Component: IPV4
> AssignedTo: shemminger@linux-foundation•org
> ReportedBy: kolo@albatani•cz
> Regression: No
>
>
> We are hiting kernel panics on Dell R610 servers with e1000e NICs; it apears
> usualy under a high network trafic ( around 100Mbit/s) but it is not a rule it
> has happened even on low trafic.
>
> Servers are used as reverse http proxy (varnish).
>
> On 6 equal servers this panic happens aprox 2 times a day depending on network
> load. Machine completly freezes till the management watchdog reboots.
>
Twice a day on six separate machines. That ain't no hardware glitch.
Vaclav, are you able to say whether this is a regression? Did those
machines run 2.6.30 (for example)?
Thanks.
> We had to put serial console on these servers to catch the oops. Is there
> anything else We can do to debug this?
> The RIP is always the same:
>
> RIP: 0010:[<ffffffff814203cc>] [<ffffffff814203cc>]
> tcp_xmit_retransmit_queue+0x8c/0x290
>
> rest of the oops always differs a litle ... here is an example:
>
> RIP: 0010:[<ffffffff814203cc>] [<ffffffff814203cc>]
> tcp_xmit_retransmit_queue+0x8c/0x290
> RSP: 0018:ffffc90000003a40 EFLAGS: 00010246
> RAX: ffff8807e7420678 RBX: ffff8807e74205c0 RCX: 0000000000000000
> RDX: 000000004598a105 RSI: 0000000000000000 RDI: ffff8807e74205c0
> RBP: ffffc90000003a80 R08: 0000000000000003 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> R13: ffff8807e74205c0 R14: ffff8807e7420678 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000001001000 CR4: 00000000000006f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process swapper (pid: 0, threadinfo ffffffff81608000, task ffffffff81631440)
> Stack:
> ffffc90000003a60 0000000000000000 4598a105e74205c0 000000004598a101
> <0> 000000000000050e ffff8807e74205c0 0000000000000003 0000000000000000
> <0> ffffc90000003b40 ffffffff8141ae4a ffff8807e7420678 0000000000000000
> Call Trace:
> <IRQ>
> [<ffffffff8141ae4a>] tcp_ack+0x170a/0x1dd0
> [<ffffffff8141c362>] tcp_rcv_state_process+0x122/0xab0
> [<ffffffff81422c6c>] tcp_v4_do_rcv+0xac/0x220
> [<ffffffff813fd02f>] ? nf_iterate+0x5f/0x90
> [<ffffffff81424b26>] tcp_v4_rcv+0x586/0x6b0
> [<ffffffff813fd0c5>] ? nf_hook_slow+0x65/0xf0
> [<ffffffff81406b70>] ? ip_local_deliver_finish+0x0/0x120
> [<ffffffff81406bcf>] ip_local_deliver_finish+0x5f/0x120
> [<ffffffff8140715b>] ip_local_deliver+0x3b/0x90
> [<ffffffff81406971>] ip_rcv_finish+0x141/0x340
> [<ffffffff8140701f>] ip_rcv+0x24f/0x350
> [<ffffffff813e7ced>] netif_receive_skb+0x20d/0x2f0
> [<ffffffff813e7e90>] napi_skb_finish+0x40/0x50
> [<ffffffff813e82f4>] napi_gro_receive+0x34/0x40
> [<ffffffff8133e0c8>] e1000_receive_skb+0x48/0x60
> [<ffffffff81342342>] e1000_clean_rx_irq+0xf2/0x330
> [<ffffffff813410a1>] e1000_clean+0x81/0x2a0
> [<ffffffff81054ce1>] ? ktime_get+0x11/0x50
> [<ffffffff813eaf1c>] net_rx_action+0x9c/0x130
> [<ffffffff81046940>] ? get_next_timer_interrupt+0x1d0/0x210
> [<ffffffff81041bd7>] __do_softirq+0xb7/0x160
> [<ffffffff8100c27c>] call_softirq+0x1c/0x30
> [<ffffffff8100e04d>] do_softirq+0x3d/0x80
> [<ffffffff81041b0b>] irq_exit+0x7b/0x90
> [<ffffffff8100d613>] do_IRQ+0x73/0xe0
> [<ffffffff8100bb13>] ret_from_intr+0x0/0xa
> <EOI>
> [<ffffffff81296e6c>] ? acpi_idle_enter_bm+0x245/0x271
> [<ffffffff81296e62>] ? acpi_idle_enter_bm+0x23b/0x271
> [<ffffffff813c7a08>] ? cpuidle_idle_call+0x98/0xf0
> [<ffffffff8100a104>] ? cpu_idle+0x94/0xd0
> [<ffffffff81468db6>] ? rest_init+0x66/0x70
> [<ffffffff816a082f>] ? start_kernel+0x2ef/0x340
> [<ffffffff8169fd54>] ? x86_64_start_reservations+0x84/0x90
> [<ffffffff8169fe32>] ? x86_64_start_kernel+0xd2/0x100
> Code: 00 eb 28 8b 83 d0 03 00 00 41 39 44 24 40 0f 89 00 01 00 00 41 0f b6 cd
> 41 bd 2f 00 00 00 83 e1 03 0f 84 fc 00 00 00 4d 8b 24 24 <49> 8b 04 24 4d 39 f4
> 0f 18 08 0f 84 d9 00 00 00 4c 3b a3 b8 01
> RIP [<ffffffff814203cc>] tcp_xmit_retransmit_queue+0x8c/0x290
> RSP <ffffc90000003a40>
> CR2: 0000000000000000
> ---[ end trace d97d99c9ae1d52cc ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> Pid: 0, comm: swapper Tainted: G D 2.6.31 #2
> Call Trace:
> <IRQ> [<ffffffff8103cab0>] panic+0xa0/0x170
> [<ffffffff8100bb13>] ? ret_from_intr+0x0/0xa
> [<ffffffff8103c74e>] ? print_oops_end_marker+0x1e/0x20
> [<ffffffff8100f38e>] oops_end+0x9e/0xb0
> [<ffffffff81025b9a>] no_context+0x15a/0x250
> [<ffffffff81025e2b>] __bad_area_nosemaphore+0xdb/0x1c0
> [<ffffffff813e89e9>] ? dev_hard_start_xmit+0x269/0x2f0
> [<ffffffff81025fae>] bad_area_nosemaphore+0xe/0x10
> [<ffffffff8102639f>] do_page_fault+0x17f/0x260
> [<ffffffff8147eadf>] page_fault+0x1f/0x30
> [<ffffffff814203cc>] ? tcp_xmit_retransmit_queue+0x8c/0x290
> [<ffffffff8141ae4a>] tcp_ack+0x170a/0x1dd0
> [<ffffffff8141c362>] tcp_rcv_state_process+0x122/0xab0
> [<ffffffff81422c6c>] tcp_v4_do_rcv+0xac/0x220
> [<ffffffff813fd02f>] ? nf_iterate+0x5f/0x90
> [<ffffffff81424b26>] tcp_v4_rcv+0x586/0x6b0
> [<ffffffff813fd0c5>] ? nf_hook_slow+0x65/0xf0
> [<ffffffff81406b70>] ? ip_local_deliver_finish+0x0/0x120
> [<ffffffff81406bcf>] ip_local_deliver_finish+0x5f/0x120
> [<ffffffff8140715b>] ip_local_deliver+0x3b/0x90
> [<ffffffff81406971>] ip_rcv_finish+0x141/0x340
> [<ffffffff8140701f>] ip_rcv+0x24f/0x350
> [<ffffffff813e7ced>] netif_receive_skb+0x20d/0x2f0
> [<ffffffff813e7e90>] napi_skb_finish+0x40/0x50
> [<ffffffff813e82f4>] napi_gro_receive+0x34/0x40
> [<ffffffff8133e0c8>] e1000_receive_skb+0x48/0x60
> [<ffffffff81342342>] e1000_clean_rx_irq+0xf2/0x330
> [<ffffffff813410a1>] e1000_clean+0x81/0x2a0
> [<ffffffff81054ce1>] ? ktime_get+0x11/0x50
> [<ffffffff813eaf1c>] net_rx_action+0x9c/0x130
> [<ffffffff81046940>] ? get_next_timer_interrupt+0x1d0/0x210
> [<ffffffff81041bd7>] __do_softirq+0xb7/0x160
> [<ffffffff8100c27c>] call_softirq+0x1c/0x30
> [<ffffffff8100e04d>] do_softirq+0x3d/0x80
> [<ffffffff81041b0b>] irq_exit+0x7b/0x90
> [<ffffffff8100d613>] do_IRQ+0x73/0xe0
> [<ffffffff8100bb13>] ret_from_intr+0x0/0xa
> <EOI> [<ffffffff81296e6c>] ? acpi_idle_enter_bm+0x245/0x271
> [<ffffffff81296e62>] ? acpi_idle_enter_bm+0x23b/0x271
> [<ffffffff813c7a08>] ? cpuidle_idle_call+0x98/0xf0
> [<ffffffff8100a104>] ? cpu_idle+0x94/0xd0
> [<ffffffff81468db6>] ? rest_init+0x66/0x70
> [<ffffffff816a082f>] ? start_kernel+0x2ef/0x340
> [<ffffffff8169fd54>] ? x86_64_start_reservations+0x84/0x90
> [<ffffffff8169fe32>] ? x86_64_start_kernel+0xd2/0x100
>
next prev parent reply other threads:[~2009-10-28 22:13 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-10-26 15:41 Fw: [Bug 14470] New: freez in TCP stack Stephen Hemminger
2009-10-28 22:13 ` Andrew Morton [this message]
2009-10-28 22:27 ` Denys Fedoryschenko
2009-10-29 5:35 ` Eric Dumazet
2009-10-29 5:59 ` Eric Dumazet
2009-10-29 6:02 ` David Miller
2009-10-29 8:00 ` David Miller
2009-11-26 21:54 ` Ilpo Järvinen
2009-11-26 23:37 ` David Miller
2009-11-27 6:17 ` Eric Dumazet
2009-12-02 23:10 ` David Miller
2009-12-03 6:24 ` David Miller
2010-03-18 21:04 ` Andrew Morton
2010-03-19 15:52 ` Ilpo Järvinen
2009-10-29 12:58 ` Fw: " Ilpo Järvinen
2009-10-29 14:08 ` Eric Dumazet
2009-10-30 20:18 ` Herbert Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20091028151313.ba4a4d23.akpm@linux-foundation.org \
--to=akpm@linux-foundation$(echo .)org \
--cc=bugzilla-daemon@bugzilla$(echo .)kernel.org \
--cc=kolo@albatani$(echo .)cz \
--cc=netdev@vger$(echo .)kernel.org \
--cc=shemminger@linux-foundation$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox