From: "Holger Hoffstätte" <holger.hoffstaette@googlemail•com>
To: Wolfgang Walter <linux@stwm•de>, Andre Tomt <andre@tomt•net>
Cc: Julian Anastasov <ja@ssi•bg>,
Greg Kroah-Hartman <gregkh@linuxfoundation•org>,
linux-kernel@vger•kernel.org,
"David S. Miller" <davem@davemloft•net>,
stable@vger•kernel.org,
"Eric W. Biederman" <ebiederm@xmission•com>,
Stephen Hemminger <stephen@networkplumber•org>,
Linux Netdev List <netdev@vger•kernel.org>
Subject: Re: kernel 4.1.9: networking hangs with rcu_preempt self-detected stall, 4.1.8 works; was: Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog
Date: Thu, 1 Oct 2015 13:32:35 +0200 [thread overview]
Message-ID: <560D19D3.70305@googlemail.com> (raw)
In-Reply-To: <3770693.fDE1nljnWm@stwm.de>
On 10/01/15 13:29, Wolfgang Walter wrote:
> Am Dienstag, 29. September 2015, 12:48:43 schrieb Andre Tomt:
>> On 29. sep. 2015 12:21, Andre Tomt wrote:
>>> Meanwhile I'll revert both the mentioned net patches and see how it goes.
>>
>> So that blew up as well, meaning it's not any of these two patches:
>> [PATCH 4.1 124/159] net: do not process device backlog during unregistration
>> [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog
>>
>> I'll be offline for a half+ day, I'll look into bisecting when back if
>> nobody has figured it out by then.
>> --
>> To unsubscribe from this list: send the line "unsubscribe stable" in
>> the body of a message to majordomo@vger•kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> We see these rcu hangs with 4.1.9 on one of our routers, too. 4.1.8 runs fine.
>
> The output I got the last time was:
>
> [ 6488.174578] igb 000:06:00.1 eth3: Reset adapter
> [ 6497.350183] INFO: rcu_preempt self-detected stall on CPU { 3} (t=6301 jiffies g=383330 c=383329 q=1323)
> [ 6497.350229] rcu_preempt kthread starved for 6007 jiffies!
> [ 6560.311093] INFO: rcu_preempt self-detected stall on CPU { 3} (t=25205 jiffies g=383330 c=383329 q=4479)
> [ 6560.311140] rcu_preempt kthread starved for 24911 jiffies!
> [ 6623.272005] INFO: rcu_preempt self-detected stall on CPU { 3} (t=44109 jiffies g=383330 c=383329 q=7107)
> [ 6623.272049] rcu_preempt kthread starved for 43815 jiffies!
> [ 6633.053892] igb 000:06:00.0 eth2: Reset adapter
> [ 6633.053892] rcu_preempt kthread starved for 62719 jiffies!
> [ 6486.232914] INFO: rcu_preempt self-detected stall on CPU { 3} (t=63013 jiffies g=383330 c=383329 q=8487)
> [ 6486.233204] rcu_preempt kthread starved for 6007 jiffies!
>
>
> All other hangs basically were the same, the cpu varies though.
>
> After that the router completely hangs: networking stops working and we need to restart it.
Please see the following thread for a suggested solution:
http://www.spinics.net/lists/kernel/msg2087337.html
Please followup there as well.
thanks
Holger
prev parent reply other threads:[~2015-10-01 11:32 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20150926205313.363686083@linuxfoundation.org>
[not found] ` <560A661C.7000406@tomt.net>
[not found] ` <560A6C8B.7080105@tomt.net>
2015-10-01 11:29 ` kernel 4.1.9: networking hangs with rcu_preempt self-detected stall, 4.1.8 works; was: Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process_backlog Wolfgang Walter
2015-10-01 11:32 ` Holger Hoffstätte [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=560D19D3.70305@googlemail.com \
--to=holger.hoffstaette@googlemail$(echo .)com \
--cc=andre@tomt$(echo .)net \
--cc=davem@davemloft$(echo .)net \
--cc=ebiederm@xmission$(echo .)com \
--cc=gregkh@linuxfoundation$(echo .)org \
--cc=ja@ssi$(echo .)bg \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linux@stwm$(echo .)de \
--cc=netdev@vger$(echo .)kernel.org \
--cc=stable@vger$(echo .)kernel.org \
--cc=stephen@networkplumber$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox