From: James Chapman <jchapman@katalix•com>
To: Jarek Poplawski <jarkao2@gmail•com>
Cc: David Miller <davem@davemloft•net>,
Paul Mackerras <paulus@samba•org>,
netdev@vger•kernel.org
Subject: Re: [PATCH][PPPOL2TP]: Fix SMP oops in pppol2tp driver
Date: Wed, 20 Feb 2008 16:02:52 +0000 [thread overview]
Message-ID: <47BC4F2C.4000802@katalix.com> (raw)
In-Reply-To: <20080219230640.GA2755@ami.dom.local>
Jarek Poplawski wrote:
> On Mon, Feb 18, 2008 at 10:09:24PM +0000, James Chapman wrote:
> ...
>> Unfortunately the ISP's syslog stops. But I've been able to borrow
>> two Quad Xeon boxes and have reproduced the problem.
>>
>> Here's a new version of the patch. The patch avoids disabling irqs
>> and fixes the sk_dst_get() usage that DaveM mentioned. But even with
>> this patch, lockdep still complains if hundreds of ppp sessions are
>> inserted into a tunnel as rapidly as possible (lockdep trace is below).
>> I can stop these errors by wrapping the call to ppp_input() in
>> pppol2tp_recv_dequeue_skb() with local_irq_save/restore. What is a
>> better fix?
>
> I send here my proposal: it's intended for testing and to check one of
> possible solutions here. IMHO your lockdep reports show there is no
> use to change anything around sk_dst_lock: it would need the global
> change of this lock to fix this problem. So the fix should be done
> around pch->upl lock and this means changing ppp_generic.
Hmm, I need to study the lockdep report again. It seems I'm misreading
the lockdep output. :(
> In the patch below I've used trylock in places which seem to allow
> for skipping some things (while config is changed only) or simply
> don't need this lock because there is no ppp struct. This could be
> modified to add some waiting loop if necessary. Another option is to
> change the write side of this lock: it looks like more vulnerable if
> something missed because there are more locks involved, but probably
> should be enough to solve this problem too.
>
> I think pppol2tp need to be first checked only with hlist_lock bh
> patch, unless there were some lockdep reports on these other locks
> too. (BTW, I added ppp maintainer to CC - I hope we get Paul's opinion
> on this.)
I tried your ppp_generic patch with only the hlist_lock bh patch in
pppol2tp and it seems to fix the ppp create/delete issue. However, when
I added much more traffic into the test (flood pings over ppp interfaces
while repeatedly creating/deleting the L2TP (PPP) sessions) I get a soft
lockup detected in pppol2tp_xmit() after anything between 1 minute and
an hour. :( I'm investigating that now.
Thanks for your help!
> (testing patch #1)
> ---
>
> drivers/net/ppp_generic.c | 33 +++++++++++++++++++++++----------
> 1 files changed, 23 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/net/ppp_generic.c b/drivers/net/ppp_generic.c
> index 4dc5b4b..5cbc534 100644
> --- a/drivers/net/ppp_generic.c
> +++ b/drivers/net/ppp_generic.c
> @@ -1473,7 +1473,7 @@ void
> ppp_input(struct ppp_channel *chan, struct sk_buff *skb)
> {
> struct channel *pch = chan->ppp;
> - int proto;
> + int proto, locked;
>
> if (!pch || skb->len == 0) {
> kfree_skb(skb);
> @@ -1481,8 +1481,13 @@ ppp_input(struct ppp_channel *chan, struct sk_buff *skb)
> }
>
> proto = PPP_PROTO(skb);
> - read_lock_bh(&pch->upl);
> - if (!pch->ppp || proto >= 0xc000 || proto == PPP_CCPFRAG) {
> + /*
> + * We use trylock to avoid dependency between soft-irq-safe upl lock
> + * and soft-irq-unsafe sk_dst_lock.
> + */
> + local_bh_disable();
> + locked = read_trylock(&pch->upl);
> + if (!locked || !pch->ppp || proto >= 0xc000 || proto == PPP_CCPFRAG) {
> /* put it on the channel queue */
> skb_queue_tail(&pch->file.rq, skb);
> /* drop old frames if queue too long */
> @@ -1493,7 +1498,10 @@ ppp_input(struct ppp_channel *chan, struct sk_buff *skb)
> } else {
> ppp_do_recv(pch->ppp, skb, pch);
> }
> - read_unlock_bh(&pch->upl);
> +
> + if (locked)
> + read_unlock(&pch->upl);
> + local_bh_enable();
> }
>
> /* Put a 0-length skb in the receive queue as an error indication */
> @@ -1506,16 +1514,18 @@ ppp_input_error(struct ppp_channel *chan, int code)
> if (!pch)
> return;
>
> - read_lock_bh(&pch->upl);
> - if (pch->ppp) {
> + /* a trylock comment in ppp_input() */
> + local_bh_disable();
> + if (read_trylock(&pch->upl) && pch->ppp) {
> skb = alloc_skb(0, GFP_ATOMIC);
> if (skb) {
> skb->len = 0; /* probably unnecessary */
> skb->cb[0] = code;
> ppp_do_recv(pch->ppp, skb, pch);
> }
> + read_unlock(&pch->upl);
> }
> - read_unlock_bh(&pch->upl);
> + local_bh_enable();
> }
>
> /*
> @@ -2044,10 +2054,13 @@ int ppp_unit_number(struct ppp_channel *chan)
> int unit = -1;
>
> if (pch) {
> - read_lock_bh(&pch->upl);
> - if (pch->ppp)
> + /* a trylock comment in ppp_input() */
> + local_bh_disable();
> + if (read_trylock(&pch->upl) && pch->ppp) {
> unit = pch->ppp->file.index;
> - read_unlock_bh(&pch->upl);
> + read_unlock(&pch->upl);
> + }
> + local_bh_enable();
> }
> return unit;
> }
> --
--
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development
next prev parent reply other threads:[~2008-02-20 16:03 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-02-11 9:22 [PATCH][PPPOL2TP]: Fix SMP oops in pppol2tp driver James Chapman
2008-02-11 18:57 ` Jarek Poplawski
2008-02-11 22:19 ` James Chapman
2008-02-11 22:49 ` Jarek Poplawski
2008-02-11 22:55 ` Jarek Poplawski
2008-02-11 23:42 ` James Chapman
2008-02-12 10:42 ` Jarek Poplawski
2008-02-11 23:41 ` James Chapman
2008-02-12 5:30 ` David Miller
2008-02-12 10:58 ` James Chapman
2008-02-12 13:24 ` Jarek Poplawski
2008-02-13 6:00 ` David Miller
2008-02-13 7:29 ` Jarek Poplawski
2008-02-14 13:00 ` Jarek Poplawski
2008-02-18 22:09 ` James Chapman
2008-02-18 23:01 ` Jarek Poplawski
2008-02-19 9:09 ` James Chapman
2008-02-19 4:29 ` David Miller
2008-02-19 9:03 ` James Chapman
2008-02-19 10:30 ` Jarek Poplawski
2008-02-19 10:36 ` Jarek Poplawski
2008-02-19 14:37 ` James Chapman
2008-02-19 23:06 ` Jarek Poplawski
2008-02-19 23:28 ` Jarek Poplawski
2008-02-20 16:02 ` James Chapman [this message]
2008-02-20 18:38 ` Jarek Poplawski
2008-02-20 22:37 ` James Chapman
2008-02-21 8:59 ` Jarek Poplawski
2008-02-21 9:53 ` James Chapman
2008-02-21 12:08 ` Jarek Poplawski
2008-02-21 17:09 ` Jarek Poplawski
2008-02-25 12:19 ` James Chapman
2008-02-25 13:05 ` Jarek Poplawski
2008-02-25 13:39 ` Jarek Poplawski
2008-02-25 14:02 ` Jarek Poplawski
2008-02-25 21:58 ` Jarek Poplawski
2008-02-26 12:14 ` James Chapman
2008-02-26 13:03 ` Jarek Poplawski
2008-02-26 13:18 ` Jarek Poplawski
2008-02-26 20:00 ` Jarek Poplawski
2008-03-02 20:29 ` James Chapman
2008-03-03 8:22 ` Jarek Poplawski
2008-03-03 9:35 ` Jarek Poplawski
2008-02-27 10:54 ` [PATCH][PPPOL2TP] add missing sock_put() in pppol2tp_recv_dequeue() Jarek Poplawski
2008-03-02 20:31 ` James Chapman
2008-03-04 4:49 ` David Miller
2008-02-27 11:48 ` [PATCH][PPPOL2TP] add missing sock_put() in pppol2tp_tunnel_closeall() Jarek Poplawski
2008-03-02 20:32 ` James Chapman
2008-03-04 4:49 ` David Miller
2008-02-22 14:16 ` [PATCH][NET] sock.c: sk_dst_lock lockdep keys and names per af_family Jarek Poplawski
2008-02-12 7:19 ` [PATCH][PPPOL2TP]: Fix SMP oops in pppol2tp driver Jarek Poplawski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47BC4F2C.4000802@katalix.com \
--to=jchapman@katalix$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=jarkao2@gmail$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=paulus@samba$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox