From: Krister Johansen <kjlx@templeofstupid•com>
To: Cong Wang <xiyou.wangcong@gmail•com>
Cc: Krister Johansen <kjlx@templeofstupid•com>,
David Miller <davem@davemloft•net>,
Eric Dumazet <eric.dumazet@gmail•com>,
Linux Kernel Network Developers <netdev@vger•kernel.org>,
Kaiwen Xu <kaiwen.xu@hulu•com>
Subject: Re: [PATCH net] Fix an intermittent pr_emerg warning about lo becoming free.
Date: Fri, 9 Jun 2017 11:43:03 -0700 [thread overview]
Message-ID: <20170609184303.GC2766@templeofstupid.com> (raw)
In-Reply-To: <CAM_iQpUaenZa0o-eyt91sAE6SF+qk6Thh5vpKqPLXSc6xpjPqg@mail.gmail.com>
On Fri, Jun 09, 2017 at 11:18:44AM -0700, Cong Wang wrote:
> On Thu, Jun 8, 2017 at 1:12 PM, Krister Johansen
> <kjlx@templeofstupid•com> wrote:
> > The way this works is that if there's still a reference on the dst entry
> > at the time we try to free it, it gets placed in the gc list by
> > __dst_free and the dst_destroy() call is invoked by the gc task once the
> > refcount is 0. If the gc task processes a 10th or less of its entries
> > on a single pass, it inreases the amount of time it waits between gc
> > intervals.
> >
> > Looking at the gc_task intervals, they started at 663ms when we invoked
> > __dst_free(). After that, they increased to 1663, 3136, 5567, 8191,
> > 10751, and 14848. The release that set the refcnt to 0 on our dst entry
> > occurred after the gc_task was enqueued for 14 second interval so we had
> > to wait longer than the warning time in wait_allrefs in order for the
> > dst entry to get free'd and the hold on 'lo' to be released.
> >
>
> I am glad to see you don't have a dst leak here.
>
> But from my experience of a similar bug (refcnt wait on lo), this goes
> infinitely rather than just 14sec, so it looked more like a real leak than
> just a gc delay. So in your case, this annoying warning eventually
> disappears, right?
That's correct. The problem occurs intermittently, and the warnings are
less frequent than the interval in netdev_wait_allrefs(). At least when
I observed it, it tended to conincide with our controlplane canary
issuing an API call that lead to a network namespace teardown on the
dataplane.
Sometimes, the message would look like this:
unregister_netdevice: waiting for lo to become free. Usage count = 0
The dst entries were getting released, it's just that often our dst
cache gc interval was longer than the warning interval in wait_allrefs.
The other concern was that because the wait_allrefs happens in the
netdev_todo path, a long gc interval can cause the rtnl_lock hold times
to be much longer than necessary if this bug is encountered.
-K
next prev parent reply other threads:[~2017-06-09 18:43 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-08 20:12 [PATCH net] Fix an intermittent pr_emerg warning about lo becoming free Krister Johansen
2017-06-08 21:06 ` Eric Dumazet
2017-06-08 21:41 ` Wei Wang
2017-06-09 16:28 ` David Miller
2017-06-09 18:18 ` Cong Wang
2017-06-09 18:43 ` Krister Johansen [this message]
2017-06-09 21:48 ` Cong Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170609184303.GC2766@templeofstupid.com \
--to=kjlx@templeofstupid$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=kaiwen.xu@hulu$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=xiyou.wangcong@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox