From: David Ahern <dsahern@kernel•org>
To: Ivan Babrou <ivan@cloudflare•com>
Cc: Linux Kernel Network Developers <netdev@vger•kernel.org>,
kernel-team <kernel-team@cloudflare•com>,
Eric Dumazet <edumazet@google•com>,
"David S. Miller" <davem@davemloft•net>,
Paolo Abeni <pabeni@redhat•com>,
Steven Rostedt <rostedt@goodmis•org>,
Masami Hiramatsu <mhiramat@kernel•org>,
Jakub Kicinski <kuba@kernel•org>
Subject: Re: Stacks leading into skb:kfree_skb
Date: Tue, 18 Jul 2023 16:43:04 -0600 [thread overview]
Message-ID: <07e598bb-aebf-b169-425f-a6e8a84016d4@kernel.org> (raw)
In-Reply-To: <CABWYdi38H3umTEqTPbt8DftF2HXZ7ba6+jNphJdvubeh6PLP8w@mail.gmail.com>
On 7/18/23 4:33 PM, Ivan Babrou wrote:
> On Fri, Jul 14, 2023 at 5:54 PM David Ahern <dsahern@kernel•org> wrote:
>>
>> On 7/14/23 4:13 PM, Ivan Babrou wrote:
>>> As requested by Jakub Kicinski and David Ahern here:
>>>
>>> * https://lore.kernel.org/netdev/20230713201427.2c50fc7b@kernel.org/
>>>
>>> I made some aggregations for the stacks we see leading into
>>> skb:kfree_skb endpoint. There's a lot of data that is not easily
>>> digestible, so I lightly massaged the data and added flamegraphs in
>>> addition to raw stack counts. Here's the gist link:
>>>
>>> * https://gist.github.com/bobrik/0e57671c732d9b13ac49fed85a2b2290
>>
>> I see a lot of packet_rcv as the tip before kfree_skb. How many packet
>> sockets do you have running on that box? Can you accumulate the total
>> packet_rcv -> kfree_skb_reasons into 1 count -- regardless of remaining
>> stacktrace?
>
> Yan will respond regarding the packet sockets later in the day, he
> knows this stuff better than I do.
>
> In the meantime, here are the aggregations you requested:
>
> * Normal: https://gist.githubusercontent.com/bobrik/0e57671c732d9b13ac49fed85a2b2290/raw/ae8aa1bc3b22fad6cf541afeb51aa8049d122d02/flamegraph.normal.packet_rcv.aggregated.svg
> * Spike: https://gist.githubusercontent.com/bobrik/0e57671c732d9b13ac49fed85a2b2290/raw/ae8aa1bc3b22fad6cf541afeb51aa8049d122d02/flamegraph.spike.packet_rcv.aggregated.svg
For the spike, 97% are drops in packet_rcv. Each raw packet socket
causes every packet to be cloned which makes an N-factor on the number
of skbs to be freed. If this is tcpdump or lldp with a filter that would
be what Jakub mentioned in his response.
>
> I just realized that Github links make flamegraphs non-interactive. If
> you download them and open a local copy, they should work better:
Firefox shows the graphs just fine.
>
> * Expand to your screen width
> * Working search with highlights
> * Tooltips with counts and percentages
> * Working zoom
next prev parent reply other threads:[~2023-07-18 22:43 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-14 22:13 Stacks leading into skb:kfree_skb Ivan Babrou
2023-07-15 0:54 ` David Ahern
2023-07-18 22:33 ` Ivan Babrou
2023-07-18 22:43 ` David Ahern [this message]
2023-07-18 22:36 ` Jakub Kicinski
2023-07-19 3:10 ` Yan Zhai
2023-07-19 3:50 ` David Ahern
2023-07-19 13:18 ` Willem de Bruijn
2023-07-19 13:26 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=07e598bb-aebf-b169-425f-a6e8a84016d4@kernel.org \
--to=dsahern@kernel$(echo .)org \
--cc=davem@davemloft$(echo .)net \
--cc=edumazet@google$(echo .)com \
--cc=ivan@cloudflare$(echo .)com \
--cc=kernel-team@cloudflare$(echo .)com \
--cc=kuba@kernel$(echo .)org \
--cc=mhiramat@kernel$(echo .)org \
--cc=netdev@vger$(echo .)kernel.org \
--cc=pabeni@redhat$(echo .)com \
--cc=rostedt@goodmis$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox