From: "Toke Høiland-Jørgensen" <toke@redhat•com>
To: Eric Dumazet <edumazet@google•com>
Cc: "David S . Miller" <davem@davemloft•net>,
Jakub Kicinski <kuba@kernel•org>, Paolo Abeni <pabeni@redhat•com>,
Simon Horman <horms@kernel•org>,
Jamal Hadi Salim <jhs@mojatatu•com>,
Cong Wang <xiyou.wangcong@gmail•com>,
Jiri Pirko <jiri@resnulli•us>,
Kuniyuki Iwashima <kuniyu@google•com>,
Willem de Bruijn <willemb@google•com>,
netdev@vger•kernel.org, eric.dumazet@gmail•com
Subject: Re: [PATCH v1 net-next 5/5] net: dev_queue_xmit() llist adoption
Date: Sun, 09 Nov 2025 17:33:11 +0100 [thread overview]
Message-ID: <871pm7np2w.fsf@toke.dk> (raw)
In-Reply-To: <CANn89iLqUtGkgXj0BgrXJD8ckqrHkMriapKpwHNcMP06V_fAGQ@mail.gmail.com>
Eric Dumazet <edumazet@google•com> writes:
> On Sun, Nov 9, 2025 at 2:09 AM Eric Dumazet <edumazet@google•com> wrote:
>>
>>
>> This might be something related to XDP, because I ran the following
>> test (IDPF, 32 TX queues)
>>
>> tc qd replace dev eth1 root cake
>> ./super_netperf 16 -H tjbp27 -t UDP_STREAM -l 1000 -- -m 64 -Nn &
>>
>> Before my series : ~360 Kpps
>> After my series : ~550 Kpps
>
> Or ... being faster uncovered an old qdisc bug.
>
> I mentioned the 'requeues' because I have seen this counter lately,
> and was wondering if this could
> be a driver bug.
>
> It seems the bug is in generic qdisc code: try_bulk_dequeue_skb() is
> trusting BQL, but can not see the driver might block before BQL.
>
> I am testing the following patch, it would be great if this solution
> works for you.
That does not seem to make any difference. I am not really seeing any
requeues either, just a whole bunch of drops:
qdisc cake 8001: dev ice0p1 root refcnt 37 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms raw overhead 0
Sent 9633155852 bytes 13658545 pkt (dropped 36165260, overlimits 0 requeues 42)
Tried with 16 netperf UDP_STREAMs instead of xdp-trafficgen, and with
that it's even worse (down to less than 100 PPS). A single netperf
instance gets me back to the ~600k PPS range, so definitely something to
do with contention.
The drops seem to come from mainly two places:
# dropwatch -l kas
Initializing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
1 drops at __netif_receive_skb_core.constprop.0+160 (0xffffffff87272de0) [software]
2132 drops at __dev_xmit_skb+3f5 (0xffffffff8726d475) [software]
1 drops at skb_queue_purge_reason+100 (0xffffffff8724e130) [software]
52901 drops at __dev_xmit_skb+3f5 (0xffffffff8726d475) [software]
153583 drops at __dev_xmit_skb+13c (0xffffffff8726d1bc) [software]
1 drops at __netif_receive_skb_core.constprop.0+160 (0xffffffff87272de0) [software]
93968 drops at __dev_xmit_skb+3f5 (0xffffffff8726d475) [software]
212982 drops at __dev_xmit_skb+13c (0xffffffff8726d1bc) [software]
239359 drops at __dev_xmit_skb+13c (0xffffffff8726d1bc) [software]
108219 drops at __dev_xmit_skb+3f5 (0xffffffff8726d475) [software]
191163 drops at __dev_xmit_skb+13c (0xffffffff8726d1bc) [software]
93300 drops at __dev_xmit_skb+3f5 (0xffffffff8726d475) [software]
131201 drops at __dev_xmit_skb+13c (0xffffffff8726d1bc) [software]
+13c corresponds to the defer_count check in your patch:
defer_count = atomic_long_inc_return(&q->defer_count);
if (unlikely(defer_count > q->limit)) {
kfree_skb_reason(skb, SKB_DROP_REASON_QDISC_DROP);
return NET_XMIT_DROP;
}
and +3f5 is the to_free drop at the end of the function:
unlock:
spin_unlock(root_lock);
if (unlikely(to_free))
kfree_skb_list_reason(to_free,
tcf_get_drop_reason(to_free));
Not sure why there's this difference between your setup or mine; some
.config or hardware difference related to the use of atomics? Any other
ideas?
-Toke
next prev parent reply other threads:[~2025-11-09 16:33 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-13 14:54 [PATCH v1 net-next 0/5] net: optimize TX throughput and efficiency Eric Dumazet
2025-10-13 14:54 ` [PATCH v1 net-next 1/5] net: add add indirect call wrapper in skb_release_head_state() Eric Dumazet
2025-10-13 14:54 ` [PATCH v1 net-next 2/5] net/sched: act_mirred: add loop detection Eric Dumazet
2025-10-13 14:54 ` [PATCH v1 net-next 3/5] Revert "net/sched: Fix mirred deadlock on device recursion" Eric Dumazet
2025-10-13 14:54 ` [PATCH v1 net-next 4/5] net: sched: claim one cache line in Qdisc Eric Dumazet
2025-10-13 14:54 ` [PATCH v1 net-next 5/5] net: dev_queue_xmit() llist adoption Eric Dumazet
2025-11-07 15:28 ` Toke Høiland-Jørgensen
2025-11-07 15:37 ` Eric Dumazet
2025-11-07 15:46 ` Eric Dumazet
2025-11-09 10:09 ` Eric Dumazet
2025-11-09 12:54 ` Eric Dumazet
2025-11-09 16:33 ` Toke Høiland-Jørgensen [this message]
2025-11-09 17:14 ` Eric Dumazet
2025-11-09 19:18 ` Jonas Köppeler
2025-11-09 19:28 ` Eric Dumazet
2025-11-09 20:18 ` Eric Dumazet
2025-11-09 20:29 ` Eric Dumazet
2025-11-10 11:31 ` Toke Høiland-Jørgensen
2025-11-10 13:26 ` Eric Dumazet
2025-11-10 14:49 ` Toke Høiland-Jørgensen
2025-11-10 17:34 ` Eric Dumazet
2025-11-11 13:44 ` Jonas Köppeler
2025-11-11 16:42 ` Toke Høiland-Jørgensen
2025-10-13 16:23 ` [PATCH v1 net-next 0/5] net: optimize TX throughput and efficiency Toke Høiland-Jørgensen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=871pm7np2w.fsf@toke.dk \
--to=toke@redhat$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=edumazet@google$(echo .)com \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=horms@kernel$(echo .)org \
--cc=jhs@mojatatu$(echo .)com \
--cc=jiri@resnulli$(echo .)us \
--cc=kuba@kernel$(echo .)org \
--cc=kuniyu@google$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=pabeni@redhat$(echo .)com \
--cc=willemb@google$(echo .)com \
--cc=xiyou.wangcong@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox