From: Eric Dumazet <dada1@cosmosbay•com>
To: "David S. Miller" <davem@davemloft•net>
Cc: Linux Netdev List <netdev@vger•kernel.org>
Subject: [RFC] net: release dst entry in dev_queue_xmit()
Date: Fri, 20 Mar 2009 12:40:22 +0100 [thread overview]
Message-ID: <49C380A6.4000904@cosmosbay.com> (raw)
One point of contention in high network loads is the dst_release() performed
when a transmited skb is freed. This is because NIC tx completion calls skb free
long after original call to dev_queue_xmit(skb).
CPU cache is cold and the atomic op in dst_release() stalls. On SMP, this is
quite visible if one CPU is 100% handling softirqs for a network device,
since dst_clone() is done by other cpus, involving cache line ping pongs.
I believe we can release dst in dev_queue_xmit(), while cpu cache is hot, since
caller of dev_queue_xmit() had to hold a reference on dst right before. This reduce
work to be done by softirq handler, and decrease cache misses.
I also believe only pktgen can call dev_queue_xmit() with skb which have
a skb->users != 1. But pkthen skbs have a NULL dst entry.
I added a WARN_ON_ONCE() to catch other cases, and not release skb->dst
if skb->users != 1
Signed-off-by: Eric Dumazet <dada1@cosmosbay•com>
diff --git a/net/core/dev.c b/net/core/dev.c
index f112970..9e0fd01 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1852,6 +1852,20 @@ gso:
if (q->enqueue) {
spinlock_t *root_lock = qdisc_lock(q);
+ /*
+ * Release dst while its refcnt is hot in CPU cache, instead
+ * of waiting NIC tx completion
+ */
+ if (likely(skb->dst)) {
+ if (!WARN_ON_ONCE(atomic_read(&skb->users) != 1)) {
+ int newrefcnt;
+
+ smp_mb__before_atomic_dec();
+ newrefcnt = atomic_dec_return(&skb->dst->__refcnt);
+ WARN_ON(newrefcnt < 0);
+ skb->dst = NULL;
+ }
+ }
spin_lock(root_lock);
if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) {
next reply other threads:[~2009-03-20 12:03 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-20 11:40 Eric Dumazet [this message]
2009-03-20 14:10 ` [RFC] net: release dst entry in dev_queue_xmit() Neil Horman
2009-03-25 6:43 ` David Miller
2009-03-25 7:13 ` Eric Dumazet
2009-03-25 7:17 ` David Miller
2009-03-25 18:22 ` Jarek Poplawski
2009-03-25 18:41 ` Eric Dumazet
2009-03-25 19:18 ` Jarek Poplawski
2009-03-25 19:40 ` Eric Dumazet
2009-03-25 19:54 ` Jarek Poplawski
2009-03-25 20:28 ` Eric Dumazet
2009-03-25 21:12 ` Jarek Poplawski
2009-03-25 21:20 ` Patrick McHardy
2009-05-12 8:12 ` [PATCH] net: release dst entry in dev_hard_start_xmit() Eric Dumazet
2009-05-12 8:21 ` Eric Dumazet
2009-05-12 19:26 ` [PATCH, v2] " Eric Dumazet
2009-05-19 5:19 ` David Miller
2009-05-19 5:44 ` Eric Dumazet
2009-05-19 19:44 ` Eric Dumazet
2009-05-19 21:09 ` Jarek Poplawski
2009-05-19 21:21 ` Eric Dumazet
2009-05-19 21:24 ` David Miller
2009-05-12 19:27 ` [PATCH] " Jarek Poplawski
2009-05-12 19:44 ` Eric Dumazet
2009-05-12 20:05 ` Jarek Poplawski
2009-05-12 20:24 ` Jarek Poplawski
2009-05-12 20:52 ` Eric Dumazet
2009-05-12 20:59 ` Jarek Poplawski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=49C380A6.4000904@cosmosbay.com \
--to=dada1@cosmosbay$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=netdev@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox