From: Andrew Morton <akpm@linux-foundation•org>
To: Eric Dumazet <eric.dumazet@gmail•com>
Cc: David Miller <davem@davemloft•net>,
netdev <netdev@vger•kernel.org>,
Michael Chan <mchan@broadcom•com>,
Eilon Greenstein <eilong@broadcom•com>,
Christoph Hellwig <hch@lst•de>,
Christoph Lameter <cl@linux-foundation•org>
Subject: Re: [PATCH net-next] net: allocate skbs on local node
Date: Mon, 11 Oct 2010 23:03:22 -0700 [thread overview]
Message-ID: <20101011230322.f0f6dd47.akpm@linux-foundation.org> (raw)
In-Reply-To: <1286859925.30423.184.camel@edumazet-laptop>
On Tue, 12 Oct 2010 07:05:25 +0200 Eric Dumazet <eric.dumazet@gmail•com> wrote:
> Le mardi 12 octobre 2010 __ 01:22 +0200, Eric Dumazet a __crit :
> > Le mardi 12 octobre 2010 __ 01:03 +0200, Eric Dumazet a __crit :
> > >
> > > For multi queue devices, it makes more sense to allocate skb on local
> > > node of the cpu handling RX interrupts. This allow each cpu to
> > > manipulate its own slub/slab queues/structures without doing expensive
> > > cross-node business.
> > >
> > > For non multi queue devices, IRQ affinity should be set so that a cpu
> > > close to the device services interrupts. Even if not set, using
> > > dev_alloc_skb() is faster.
> > >
> > > Signed-off-by: Eric Dumazet <eric.dumazet@gmail•com>
> >
> > Or maybe revert :
> >
> > commit b30973f877fea1a3fb84e05599890fcc082a88e5
> > Author: Christoph Hellwig <hch@lst•de>
> > Date: Wed Dec 6 20:32:36 2006 -0800
> >
> > [PATCH] node-aware skb allocation
> >
> > Node-aware allocation of skbs for the receive path.
> >
> > Details:
> >
> > - __alloc_skb gets a new node argument and cals the node-aware
> > slab functions with it.
> > - netdev_alloc_skb passed the node number it gets from dev_to_node
> > to it, everyone else passes -1 (any node)
> >
> > Signed-off-by: Christoph Hellwig <hch@lst•de>
> > Cc: Christoph Lameter <clameter@engr•sgi.com>
> > Cc: "David S. Miller" <davem@davemloft•net>
> > Signed-off-by: Andrew Morton <akpm@osdl•org>
> >
> >
> > Apparently, only Christoph and Andrew signed it.
> >
> >
>
> [PATCH net-next] net: allocate skbs on local node
>
> commit b30973f877 (node-aware skb allocation) spread a wrong habit of
> allocating net drivers skbs on a given memory node : The one closest to
> the NIC hardware. This is wrong because as soon as we try to scale
> network stack, we need to use many cpus to handle traffic and hit
> slub/slab management on cross-node allocations/frees when these cpus
> have to alloc/free skbs bound to a central node.
>
> skb allocated in RX path are ephemeral, they have a very short
> lifetime : Extra cost to maintain NUMA affinity is too expensive. What
> appeared as a nice idea four years ago is in fact a bad one.
>
> In 2010, NIC hardwares are multiqueue, or we use RPS to spread the load,
> and two 10Gb NIC might deliver more than 28 million packets per second,
> needing all the available cpus.
>
> Cost of cross-node handling in network and vm stacks outperforms the
> small benefit hardware had when doing its DMA transfert in its 'local'
> memory node at RX time. Even trying to differentiate the two allocations
> done for one skb (the sk_buff on local node, the data part on NIC
> hardware node) is not enough to bring good performance.
>
This is all conspicuously hand-wavy and unquantified. (IOW: prove it!)
The mooted effects should be tested for on both slab and slub, I
suggest. They're pretty different beasts.
next prev parent reply other threads:[~2010-10-12 6:02 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-10-11 23:03 [PATCH net-next] bnx2x: dont use netdev_alloc_skb() Eric Dumazet
2010-10-11 23:22 ` Eric Dumazet
2010-10-12 5:03 ` Tom Herbert
2010-10-12 5:16 ` Eric Dumazet
2010-10-12 9:12 ` Vladislav Zolotarov
2010-10-14 17:39 ` David Miller
2010-10-14 18:17 ` Eilon Greenstein
2010-10-14 18:20 ` Eric Dumazet
2010-10-14 18:25 ` David Miller
2010-10-14 18:17 ` Tom Herbert
2010-10-12 5:05 ` [PATCH net-next] net: allocate skbs on local node Eric Dumazet
2010-10-12 5:35 ` Tom Herbert
2010-10-12 6:03 ` Andrew Morton [this message]
2010-10-12 6:58 ` Eric Dumazet
2010-10-12 7:24 ` Andrew Morton
2010-10-12 7:49 ` Eric Dumazet
2010-10-12 7:58 ` Andrew Morton
2010-10-12 11:08 ` Pekka Enberg
2010-10-12 12:50 ` Christoph Lameter
2010-10-12 19:43 ` David Rientjes
2010-10-13 6:17 ` Pekka Enberg
2010-10-13 6:31 ` David Rientjes
2010-10-13 6:36 ` Pekka Enberg
2010-10-13 16:00 ` Christoph Lameter
2010-10-13 20:48 ` David Rientjes
2010-10-13 21:43 ` Christoph Lameter
2010-10-13 22:41 ` David Rientjes
2010-10-14 6:22 ` Pekka Enberg
2010-10-14 7:23 ` David Rientjes
2010-10-15 14:23 ` Christoph Lameter
2010-10-14 15:31 ` Tom Herbert
2010-10-14 16:05 ` Eric Dumazet
2010-10-15 16:57 ` Christoph Lameter
2010-10-14 19:27 ` Andrew Morton
2010-10-14 19:59 ` Eric Dumazet
2010-10-16 18:54 ` David Miller
2010-10-12 16:07 ` [BUG net-next] bnx2x: all traffic comes to RX queue 0 Eric Dumazet
2010-10-12 16:20 ` Dmitry Kravkov
2010-10-12 18:11 ` Eric Dumazet
2010-10-12 18:18 ` Vladislav Zolotarov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20101011230322.f0f6dd47.akpm@linux-foundation.org \
--to=akpm@linux-foundation$(echo .)org \
--cc=cl@linux-foundation$(echo .)org \
--cc=davem@davemloft$(echo .)net \
--cc=eilong@broadcom$(echo .)com \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=hch@lst$(echo .)de \
--cc=mchan@broadcom$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox