From: Bill Fink <billfink@mindspring•com>
To: Tom Herbert <therbert@google•com>
Cc: davem@davemloft•net, netdev@vger•kernel.org, eric.dumazet@gmail•com
Subject: Re: [PATCH] xps-mq: Transmit Packet Steering for multiqueue
Date: Tue, 24 Aug 2010 00:31:43 -0400 [thread overview]
Message-ID: <20100824003143.243eb046.billfink@mindspring.com> (raw)
In-Reply-To: <alpine.DEB.1.00.1008222233530.31382@pokey.mtv.corp.google.com>
On Sun, 22 Aug 2010, Tom Herbert wrote:
> This patch implements transmit packet steering (XPS) for multiqueue
> devices. XPS selects a transmit queue during packet transmission based
> on configuration. This is done by mapping the CPU transmitting the
> packet to a queue. This is the transmit side analogue to RPS-- where
> RPS is selecting a CPU based on receive queue, XPS selects a queue
> based on the CPU (previously there was an XPS patch from Eric
> Dumazet, but that might more appropriately be called transmit completion
> steering).
>
> Each transmit queue can be associated with a number of CPUs which will
> used the queue to send packets. This is configured as a CPU mask on a
> per queue basis in:
>
> /sys/class/net/eth<n>/queues/tx-<n>/xps_cpus
>
> The mappings are stored per device in an inverted data structure that
> maps CPUs to queues. In the netdevice structure this is an array of
> num_possible_cpu structures where each array entry contains a bit map
> of queues which that CPU can use.
>
> We also allow the mapping of a socket to queue to be modified, for
> instance if a thread is scheduled on a different CPU the desired queue
> for transmitting packets would likely change. To maintain in order
> packet transmission a flag (ooo_okay) has been added to the sk_buf
> structure. If a transport layer sets this flag on a packet, the
> transmit queue can be changed for this socket. Presumably, the
> transport would set this is there was no possbility of creating ooo
> packets (for instance there are no packets in flight for the socket).
> This patch includes the modification in TCP output for setting this
> flag.
>
> The benefits of XPS are improved locality in the per queue data
> strutures. Also, transmit completions are more likely to be done
> nearer to the sending thread so this should promote locality back
> to the socket (e.g. UDP). The benefits of XPS are dependent on
> cache hierarchy, application load, and other factors. XPS would
> nominally be configured so that a queue would only be shared by CPUs
> which are sharing a cache, the degenerative configuration woud be that
> each CPU has it's own queue.
>
> Below are some benchmark results which show the potential benfit of
> this patch. The netperf test has 500 instances of netperf TCP_RR test
> with 1 byte req. and resp.
>
> bnx2x on 16 core AMD
> XPS (16 queues, 1 TX queue per CPU) 1015K at 99% CPU
> No XPS (16 queues) 1127K at 98% CPU
I don't grok your performance numbers. What do the 1015K and 1127K
numbers represent? I was originally guessing that they were basically
transactions per second, but that would seem to imply that the No XPS
case was better. Please clarify.
-Thanks
-Bill
next prev parent reply other threads:[~2010-08-24 4:31 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-23 5:39 [PATCH] xps-mq: Transmit Packet Steering for multiqueue Tom Herbert
2010-08-23 17:09 ` Ben Hutchings
[not found] ` <AANLkTinST5zaS0NtBjrzyLbsg=w_EVsHE3DCDcrmQNc6@mail.gmail.com>
2010-08-23 17:50 ` Ben Hutchings
2010-08-23 17:59 ` Stephen Hemminger
2010-09-01 15:41 ` Tom Herbert
2010-09-01 15:54 ` Eric Dumazet
2010-09-01 16:24 ` Tom Herbert
2010-09-02 1:32 ` David Miller
2010-09-02 1:48 ` Stephen Hemminger
2010-09-02 16:00 ` Loke, Chetan
2010-09-02 19:52 ` Tom Herbert
2010-09-02 23:17 ` Loke, Chetan
2010-09-02 1:56 ` Stephen Hemminger
2010-09-02 6:41 ` Greg Lindahl
2010-09-02 16:18 ` Loke, Chetan
2010-09-02 15:55 ` Loke, Chetan
2010-09-16 21:52 ` Ben Hutchings
2010-09-19 17:24 ` Michael S. Tsirkin
2010-09-20 12:44 ` [RFC][PATCH 1/3] IRQ: Add irq_get_numa_node() Ben Hutchings
2010-09-20 13:04 ` Eric Dumazet
2010-09-20 12:45 ` [RFC][PATCH 2/3] ethtool: NUMA affinity control Ben Hutchings
2010-09-20 12:48 ` [RFC][PATCH 3/3] sfc: Add support for " Ben Hutchings
2011-02-21 18:19 ` [PATCH] xps-mq: Transmit Packet Steering for multiqueue Ben Hutchings
2011-02-21 19:31 ` Jeremy Eder
2011-02-26 7:09 ` David Miller
2010-09-01 16:09 ` David Miller
2010-08-24 4:31 ` Bill Fink [this message]
2010-08-24 4:37 ` Tom Herbert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100824003143.243eb046.billfink@mindspring.com \
--to=billfink@mindspring$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=therbert@google$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox