From: Alexander Duyck <alexander.h.duyck@intel•com>
To: Prarit Bhargava <prarit@redhat•com>, netdev@vger•kernel.org
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel•com>,
Jesse Brandeburg <jesse.brandeburg@intel•com>,
Bruce Allan <bruce.w.allan@intel•com>,
Carolyn Wyborny <carolyn.wyborny@intel•com>,
Don Skidmore <donald.c.skidmore@intel•com>,
Greg Rose <gregory.v.rose@intel•com>,
John Ronciak <john.ronciak@intel•com>,
Mitch Williams <mitch.a.williams@intel•com>,
"David S. Miller" <davem@davemloft•net>,
nhorman@redhat•com, agospoda@redhat•com,
e1000-devel@lists•sourceforge.net
Subject: Re: [PATCH 0/2] ixgbe, fix numa issues
Date: Mon, 24 Feb 2014 11:23:42 -0800 [thread overview]
Message-ID: <530B9C3E.1000308@intel.com> (raw)
In-Reply-To: <1393267913-28212-1-git-send-email-prarit@redhat.com>
On 02/24/2014 10:51 AM, Prarit Bhargava wrote:
> The ixgbe driver makes some assumptions about the layout of cpus in the
> system which are not always correct given a particular system layout. The
> ixgbe driver allocates one MSI/cpu for queue usage but the code does not take
> into account that devices are located on NUMA nodes and that the cpus in a node
> are not contiguous.
>
> These issues were found while doing cpu hotplug testing, however, both of these
> issues can lead to obvious system performance issues as they defeat the
> purpose of having one MSI processing a queue per cpu.
>
> Cc: Jeff Kirsher <jeffrey.t.kirsher@intel•com>
> Cc: Jesse Brandeburg <jesse.brandeburg@intel•com>
> Cc: Bruce Allan <bruce.w.allan@intel•com>
> Cc: Carolyn Wyborny <carolyn.wyborny@intel•com>
> Cc: Don Skidmore <donald.c.skidmore@intel•com>
> Cc: Greg Rose <gregory.v.rose@intel•com>
> Cc: Alex Duyck <alexander.h.duyck@intel•com>
> Cc: John Ronciak <john.ronciak@intel•com>
> Cc: Mitch Williams <mitch.a.williams@intel•com>
> Cc: "David S. Miller" <davem@davemloft•net>
> Cc: nhorman@redhat•com
> Cc: agospoda@redhat•com
> Cc: e1000-devel@lists•sourceforge.net
>
> Prarit Bhargava (2):
> ixgbe, make interrupt allocations NUMA aware
> ixgbe, don't assume mapping of numa node cpus
>
> drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 ++
> drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 44 ++++++++++++++++++------
> drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 6 ++--
> drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c | 5 +--
> 4 files changed, 42 insertions(+), 15 deletions(-)
>
This is a step in the right direction but totally defeats the purpose of
ATR. With this change we might as well defeature ATR all together since
things are now back to RSS w/ NUMA specific allocations which is what we
had a couple of years ago. The code as it is written now would be a
better for for igb which doesn't have ATR than ixgbe.
ATR is supposed to map 1:1 queues to CPUs. The problem is RSS is also a
factor and not especially smart or NUMA aware. The ideal solution would
be to allocate the first N CPUs, where N is the number in the local node
for ATR/RSS. Then map all other queues as ATR with a 1:1 mapping to CPUs.
Thanks,
Alex
next prev parent reply other threads:[~2014-02-24 19:23 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-02-24 18:51 [PATCH 0/2] ixgbe, fix numa issues Prarit Bhargava
2014-02-24 18:51 ` [PATCH 1/2] ixgbe, make interrupt allocations NUMA aware Prarit Bhargava
2014-02-24 19:26 ` Alexander Duyck
2014-02-24 19:39 ` Prarit Bhargava
2014-02-24 19:49 ` Alexander Duyck
2014-02-24 18:51 ` [PATCH 2/2] ixgbe, don't assume mapping of numa node cpus Prarit Bhargava
2014-02-24 19:39 ` Alexander Duyck
2014-02-25 17:27 ` Amir Vadai
2014-02-25 17:43 ` Prarit Bhargava
2014-02-24 19:23 ` Alexander Duyck [this message]
2014-02-24 19:34 ` [PATCH 0/2] ixgbe, fix numa issues Prarit Bhargava
2014-02-24 19:57 ` Alexander Duyck
2014-02-25 1:06 ` Prarit Bhargava
2014-02-25 10:21 ` David Laight
2014-02-25 11:00 ` Prarit Bhargava
2014-02-25 15:10 ` Alexander Duyck
2014-02-25 15:13 ` Prarit Bhargava
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=530B9C3E.1000308@intel.com \
--to=alexander.h.duyck@intel$(echo .)com \
--cc=agospoda@redhat$(echo .)com \
--cc=bruce.w.allan@intel$(echo .)com \
--cc=carolyn.wyborny@intel$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=donald.c.skidmore@intel$(echo .)com \
--cc=e1000-devel@lists$(echo .)sourceforge.net \
--cc=gregory.v.rose@intel$(echo .)com \
--cc=jeffrey.t.kirsher@intel$(echo .)com \
--cc=jesse.brandeburg@intel$(echo .)com \
--cc=john.ronciak@intel$(echo .)com \
--cc=mitch.a.williams@intel$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=nhorman@redhat$(echo .)com \
--cc=prarit@redhat$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox