From: Jesper Dangaard Brouer <brouer@redhat•com>
To: Tariq Toukan <tariqt@mellanox•com>
Cc: netdev@vger•kernel.org, BjörnTöpel <bjorn.topel@intel•com>,
magnus.karlsson@intel•com, eugenia@mellanox•com,
"Jason Wang" <jasowang@redhat•com>,
"John Fastabend" <john.fastabend@gmail•com>,
"Eran Ben Elisha" <eranbe@mellanox•com>,
"Saeed Mahameed" <saeedm@mellanox•com>,
galp@mellanox•com, "Daniel Borkmann" <borkmann@iogearbox•net>,
"Alexei Starovoitov" <alexei.starovoitov@gmail•com>,
brouer@redhat•com
Subject: Re: [bpf-next V3 PATCH 13/15] mlx5: use page_pool for xdp_return_frame call
Date: Mon, 19 Mar 2018 14:12:17 +0100 [thread overview]
Message-ID: <20180319141217.416d269a@redhat.com> (raw)
In-Reply-To: <6e62fa6a-53c7-57a9-0493-3a48d832b479@mellanox.com>
On Mon, 12 Mar 2018 15:20:06 +0200 Tariq Toukan <tariqt@mellanox•com> wrote:
> On 12/03/2018 12:16 PM, Tariq Toukan wrote:
> >
> > On 12/03/2018 12:08 PM, Tariq Toukan wrote:
> >>
> >> On 09/03/2018 10:56 PM, Jesper Dangaard Brouer wrote:
> >>> This patch shows how it is possible to have both the driver local page
> >>> cache, which uses elevated refcnt for "catching"/avoiding SKB
> >>> put_page. And at the same time, have pages getting returned to the
> >>> page_pool from ndp_xdp_xmit DMA completion.
> >>>
[...]
> >>>
> >>> Before this patch: single flow performance was 6Mpps, and if I started
> >>> two flows the collective performance drop to 4Mpps, because we hit the
> >>> page allocator lock (further negative scaling occurs).
> >>>
> >>> V2: Adjustments requested by Tariq
> >>> - Changed page_pool_create return codes not return NULL, only
> >>> ERR_PTR, as this simplifies err handling in drivers.
> >>> - Save a branch in mlx5e_page_release
> >>> - Correct page_pool size calc for MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ
> >>>
> >>> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat•com>
> >>> ---
> >>
> >> I am running perf tests with your series. I sense a drastic
> >> degradation in regular TCP flows, I'm double checking the numbers now...
> >
> > Well, there's a huge performance degradation indeed, whenever the
> > regular flows (non-XDP) use the new page pool. Cannot merge before
> > fixing this.
> >
> > If I disable the local page-cache, numbers get as low as 100's of Mbps
> > in TCP stream tests.
>
> It seems that the page-pool doesn't fit as a general fallback (when page
> in local rx cache is busy), as the refcnt is elevated/changing:
I see the issue. I have to go over the details in the driver, but I
think it should be sufficient to remove the WARN(). When the page_pool
was integrated with the MM-layer, being invoked from the put_page()
call itself, this would indicate a likely API misuse. But now, with
the page refcnt based recycle tricks, it is the norm (for non-XDP) that
put_page is called without the knowledge of page_pool.
> [ 7343.086102] ------------[ cut here ]------------
> [ 7343.086103] __page_pool_put_page() violating page_pool invariance refcnt:0
> [ 7343.086114] WARNING: CPU: 1 PID: 17 at net/core/page_pool.c:291 __page_pool_put_page+0x7c/0xa0
Here page_pool actually catch the page refcnt race correctly, and does
the proper handling of returning it to the page allocator (via __put_page).
I do notice (in the page_pool code) that in case page_pool handles DMA
mapping (which isn't the case, yet), that I'm missing a DMA unmap
release in the code.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2018-03-19 13:12 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-03-09 20:55 [bpf-next V3 PATCH 00/15] XDP redirect memory return API Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 01/15] mlx5: basic XDP_REDIRECT forward support Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 02/15] xdp: introduce xdp_return_frame API and use in cpumap Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 03/15] ixgbe: use xdp_return_frame API Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 04/15] xdp: move struct xdp_buff from filter.h to xdp.h Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 05/15] xdp: introduce a new xdp_frame type Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 06/15] tun: convert to use generic xdp_frame and xdp_return_frame API Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 07/15] virtio_net: " Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 08/15] bpf: cpumap convert to use generic xdp_frame Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 09/15] mlx5: register a memory model when XDP is enabled Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 10/15] xdp: rhashtable with allocator ID to pointer mapping Jesper Dangaard Brouer
2018-03-09 20:55 ` [bpf-next V3 PATCH 11/15] page_pool: refurbish version of page_pool code Jesper Dangaard Brouer
2018-03-09 20:56 ` [bpf-next V3 PATCH 12/15] xdp: allow page_pool as an allocator type in xdp_return_frame Jesper Dangaard Brouer
2018-03-09 20:56 ` [bpf-next V3 PATCH 13/15] mlx5: use page_pool for xdp_return_frame call Jesper Dangaard Brouer
2018-03-12 10:08 ` Tariq Toukan
2018-03-12 10:16 ` Tariq Toukan
2018-03-12 13:20 ` Tariq Toukan
2018-03-19 13:12 ` Jesper Dangaard Brouer [this message]
2018-03-20 7:43 ` Tariq Toukan
2018-03-20 8:18 ` Tariq Toukan
2018-03-09 20:56 ` [bpf-next V3 PATCH 14/15] xdp: transition into using xdp_frame for return API Jesper Dangaard Brouer
2018-03-09 20:56 ` [bpf-next V3 PATCH 15/15] xdp: transition into using xdp_frame for ndo_xdp_xmit Jesper Dangaard Brouer
2018-03-16 9:04 ` [bpf-next V3 PATCH 00/15] XDP redirect memory return API Jason Wang
2018-03-19 10:10 ` Jesper Dangaard Brouer
2018-03-20 2:28 ` Jason Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180319141217.416d269a@redhat.com \
--to=brouer@redhat$(echo .)com \
--cc=alexei.starovoitov@gmail$(echo .)com \
--cc=bjorn.topel@intel$(echo .)com \
--cc=borkmann@iogearbox$(echo .)net \
--cc=eranbe@mellanox$(echo .)com \
--cc=eugenia@mellanox$(echo .)com \
--cc=galp@mellanox$(echo .)com \
--cc=jasowang@redhat$(echo .)com \
--cc=john.fastabend@gmail$(echo .)com \
--cc=magnus.karlsson@intel$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=saeedm@mellanox$(echo .)com \
--cc=tariqt@mellanox$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox