From: Pavel Begunkov <asml.silence@gmail•com>
To: Jakub Kicinski <kuba@kernel•org>
Cc: David Wei <dw@davidwei•uk>,
io-uring@vger•kernel.org, netdev@vger•kernel.org,
Jens Axboe <axboe@kernel•dk>, Paolo Abeni <pabeni@redhat•com>,
"David S. Miller" <davem@davemloft•net>,
Eric Dumazet <edumazet@google•com>,
Jesper Dangaard Brouer <hawk@kernel•org>,
David Ahern <dsahern@kernel•org>,
Mina Almasry <almasrymina@google•com>,
Stanislav Fomichev <stfomichev@gmail•com>,
Joe Damato <jdamato@fastly•com>,
Pedro Tammela <pctammela@mojatatu•com>
Subject: Re: [PATCH net-next v8 11/17] io_uring/zcrx: implement zerocopy receive pp memory provider
Date: Wed, 11 Dec 2024 14:42:43 +0000 [thread overview]
Message-ID: <95e02ca4-4f0d-4f74-a882-6c975b345daa@gmail.com> (raw)
In-Reply-To: <20241210162412.6f04a505@kernel.org>
On 12/11/24 00:24, Jakub Kicinski wrote:
> On Tue, 10 Dec 2024 04:45:23 +0000 Pavel Begunkov wrote:
>>> Can you say more about the IO_ZC_RX_UREF bias? net_iov is not the page
>>> struct, we can add more fields. In fact we have 8B of padding in it
>>> that can be allocated without growing the struct. So why play with
>>
>> I guess we can, though it's growing it for everyone not just
>> io_uring considering how indexing works, i.e. no embedding into
>> a larger struct.
>
> Right but we literally have 8B of "padding". We only need 32b counter
> here, so there will still be 4B of unused space. Not to mention that
> net_iov is not cacheline aligned today. Space is not a concern.
>
>>> biases? You can add a 32b atomic counter for how many refs have been
>>> handed out to the user.
>>
>> This set does it in a stupid way, but the bias allows to coalesce
>> operations with it into a single atomic. Regardless, it can be
>> placed separately, though we still need a good way to optimise
>> counting. Take a look at my reply with questions in the v7 thread,
>> I outlined what can work quite well in terms of performance but
>> needs a clear api for that from net/
>
> I was thinking along the lines of transferring the ownership of
> the frags. But let's work on that as a follow up. Atomic add on
That's fine to leave it out for now and deal later, but what's
important for me when going through preliminary shittification of
the project is to have a way to optimise it after and a clear
understanding that it can't be left w/o it, and that there are
no strong opinions that would block it.
The current cache situation is too unfortunate, understandably so
with it being aliased to struct page. pp_ref_count is in the
same line with ->pp and others. Here an iov usually gets modified
by napi, then refcounted from syscall, after deferred skb put will
put it down back at napi context, and in some time after it gets
washed out from the cache, the user will finally return it back
to page pool.
> an exclusively owned cacheline is 2 cycles on AMD if I'm looking
> correctly.
Sounds too good to be true considering x86 implies a full barrier
for atomics. I wonder where the data comes from?
--
Pavel Begunkov
next prev parent reply other threads:[~2024-12-11 14:41 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-12-04 17:21 [PATCH net-next v8 00/17] io_uring zero copy rx David Wei
2024-12-04 17:21 ` [PATCH net-next v8 01/17] net: prefix devmem specific helpers David Wei
2024-12-04 21:00 ` Mina Almasry
2024-12-04 21:24 ` David Wei
2024-12-04 17:21 ` [PATCH net-next v8 02/17] net: generalise net_iov chunk owners David Wei
2024-12-09 17:01 ` Mina Almasry
2024-12-04 17:21 ` [PATCH net-next v8 03/17] net: page_pool: create hooks for custom page providers David Wei
2024-12-10 3:02 ` Jakub Kicinski
2024-12-10 16:31 ` David Wei
2024-12-04 17:21 ` [PATCH net-next v8 04/17] net: prepare for non devmem TCP memory providers David Wei
2024-12-09 17:04 ` Mina Almasry
2024-12-10 3:15 ` Jakub Kicinski
2024-12-10 3:53 ` Pavel Begunkov
2024-12-10 4:06 ` Jakub Kicinski
2024-12-10 4:15 ` Pavel Begunkov
2024-12-04 17:21 ` [PATCH net-next v8 05/17] net: page_pool: add ->scrub mem provider callback David Wei
2024-12-09 17:08 ` Mina Almasry
2024-12-09 17:24 ` Pavel Begunkov
2024-12-04 17:21 ` [PATCH net-next v8 06/17] net: page pool: add helper creating area from pages David Wei
2024-12-10 3:29 ` Jakub Kicinski
2024-12-10 3:58 ` Pavel Begunkov
2024-12-04 17:21 ` [PATCH net-next v8 07/17] net: page_pool: introduce page_pool_mp_return_in_cache David Wei
2024-12-09 17:15 ` Mina Almasry
2024-12-09 17:28 ` Pavel Begunkov
2024-12-10 3:40 ` Jakub Kicinski
2024-12-10 4:31 ` Pavel Begunkov
2024-12-11 0:06 ` Jakub Kicinski
2024-12-04 17:21 ` [PATCH net-next v8 08/17] net: add helper executing custom callback from napi David Wei
2024-12-10 3:44 ` Jakub Kicinski
2024-12-10 4:11 ` Pavel Begunkov
2024-12-04 17:21 ` [PATCH net-next v8 09/17] io_uring/zcrx: add interface queue and refill queue David Wei
2024-12-06 16:05 ` Simon Horman
2024-12-09 23:50 ` David Wei
2024-12-10 3:49 ` Jakub Kicinski
2024-12-10 4:03 ` Pavel Begunkov
2024-12-10 4:07 ` Jakub Kicinski
2024-12-04 17:21 ` [PATCH net-next v8 10/17] io_uring/zcrx: add io_zcrx_area David Wei
2024-12-04 17:21 ` [PATCH net-next v8 11/17] io_uring/zcrx: implement zerocopy receive pp memory provider David Wei
2024-12-10 4:01 ` Jakub Kicinski
2024-12-10 4:45 ` Pavel Begunkov
2024-12-10 4:50 ` Pavel Begunkov
2024-12-11 0:24 ` Jakub Kicinski
2024-12-11 14:42 ` Pavel Begunkov [this message]
2024-12-12 1:38 ` Jakub Kicinski
2024-12-12 13:42 ` Pavel Begunkov
2024-12-10 16:55 ` Mina Almasry
2024-12-04 17:21 ` [PATCH net-next v8 12/17] io_uring/zcrx: add io_recvzc request David Wei
2024-12-04 17:21 ` [PATCH net-next v8 13/17] io_uring/zcrx: set pp memory provider for an rx queue David Wei
2024-12-04 17:21 ` [PATCH net-next v8 14/17] io_uring/zcrx: add copy fallback David Wei
2024-12-04 17:21 ` [PATCH net-next v8 15/17] io_uring/zcrx: throttle receive requests David Wei
2024-12-04 17:21 ` [PATCH net-next v8 16/17] net: add documentation for io_uring zcrx David Wei
2024-12-09 17:51 ` Mina Almasry
2024-12-10 16:53 ` David Wei
2024-12-09 17:52 ` Mina Almasry
2024-12-10 16:54 ` David Wei
2024-12-04 17:21 ` [PATCH net-next v8 17/17] io_uring/zcrx: add selftest David Wei
2024-12-04 18:59 ` [PATCH net-next v8 00/17] io_uring zero copy rx Pavel Begunkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=95e02ca4-4f0d-4f74-a882-6c975b345daa@gmail.com \
--to=asml.silence@gmail$(echo .)com \
--cc=almasrymina@google$(echo .)com \
--cc=axboe@kernel$(echo .)dk \
--cc=davem@davemloft$(echo .)net \
--cc=dsahern@kernel$(echo .)org \
--cc=dw@davidwei$(echo .)uk \
--cc=edumazet@google$(echo .)com \
--cc=hawk@kernel$(echo .)org \
--cc=io-uring@vger$(echo .)kernel.org \
--cc=jdamato@fastly$(echo .)com \
--cc=kuba@kernel$(echo .)org \
--cc=netdev@vger$(echo .)kernel.org \
--cc=pabeni@redhat$(echo .)com \
--cc=pctammela@mojatatu$(echo .)com \
--cc=stfomichev@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox