Subject: Re: [xdp-hints] Re: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff

From: Maciej Fijalkowski <[email-masked]>
Date: Thu, 24 Nov 2022 16:17:01 +0100

On Thu, Nov 24, 2022 at 03:39:20PM +0100, Toke Høiland-Jørgensen wrote:
> Jakub Kicinski <[email-masked]> writes:
> 
> > On Wed, 23 Nov 2022 22:55:21 +0100 Toke Høiland-Jørgensen wrote:
> >> > Good idea, prototyped below, lmk if it that's not what you had in mind.
> >> >
> >> > struct xdp_buff_xsk {
> >> > 	struct xdp_buff            xdp;                  /*     0    56 */
> >> > 	u8                         cb[16];               /*    56    16 */
> >> > 	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */  
> >> 
> >> As pahole helpfully says here, xdp_buff is actually only 8 bytes from
> >> being a full cache line. I thought about adding a 'cb' field like this
> >> to xdp_buff itself, but figured that since there's only room for a
> >> single pointer, why not just add that and let the driver point it to
> >> where it wants to store the extra context data?
> >
> > What if the driver wants to store multiple pointers or an integer or
> > whatever else? The single pointer seems quite arbitrary and not
> > strictly necessary.
> 
> Well, then you allocate a separate struct and point to that? Like I did
> in mlx5:
> 
> 
> +	struct mlx5_xdp_ctx mlctx = { .cqe = cqe, .rq = rq };
> +	struct xdp_buff xdp = { .drv_priv = &mlctx };
> 
> but yeah, this does give an extra pointer deref on access. I'm not
> really opposed to the cb field either, I just think it's a bit odd to
> put it in struct xdp_buff_xsk; that basically requires the driver to
> keep the layouts in sync.
> 
> Instead, why not but a cb field into xdp_buff itself so it can be used
> for both the XSK and the non-XSK paths? Then the driver can just
> typecast the xdp_buff into its own struct that has whatever data it
> wants in place of the cb field?

Why can't you simply have a pointer to xdp_buff in driver specific
xdp_buff container which would point to xdp_buff that is stack based (or
whatever else memory that will back it up - I am about to push a change
that makes ice driver embed xdp_buff within a struct that represents Rx
ring) for XDP path and for ZC the pointer to xdp_buff that you get from
xsk_buff_pool ? This would satisfy both sides I believe and would let us
keep the same container struct.

struct mlx4_xdp_buff {
	struct xdp_buff *xdp;
	struct mlx4_cqe *cqe;
	struct mlx4_en_dev *mdev;
	struct mlx4_en_rx_ring *ring;
	struct net_device *dev;
};

(...)

	struct mlx4_xdp_buff mxbuf;
	struct xdp_buff xdp;

	mxbuf.xdp = &xdp;
	xdp_init_buff(mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);

Also these additional things

+			mxbuf.cqe = cqe;
+			mxbuf.mdev = priv->mdev;
+			mxbuf.ring = ring;
+			mxbuf.dev = dev;

could be assigned once at a setup time or in worse case once per NAPI. So
maybe mlx4_xdp_buff shouldn't be stack based?