Subject: Re: [PATCH V2 mlx5-next 12/14] vfio/mlx5: Implement vfio_pci driver for mlx5 devices

From: Jason Gunthorpe <[email-masked]>
Date: Tue, 26 Oct 2021 12:56:07 -0300

On Tue, Oct 26, 2021 at 08:52:10AM -0600, Alex Williamson wrote:
> On Tue, 26 Oct 2021 09:13:53 -0300
> Jason Gunthorpe <[email-masked]> wrote:
> 
> > On Tue, Oct 26, 2021 at 09:40:34AM +0100, Dr. David Alan Gilbert wrote:
> > > * Jason Gunthorpe ([email-masked]) wrote:  
> > > > On Mon, Oct 25, 2021 at 07:47:29PM +0100, Dr. David Alan Gilbert wrote:
> > > >   
> > > > > It may need some further refinement; for example in that quiesed state
> > > > > do counters still tick? will a NIC still respond to packets that don't
> > > > > get forwarded to the host?  
> > > > 
> > > > At least for the mlx5 NIC the two states are 'able to issue outbound
> > > > DMA' and 'all internal memories and state are frozen and unchanging'.  
> > > 
> > > Yeh, so my point was just that if you're adding a new state to this
> > > process, you need to define the details like that.  
> > 
> > We are not planning to propose any patches/uAPI specification for this
> > problem until after the mlx5 vfio driver is merged..
> 
> I'm not super comfortable with that.  If we're expecting to add a new
> bit to define a quiescent state prior to clearing the running flag and
> this is an optional device feature that userspace migration needs to be
> aware of and it's really not clear from a hypervisor when p2p DMA might
> be in use, I think that leaves userspace in a pickle how and when
> they'd impose restrictions on assignment with multiple assigned
> devices.  It's likely that the majority of initial use cases wouldn't
> need this feature, which would make it difficult to arbitrarily impose
> later.

Either supporting a freeze/quiesce split is an optional feature, and
current mlx5_vfio is an example of a driver that does not implement
it

Or, freeze/quiesce is a mandatory HW feature and no VFIO driver will
be merged that doesn't implement it - even if the HW cannot support
it.

Frankly, I think we will see HW that doesn't support this and VFIO
will be required to have it as an optional feature

What does userspace do? I don't know - but it does need to be aware if
the optional HW feature is available and it does need to make choices
if the VM will be migratable or not.

I don't see any path where userspace truly cannot care unless HW
support for this is mandatory.

Jason