From: "hch@lst•de" <hch@lst•de>
To: Michael Kelley <mikelley@microsoft•com>
Cc: Tianyu Lan <ltykernel@gmail•com>,
KY Srinivasan <kys@microsoft•com>,
Haiyang Zhang <haiyangz@microsoft•com>,
Stephen Hemminger <sthemmin@microsoft•com>,
"wei.liu@kernel•org" <wei.liu@kernel•org>,
Dexuan Cui <decui@microsoft•com>,
"tglx@linutronix•de" <tglx@linutronix•de>,
"mingo@redhat•com" <mingo@redhat•com>,
"bp@alien8•de" <bp@alien8•de>, "x86@kernel•org" <x86@kernel•org>,
"hpa@zytor•com" <hpa@zytor•com>,
"dave.hansen@linux•intel.com" <dave.hansen@linux•intel.com>,
"luto@kernel•org" <luto@kernel•org>,
"peterz@infradead•org" <peterz@infradead•org>,
"konrad.wilk@oracle•com" <konrad.wilk@oracle•com>,
"boris.ostrovsky@oracle•com" <boris.ostrovsky@oracle•com>,
"jgross@suse•com" <jgross@suse•com>,
"sstabellini@kernel•org" <sstabellini@kernel•org>,
"joro@8bytes•org" <joro@8bytes•org>,
"will@kernel•org" <will@kernel•org>,
"davem@davemloft•net" <davem@davemloft•net>,
"kuba@kernel•org" <kuba@kernel•org>,
"jejb@linux•ibm.com" <jejb@linux•ibm.com>,
"martin.petersen@oracle•com" <martin.petersen@oracle•com>,
"arnd@arndb•de" <arnd@arndb•de>, "hch@lst•de" <hch@lst•de>,
"m.szyprowski@samsung•com" <m.szyprowski@samsung•com>,
"robin.murphy@arm•com" <robin.murphy@arm•com>,
"thomas.lendacky@amd•com" <thomas.lendacky@amd•com>,
"brijesh.singh@amd•com" <brijesh.singh@amd•com>,
"ardb@kernel•org" <ardb@kernel•org>,
Tianyu Lan <Tianyu.Lan@microsoft•com>,
"pgonda@google•com" <pgonda@google•com>,
"martin.b.radev@gmail•com" <martin.b.radev@gmail•com>,
"akpm@linux-foundation•org" <akpm@linux-foundation•org>,
"kirill.shutemov@linux•intel.com"
<kirill.shutemov@linux•intel.com>,
"rppt@kernel•org" <rppt@kernel•org>,
"sfr@canb•auug.org.au" <sfr@canb•auug.org.au>,
"saravanand@fb•com" <saravanand@fb•com>,
"krish.sadhukhan@oracle•com" <krish.sadhukhan@oracle•com>,
"aneesh.kumar@linux•ibm.com" <aneesh.kumar@linux•ibm.com>,
"xen-devel@lists•xenproject.org" <xen-devel@lists•xenproject.org>,
"rientjes@google•com" <rientjes@google•com>,
"hannes@cmpxchg•org" <hannes@cmpxchg•org>,
"tj@kernel•org" <tj@kernel•org>,
"iommu@lists•linux-foundation.org"
<iommu@lists•linux-foundation.org>,
"linux-arch@vger•kernel.org" <linux-arch@vger•kernel.org>,
"linux-hyperv@vger•kernel.org" <linux-hyperv@vger•kernel.org>,
"linux-kernel@vger•kernel.org" <linux-kernel@vger•kernel.org>,
"linux-scsi@vger•kernel.org" <linux-scsi@vger•kernel.org>,
"netdev@vger•kernel.org" <netdev@vger•kernel.org>,
vkuznets <vkuznets@redhat•com>,
"parri.andrea@gmail•com" <parri.andrea@gmail•com>,
"dave.hansen@intel•com" <dave.hansen@intel•com>
Subject: Re: [PATCH V3 13/13] HV/Storvsc: Add Isolation VM support for storvsc driver
Date: Fri, 20 Aug 2021 06:32:37 +0200 [thread overview]
Message-ID: <20210820043237.GC26450@lst.de> (raw)
In-Reply-To: <MWHPR21MB1593EEF30FFD5C60ED744985D7C09@MWHPR21MB1593.namprd21.prod.outlook.com>
On Thu, Aug 19, 2021 at 06:17:40PM +0000, Michael Kelley wrote:
> > +#define storvsc_dma_map(dev, page, offset, size, dir) \
> > + dma_map_page(dev, page, offset, size, dir)
> > +
> > +#define storvsc_dma_unmap(dev, dma_range, dir) \
> > + dma_unmap_page(dev, dma_range.dma, \
> > + dma_range.mapping_size, \
> > + dir ? DMA_FROM_DEVICE : DMA_TO_DEVICE)
> > +
>
> Each of these macros is used only once. IMHO, they don't
> add a lot of value. Just coding dma_map/unmap_page()
> inline would be fine and eliminate these lines of code.
Yes, I had the same thought when looking over the code. Especially
as macros tend to further obsfucate the code (compared to actual helper
functions).
> > + for (i = 0; i < request->hvpg_count; i++)
> > + storvsc_dma_unmap(&device->device,
> > + request->dma_range[i],
> > + request->vstor_packet.vm_srb.data_in == READ_TYPE);
>
> I think you can directly get the DMA direction as request->cmd->sc_data_direction.
Yes.
> >
> > @@ -1824,6 +1848,13 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *scmnd)
> > payload->range.len = length;
> > payload->range.offset = offset_in_hvpg;
> >
> > + cmd_request->dma_range = kcalloc(hvpg_count,
> > + sizeof(*cmd_request->dma_range),
> > + GFP_ATOMIC);
>
> With this patch, it appears that storvsc_queuecommand() is always
> doing bounce buffering, even when running in a non-isolated VM.
> The dma_range is always allocated, and the inner loop below does
> the dma mapping for every I/O page. The corresponding code in
> storvsc_on_channel_callback() that does the dma unmap allows for
> the dma_range to be NULL, but that never happens.
Maybe I'm missing something in the hyperv code, but I don't think
dma_map_page would bounce buffer for the non-isolated case. It
will just return the physical address.
> > + if (!cmd_request->dma_range) {
> > + ret = -ENOMEM;
>
> The other memory allocation failure in this function returns
> SCSI_MLQUEUE_DEVICE_BUSY. It may be debatable as to whether
> that's the best approach, but that's a topic for a different patch. I
> would suggest being consistent and using the same return code
> here.
Independent of if SCSI_MLQUEUE_DEVICE_BUSY is good (it it a common
pattern in SCSI drivers), ->queuecommand can't return normal
negative errnos. It must return the SCSI_MLQUEUE_* codes or 0.
We should probably change the return type of the method definition
to a suitable enum to make this more clear.
> > + if (offset_in_hvpg) {
> > + payload->range.offset = dma & ~HV_HYP_PAGE_MASK;
> > + offset_in_hvpg = 0;
> > + }
>
> I'm not clear on why payload->range.offset needs to be set again.
> Even after the dma mapping is done, doesn't the offset in the first
> page have to be the same? If it wasn't the same, Hyper-V wouldn't
> be able to process the PFN list correctly. In fact, couldn't the above
> code just always set offset_in_hvpg = 0?
Careful. DMA mapping is supposed to keep the offset in the page, but
for that the DMA mapping code needs to know what the device considers a
"page". For that the driver needs to set the min_align_mask field in
struct device_dma_parameters.
>
> The whole approach here is to do dma remapping on each individual page
> of the I/O buffer. But wouldn't it be possible to use dma_map_sg() to map
> each scatterlist entry as a unit? Each scatterlist entry describes a range of
> physically contiguous memory. After dma_map_sg(), the resulting dma
> address must also refer to a physically contiguous range in the swiotlb
> bounce buffer memory. So at the top of the "for" loop over the scatterlist
> entries, do dma_map_sg() if we're in an isolated VM. Then compute the
> hvpfn value based on the dma address instead of sg_page(). But everything
> else is the same, and the inner loop for populating the pfn_arry is unmodified.
> Furthermore, the dma_range array that you've added is not needed, since
> scatterlist entries already have a dma_address field for saving the mapped
> address, and dma_unmap_sg() uses that field.
Yes, I think dma_map_sg is the right thing to use here, probably even
for the non-isolated case so that we can get the hv drivers out of their
little corner and into being more like a normal kernel driver. That
is, use the scsi_dma_map/scsi_dma_unmap helpers, and then iterate over
the dma addresses one page at a time using for_each_sg_dma_page.
>
> One thing: There's a maximum swiotlb mapping size, which I think works
> out to be 256 Kbytes. See swiotlb_max_mapping_size(). We need to make
> sure that we don't get a scatterlist entry bigger than this size. But I think
> this already happens because you set the device->dma_mask field in
> Patch 11 of this series. __scsi_init_queue checks for this setting and
> sets max_sectors to limits transfers to the max mapping size.
Indeed.
next prev parent reply other threads:[~2021-08-20 4:32 UTC|newest]
Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-09 17:56 [PATCH V3 00/13] x86/Hyper-V: Add Hyper-V Isolation VM support Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 01/13] x86/HV: Initialize GHCB page in Isolation VM Tianyu Lan
2021-08-10 10:56 ` Wei Liu
2021-08-10 12:17 ` Tianyu Lan
2021-08-12 19:14 ` Michael Kelley
2021-08-13 15:46 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 02/13] x86/HV: Initialize shared memory boundary in the " Tianyu Lan
2021-08-12 19:18 ` Michael Kelley
2021-08-14 13:32 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 03/13] x86/HV: Add new hvcall guest address host visibility support Tianyu Lan
2021-08-09 22:12 ` Dave Hansen
2021-08-10 13:09 ` Tianyu Lan
2021-08-10 11:03 ` Wei Liu
2021-08-10 12:25 ` Tianyu Lan
2021-08-12 19:36 ` Michael Kelley
2021-08-12 21:10 ` Michael Kelley
2021-08-09 17:56 ` [PATCH V3 04/13] HV: Mark vmbus ring buffer visible to host in Isolation VM Tianyu Lan
2021-08-12 22:20 ` Michael Kelley
2021-08-15 15:21 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 05/13] HV: Add Write/Read MSR registers via ghcb page Tianyu Lan
2021-08-13 19:31 ` Michael Kelley
2021-08-13 20:26 ` Michael Kelley
2021-08-24 8:45 ` Christoph Hellwig
2021-08-09 17:56 ` [PATCH V3 06/13] HV: Add ghcb hvcall support for SNP VM Tianyu Lan
2021-08-13 20:42 ` Michael Kelley
2021-08-09 17:56 ` [PATCH V3 07/13] HV/Vmbus: Add SNP support for VMbus channel initiate message Tianyu Lan
2021-08-13 21:28 ` Michael Kelley
2021-08-09 17:56 ` [PATCH V3 08/13] HV/Vmbus: Initialize VMbus ring buffer for Isolation VM Tianyu Lan
2021-08-16 17:28 ` Michael Kelley
2021-08-17 15:36 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 09/13] DMA: Add dma_map_decrypted/dma_unmap_encrypted() function Tianyu Lan
2021-08-12 12:26 ` Christoph Hellwig
2021-08-12 15:38 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 10/13] x86/Swiotlb: Add Swiotlb bounce buffer remap function for HV IVM Tianyu Lan
2021-08-12 12:27 ` Christoph Hellwig
2021-08-13 17:58 ` Tianyu Lan
2021-08-16 14:50 ` Tianyu Lan
2021-08-19 8:49 ` Christoph Hellwig
2021-08-19 9:59 ` Tianyu Lan
2021-08-19 10:02 ` Christoph Hellwig
2021-08-19 10:03 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 11/13] HV/IOMMU: Enable swiotlb bounce buffer for Isolation VM Tianyu Lan
2021-08-19 18:11 ` Michael Kelley
2021-08-20 4:13 ` hch
2021-08-20 9:32 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 12/13] HV/Netvsc: Add Isolation VM support for netvsc driver Tianyu Lan
2021-08-19 18:14 ` Michael Kelley
2021-08-20 4:21 ` hch
2021-08-20 13:11 ` Tianyu Lan
2021-08-20 13:30 ` Tom Lendacky
2021-08-20 18:20 ` Tianyu Lan
2021-08-09 17:56 ` [PATCH V3 13/13] HV/Storvsc: Add Isolation VM support for storvsc driver Tianyu Lan
2021-08-19 18:17 ` Michael Kelley
2021-08-20 4:32 ` hch [this message]
2021-08-20 15:40 ` Michael Kelley
2021-08-24 8:49 ` min_align_mask " hch
2021-08-20 16:01 ` Tianyu Lan
2021-08-20 15:20 ` Tianyu Lan
2021-08-20 15:37 ` Tianyu Lan
2021-08-20 16:08 ` Michael Kelley
2021-08-20 18:04 ` Tianyu Lan
2021-08-20 19:22 ` Michael Kelley
2021-08-24 8:46 ` hch
2021-08-16 14:55 ` [PATCH V3 00/13] x86/Hyper-V: Add Hyper-V Isolation VM support Michael Kelley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210820043237.GC26450@lst.de \
--to=hch@lst$(echo .)de \
--cc=Tianyu.Lan@microsoft$(echo .)com \
--cc=akpm@linux-foundation$(echo .)org \
--cc=aneesh.kumar@linux$(echo .)ibm.com \
--cc=ardb@kernel$(echo .)org \
--cc=arnd@arndb$(echo .)de \
--cc=boris.ostrovsky@oracle$(echo .)com \
--cc=bp@alien8$(echo .)de \
--cc=brijesh.singh@amd$(echo .)com \
--cc=dave.hansen@intel$(echo .)com \
--cc=dave.hansen@linux$(echo .)intel.com \
--cc=davem@davemloft$(echo .)net \
--cc=decui@microsoft$(echo .)com \
--cc=haiyangz@microsoft$(echo .)com \
--cc=hannes@cmpxchg$(echo .)org \
--cc=hpa@zytor$(echo .)com \
--cc=iommu@lists$(echo .)linux-foundation.org \
--cc=jejb@linux$(echo .)ibm.com \
--cc=jgross@suse$(echo .)com \
--cc=joro@8bytes$(echo .)org \
--cc=kirill.shutemov@linux$(echo .)intel.com \
--cc=konrad.wilk@oracle$(echo .)com \
--cc=krish.sadhukhan@oracle$(echo .)com \
--cc=kuba@kernel$(echo .)org \
--cc=kys@microsoft$(echo .)com \
--cc=linux-arch@vger$(echo .)kernel.org \
--cc=linux-hyperv@vger$(echo .)kernel.org \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linux-scsi@vger$(echo .)kernel.org \
--cc=ltykernel@gmail$(echo .)com \
--cc=luto@kernel$(echo .)org \
--cc=m.szyprowski@samsung$(echo .)com \
--cc=martin.b.radev@gmail$(echo .)com \
--cc=martin.petersen@oracle$(echo .)com \
--cc=mikelley@microsoft$(echo .)com \
--cc=mingo@redhat$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=parri.andrea@gmail$(echo .)com \
--cc=peterz@infradead$(echo .)org \
--cc=pgonda@google$(echo .)com \
--cc=rientjes@google$(echo .)com \
--cc=robin.murphy@arm$(echo .)com \
--cc=rppt@kernel$(echo .)org \
--cc=saravanand@fb$(echo .)com \
--cc=sfr@canb$(echo .)auug.org.au \
--cc=sstabellini@kernel$(echo .)org \
--cc=sthemmin@microsoft$(echo .)com \
--cc=tglx@linutronix$(echo .)de \
--cc=thomas.lendacky@amd$(echo .)com \
--cc=tj@kernel$(echo .)org \
--cc=vkuznets@redhat$(echo .)com \
--cc=wei.liu@kernel$(echo .)org \
--cc=will@kernel$(echo .)org \
--cc=x86@kernel$(echo .)org \
--cc=xen-devel@lists$(echo .)xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox