From: Alexey Kardashevskiy <aik@ozlabs•ru>
To: Gavin Shan <gwshan@linux•vnet.ibm.com>,
Wei Yang <weiyang@linux•vnet.ibm.com>
Cc: benh@kernel•crashing.org, linuxppc-dev@ozlabs•org
Subject: Re: [PATCH V2 4/6] powerpc/powernv: replace the hard coded boundary with gate
Date: Fri, 7 Aug 2015 19:11:25 +1000 [thread overview]
Message-ID: <55C4763D.6050703@ozlabs.ru> (raw)
In-Reply-To: <20150806052633.GA4767@gwshan>
On 08/06/2015 03:26 PM, Gavin Shan wrote:
> On Wed, Aug 05, 2015 at 09:25:01AM +0800, Wei Yang wrote:
>> Based on the limitation of M64 Window size, when VF BAR size is bigger than
>> 64MB, IOV BAR just round up power of 2 of the total_vfs. While the 64MB is
>> a magic boundary in code, which is hard to maintain.
>>
>> This patch replaces the hard coded boundary with gate, which is calculated
>>from m64_segsize and adds comment to explain the reason for it.
>>
>> Signed-off-by: Wei Yang <weiyang@linux•vnet.ibm.com>
>> ---
>> arch/powerpc/platforms/powernv/pci-ioda.c | 22 +++++++++++++++++-----
>> 1 file changed, 17 insertions(+), 5 deletions(-)
>>
>> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
>> index f5d110c..31dcedc 100644
>> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>> @@ -2702,7 +2702,7 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
>> struct pnv_phb *phb;
>> struct resource *res;
>> int i;
>> - resource_size_t size;
>> + resource_size_t size, gate;
>> struct pci_dn *pdn;
>> int mul, total_vfs;
>>
>> @@ -2718,6 +2718,17 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
>>
>> total_vfs = pci_sriov_get_totalvfs(pdev);
>> mul = phb->ioda.total_pe;
>> + /*
>> + * If bigger than or equal to half of m64_segsize, just round up power
>> + * of two.
>> + *
>> + * Generally, one M64 BAR maps one IOV BAR. To avoid conflict with
>> + * other devices, IOV BAR size is expanded to be (total_pe * VF size).
>> + * When VF size is half of m64_segsize , the expanded size would equal
>> + * to half of the whole M64 Window size, which will exhaust the M64
>> + * Window and limit the system flexibility.
>> + */
>
> s/VF size/VF BAR size
> s/m64_segsize/M64 segment size
> s/M64 Window/M64 space
I thought I started understanding the stuff and you just introduces new
term - "M64 space". Not "64bit MMIO space" but "M64 space" - what is this?
Is that 64GB 64bit MMIO window which we get from the hostboot?
>
>> + gate = phb->ioda.m64_segsize >> 1;
>>
>> for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
>> res = &pdev->resource[i + PCI_IOV_RESOURCES];
>> @@ -2732,10 +2743,11 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
>>
>> size = pci_iov_resource_size(pdev, i + PCI_IOV_RESOURCES);
>>
>> - /* bigger than 64M */
>> - if (size > (1 << 26)) {
>> - dev_info(&pdev->dev, "PowerNV: VF BAR%d: %pR IOV size is bigger than 64M, roundup power2\n",
>> - i, res);
>> + /* bigger than or equal to gate */
>> + if (size >= gate) {
>> + dev_info(&pdev->dev, "PowerNV: VF BAR%d: %pR IOV size "
>> + "is bigger than %lld, roundup power2\n",
>> + i, res, gate);
>
> If I understand the changes correctly, single VF BAR size is still checked against
> the "gate" (128MB), not the total VF BAR size. Recap the comments I gave last time:
>
> I mean to check the sum of all VF BARs. For example, the VFs attached to its PF has two
> VF BARs and each of them is 64MB. For this case, the MMIO resource can't be allocated
> once extending them to 256 VFs. So we have to try "single-pe-mode" for this situation.
> So the check becomes as below:
>
> struct pci_controller *hose = pci_bus_to_host(pdev->bus);
> struct pnv_phb *phb = hose->private_data;
> resource_size_t total_vf_bar_sz = 0;
> resource_size_t gate;
>
> /* Some comments to explain the "gate" */
> gate = phb->m64_segsize / 2;
> for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> total_vf_bar_sz += pci_iov_resource_size(pdev, PCI_IOV_RESOURCES + i);
>
> if (total_vf_bar_sz >= gate)
Why would be compare to the total size of the BARs? If VFs have 3 64MB BARs
each (these are 64bit BARs so up to 3 per VF, right?), which is 192MB in
total per VF, we can use 3 M64's, each in segmented mode (1 segment ==
64MB) and cover many VFs.
> /* single-pe-mode */
> else
> /* shared-mode */
>
>> mul = roundup_pow_of_two(total_vfs);
>> pdn->m64_single_mode = true;
>> break;
>> --
>> 1.7.9.5
>>
>
--
Alexey
next prev parent reply other threads:[~2015-08-07 9:11 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-29 7:22 [PATCH] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR Wei Yang
2015-07-30 1:15 ` Gavin Shan
2015-07-30 5:43 ` Wei Yang
2015-07-31 0:13 ` Gavin Shan
2015-07-31 2:01 ` Wei Yang
2015-08-05 1:24 ` [PATCH V2 0/6] Redesign SR-IOV on PowerNV Wei Yang
2015-08-05 1:24 ` [PATCH V2 1/6] powerpc/powernv: don't enable SRIOV when VF BAR contains non M64 BAR Wei Yang
2015-08-06 4:35 ` Gavin Shan
2015-08-06 6:10 ` Alexey Kardashevskiy
2015-08-06 6:57 ` Gavin Shan
2015-08-06 7:47 ` Alexey Kardashevskiy
2015-08-06 11:07 ` Gavin Shan
2015-08-06 14:13 ` Wei Yang
2015-08-07 1:24 ` Alexey Kardashevskiy
2015-08-06 14:10 ` Wei Yang
2015-08-07 1:20 ` Gavin Shan
2015-08-07 2:24 ` Wei Yang
2015-08-07 3:50 ` Gavin Shan
2015-08-07 7:14 ` Alexey Kardashevskiy
2015-08-10 1:40 ` Wei Yang
2015-08-05 1:24 ` [PATCH V2 2/6] powerpc/powernv: simplify the calculation of iov resource Wei Yang
2015-08-06 4:51 ` Gavin Shan
2015-08-06 9:00 ` Alexey Kardashevskiy
2015-08-06 9:41 ` Wei Yang
2015-08-06 10:15 ` Alexey Kardashevskiy
2015-08-07 1:36 ` Wei Yang
2015-08-06 13:49 ` Wei Yang
2015-08-07 1:08 ` Gavin Shan
2015-08-05 1:25 ` [PATCH V2 3/6] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR Wei Yang
2015-08-06 5:20 ` Gavin Shan
2015-08-06 9:36 ` Wei Yang
2015-08-06 10:07 ` Gavin Shan
2015-08-07 1:48 ` Wei Yang
2015-08-07 8:13 ` Alexey Kardashevskiy
2015-08-06 10:04 ` Alexey Kardashevskiy
2015-08-07 2:01 ` Wei Yang
2015-08-07 8:59 ` Alexey Kardashevskiy
2015-08-10 1:48 ` Wei Yang
2015-08-05 1:25 ` [PATCH V2 4/6] powerpc/powernv: replace the hard coded boundary with gate Wei Yang
2015-08-06 5:26 ` Gavin Shan
2015-08-07 9:11 ` Alexey Kardashevskiy [this message]
2015-08-05 1:25 ` [PATCH V2 5/6] powerpc/powernv: boundary the total vf bar size instead of the individual one Wei Yang
2015-08-06 5:28 ` Gavin Shan
2015-08-06 14:03 ` Wei Yang
2015-08-07 1:23 ` Gavin Shan
2015-08-07 2:25 ` Wei Yang
2015-08-05 1:25 ` [PATCH V2 6/6] powerpc/powernv: allocate discrete PE# when using M64 BAR in Single PE mode Wei Yang
2015-08-06 5:36 ` Gavin Shan
2015-08-06 13:41 ` Wei Yang
2015-08-07 1:36 ` Gavin Shan
2015-08-07 2:33 ` Wei Yang
2015-08-07 3:43 ` Gavin Shan
2015-08-07 5:44 ` Wei Yang
2015-08-07 5:54 ` Gavin Shan
2015-08-07 6:25 ` Wei Yang
2015-08-07 10:00 ` Alexey Kardashevskiy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55C4763D.6050703@ozlabs.ru \
--to=aik@ozlabs$(echo .)ru \
--cc=benh@kernel$(echo .)crashing.org \
--cc=gwshan@linux$(echo .)vnet.ibm.com \
--cc=linuxppc-dev@ozlabs$(echo .)org \
--cc=weiyang@linux$(echo .)vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox