From: Alexey Kardashevskiy <aik@ozlabs•ru>
To: Wei Yang <weiyang@linux•vnet.ibm.com>,
Gavin Shan <gwshan@linux•vnet.ibm.com>
Cc: benh@kernel•crashing.org, linuxppc-dev@ozlabs•org
Subject: Re: [PATCH V2 1/6] powerpc/powernv: don't enable SRIOV when VF BAR contains non M64 BAR
Date: Fri, 7 Aug 2015 17:14:41 +1000 [thread overview]
Message-ID: <55C45AE1.3040404@ozlabs.ru> (raw)
In-Reply-To: <20150807022405.GF8292@richard>
On 08/07/2015 12:24 PM, Wei Yang wrote:
> On Fri, Aug 07, 2015 at 11:20:10AM +1000, Gavin Shan wrote:
>> On Thu, Aug 06, 2015 at 10:10:10PM +0800, Wei Yang wrote:
>>> On Thu, Aug 06, 2015 at 02:35:57PM +1000, Gavin Shan wrote:
>>>> On Wed, Aug 05, 2015 at 09:24:58AM +0800, Wei Yang wrote:
>>>>> On PHB_IODA2, we enable SRIOV devices by mapping IOV BAR with M64 BARs. If
>>>>> a SRIOV device's BAR is not 64-bit prefetchable, this is not assigned from
>>>>> M64 windwo, which means M64 BAR can't work on it.
>>>>>
>>>>
>>>> s/PHB_IODA2/PHB3
>>>> s/windwo/window
>>>>
>>>>> This patch makes this explicit.
>>>>>
>>>>> Signed-off-by: Wei Yang <weiyang@linux•vnet.ibm.com>
>>>>
>>>> The idea sounds right, but there is one question as below.
>>>>
>>>>> ---
>>>>> arch/powerpc/platforms/powernv/pci-ioda.c | 25 +++++++++----------------
>>>>> 1 file changed, 9 insertions(+), 16 deletions(-)
>>>>>
>>>>> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
>>>>> index 5738d31..9b41dba 100644
>>>>> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>>>>> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>>>>> @@ -908,9 +908,6 @@ static int pnv_pci_vf_resource_shift(struct pci_dev *dev, int offset)
>>>>> if (!res->flags || !res->parent)
>>>>> continue;
>>>>>
>>>>> - if (!pnv_pci_is_mem_pref_64(res->flags))
>>>>> - continue;
>>>>> -
>>>>> /*
>>>>> * The actual IOV BAR range is determined by the start address
>>>>> * and the actual size for num_vfs VFs BAR. This check is to
>>>>> @@ -939,9 +936,6 @@ static int pnv_pci_vf_resource_shift(struct pci_dev *dev, int offset)
>>>>> if (!res->flags || !res->parent)
>>>>> continue;
>>>>>
>>>>> - if (!pnv_pci_is_mem_pref_64(res->flags))
>>>>> - continue;
>>>>> -
>>>>> size = pci_iov_resource_size(dev, i + PCI_IOV_RESOURCES);
>>>>> res2 = *res;
>>>>> res->start += size * offset;
>>>>> @@ -1221,9 +1215,6 @@ static int pnv_pci_vf_assign_m64(struct pci_dev *pdev, u16 num_vfs)
>>>>> if (!res->flags || !res->parent)
>>>>> continue;
>>>>>
>>>>> - if (!pnv_pci_is_mem_pref_64(res->flags))
>>>>> - continue;
>>>>> -
>>>>> for (j = 0; j < vf_groups; j++) {
>>>>> do {
>>>>> win = find_next_zero_bit(&phb->ioda.m64_bar_alloc,
>>>>> @@ -1510,6 +1501,12 @@ int pnv_pci_sriov_enable(struct pci_dev *pdev, u16 num_vfs)
>>>>> pdn = pci_get_pdn(pdev);
>>>>>
>>>>> if (phb->type == PNV_PHB_IODA2) {
>>>>> + if (!pdn->vfs_expanded) {
>>>>> + dev_info(&pdev->dev, "don't support this SRIOV device"
>>>>> + " with non M64 VF BAR\n");
>>>>> + return -EBUSY;
>>>>> + }
>>>>> +
>>>>
>>>> It would be -ENOSPC since -EBUSY indicates the devices (VFs) are temparily
>>>> unavailable. For this case, the VFs are permanently unavailable because of
>>>> running out of space to accomodate M64 and non-M64 VF BARs.
>>>>
>>>> The error message could be printed with dev_warn() and it would be precise
>>>> as below or something else you prefer:
>>>>
>>>> dev_warn(&pdev->dev, "SRIOV not supported because of non-M64 VF BAR\n");
>>>>
>>>
>>> Thanks for the comment, will change accordingly.
>>>
>>>>
>>>>> /* Calculate available PE for required VFs */
>>>>> mutex_lock(&phb->ioda.pe_alloc_mutex);
>>>>> pdn->offset = bitmap_find_next_zero_area(
>>>>> @@ -2774,9 +2771,10 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
>>>>> if (!res->flags || res->parent)
>>>>> continue;
>>>>> if (!pnv_pci_is_mem_pref_64(res->flags)) {
>>>>> - dev_warn(&pdev->dev, " non M64 VF BAR%d: %pR\n",
>>>>> + dev_warn(&pdev->dev, "Don't support SR-IOV with"
>>>>> + " non M64 VF BAR%d: %pR. \n",
>>>>> i, res);
>>>>> - continue;
>>>>> + return;
>>>>> }
>>>>>
>>>>> size = pci_iov_resource_size(pdev, i + PCI_IOV_RESOURCES);
>>>>> @@ -2795,11 +2793,6 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
>>>>> res = &pdev->resource[i + PCI_IOV_RESOURCES];
>>>>> if (!res->flags || res->parent)
>>>>> continue;
>>>>> - if (!pnv_pci_is_mem_pref_64(res->flags)) {
>>>>> - dev_warn(&pdev->dev, "Skipping expanding VF BAR%d: %pR\n",
>>>>> - i, res);
>>>>> - continue;
>>>>> - }
>>>>
>>>> When any one IOV BAR on the PF is non-M64, none of the VFs can be enabled.
>>>> Will we still allocate/assign M64 or M32 resources for the IOV BARs? If so,
>>>> I think it can be avoided.
>>>>
>>>
>>> Don't get your point. You mean to avoid this function?
>>>
>>> Or clear the IOV BAR when we found one of it is non-M64?
>>>
>>
>> I mean to clear all IOV BARs in case any more of them are IO or M32. In this
>> case, the SRIOV capability won't be enabled. Otherwise, the resources for
>> all IOV BARs are assigned and allocated by PCI subsystem, but they won't
>> be used. Does it make sense to you?
>>
>
> If we want to save MMIO space, this is not necessary.
>
> The IOV BAR will be put into the optional list in assignment stage. So when
> there is not enough MMIO space, they will not be assigned.
If we are not going to use non-64bit IOV BAR, why would we assign anything
to it at the first place? Or it is a common PCI code which does it?
> For the long term, maybe P9/P10, we will finally adjust the solution to
> support SRIOV devices with M32 MMIO. So I suggest to leave as it is.
>
>>>>>
>>>>> dev_dbg(&pdev->dev, " Fixing VF BAR%d: %pR to\n", i, res);
>>>>> size = pci_iov_resource_size(pdev, i + PCI_IOV_RESOURCES);
--
Alexey
next prev parent reply other threads:[~2015-08-07 7:14 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-29 7:22 [PATCH] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR Wei Yang
2015-07-30 1:15 ` Gavin Shan
2015-07-30 5:43 ` Wei Yang
2015-07-31 0:13 ` Gavin Shan
2015-07-31 2:01 ` Wei Yang
2015-08-05 1:24 ` [PATCH V2 0/6] Redesign SR-IOV on PowerNV Wei Yang
2015-08-05 1:24 ` [PATCH V2 1/6] powerpc/powernv: don't enable SRIOV when VF BAR contains non M64 BAR Wei Yang
2015-08-06 4:35 ` Gavin Shan
2015-08-06 6:10 ` Alexey Kardashevskiy
2015-08-06 6:57 ` Gavin Shan
2015-08-06 7:47 ` Alexey Kardashevskiy
2015-08-06 11:07 ` Gavin Shan
2015-08-06 14:13 ` Wei Yang
2015-08-07 1:24 ` Alexey Kardashevskiy
2015-08-06 14:10 ` Wei Yang
2015-08-07 1:20 ` Gavin Shan
2015-08-07 2:24 ` Wei Yang
2015-08-07 3:50 ` Gavin Shan
2015-08-07 7:14 ` Alexey Kardashevskiy [this message]
2015-08-10 1:40 ` Wei Yang
2015-08-05 1:24 ` [PATCH V2 2/6] powerpc/powernv: simplify the calculation of iov resource Wei Yang
2015-08-06 4:51 ` Gavin Shan
2015-08-06 9:00 ` Alexey Kardashevskiy
2015-08-06 9:41 ` Wei Yang
2015-08-06 10:15 ` Alexey Kardashevskiy
2015-08-07 1:36 ` Wei Yang
2015-08-06 13:49 ` Wei Yang
2015-08-07 1:08 ` Gavin Shan
2015-08-05 1:25 ` [PATCH V2 3/6] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR Wei Yang
2015-08-06 5:20 ` Gavin Shan
2015-08-06 9:36 ` Wei Yang
2015-08-06 10:07 ` Gavin Shan
2015-08-07 1:48 ` Wei Yang
2015-08-07 8:13 ` Alexey Kardashevskiy
2015-08-06 10:04 ` Alexey Kardashevskiy
2015-08-07 2:01 ` Wei Yang
2015-08-07 8:59 ` Alexey Kardashevskiy
2015-08-10 1:48 ` Wei Yang
2015-08-05 1:25 ` [PATCH V2 4/6] powerpc/powernv: replace the hard coded boundary with gate Wei Yang
2015-08-06 5:26 ` Gavin Shan
2015-08-07 9:11 ` Alexey Kardashevskiy
2015-08-05 1:25 ` [PATCH V2 5/6] powerpc/powernv: boundary the total vf bar size instead of the individual one Wei Yang
2015-08-06 5:28 ` Gavin Shan
2015-08-06 14:03 ` Wei Yang
2015-08-07 1:23 ` Gavin Shan
2015-08-07 2:25 ` Wei Yang
2015-08-05 1:25 ` [PATCH V2 6/6] powerpc/powernv: allocate discrete PE# when using M64 BAR in Single PE mode Wei Yang
2015-08-06 5:36 ` Gavin Shan
2015-08-06 13:41 ` Wei Yang
2015-08-07 1:36 ` Gavin Shan
2015-08-07 2:33 ` Wei Yang
2015-08-07 3:43 ` Gavin Shan
2015-08-07 5:44 ` Wei Yang
2015-08-07 5:54 ` Gavin Shan
2015-08-07 6:25 ` Wei Yang
2015-08-07 10:00 ` Alexey Kardashevskiy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55C45AE1.3040404@ozlabs.ru \
--to=aik@ozlabs$(echo .)ru \
--cc=benh@kernel$(echo .)crashing.org \
--cc=gwshan@linux$(echo .)vnet.ibm.com \
--cc=linuxppc-dev@ozlabs$(echo .)org \
--cc=weiyang@linux$(echo .)vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox