From: Alexey Kardashevskiy <aik@ozlabs•ru>
To: Wei Yang <weiyang@linux•vnet.ibm.com>,
Gavin Shan <gwshan@linux•vnet.ibm.com>
Cc: benh@kernel•crashing.org, linuxppc-dev@ozlabs•org
Subject: Re: [PATCH V2 3/6] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR
Date: Fri, 7 Aug 2015 18:13:14 +1000 [thread overview]
Message-ID: <55C4689A.6020805@ozlabs.ru> (raw)
In-Reply-To: <20150807014818.GC8292@richard>
On 08/07/2015 11:48 AM, Wei Yang wrote:
> On Thu, Aug 06, 2015 at 08:07:01PM +1000, Gavin Shan wrote:
>> On Thu, Aug 06, 2015 at 05:36:02PM +0800, Wei Yang wrote:
>>> On Thu, Aug 06, 2015 at 03:20:25PM +1000, Gavin Shan wrote:
>>>> On Wed, Aug 05, 2015 at 09:25:00AM +0800, Wei Yang wrote:
>>>>> In current implementation, when VF BAR is bigger than 64MB, it uses 4 M64
>>>>> BAR in Single PE mode to cover the number of VFs required to be enabled.
>>>>> By doing so, several VFs would be in one VF Group and leads to interference
>>>>> between VFs in the same group.
>>>>>
>>>>> This patch changes the design by using one M64 BAR in Single PE mode for
>>>>> one VF BAR. This gives absolute isolation for VFs.
>>>>>
>>>>> Signed-off-by: Wei Yang <weiyang@linux•vnet.ibm.com>
>>>>> ---
>>>>> arch/powerpc/include/asm/pci-bridge.h | 5 +-
>>>>> arch/powerpc/platforms/powernv/pci-ioda.c | 180 ++++++++++++-----------------
>>>>> 2 files changed, 76 insertions(+), 109 deletions(-)
>>>>>
>>>>> diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
>>>>> index 712add5..8aeba4c 100644
>>>>> --- a/arch/powerpc/include/asm/pci-bridge.h
>>>>> +++ b/arch/powerpc/include/asm/pci-bridge.h
>>>>> @@ -214,10 +214,9 @@ struct pci_dn {
>>>>> u16 vfs_expanded; /* number of VFs IOV BAR expanded */
>>>>> u16 num_vfs; /* number of VFs enabled*/
>>>>> int offset; /* PE# for the first VF PE */
>>>>> -#define M64_PER_IOV 4
>>>>> - int m64_per_iov;
>>>>> + bool m64_single_mode; /* Use M64 BAR in Single Mode */
>>>>> #define IODA_INVALID_M64 (-1)
>>>>> - int m64_wins[PCI_SRIOV_NUM_BARS][M64_PER_IOV];
>>>>> + int (*m64_map)[PCI_SRIOV_NUM_BARS];
>>>>
>>>> It can be explicit? For example:
>>>>
>>>> int *m64_map;
>>>>
>>>> /* Initialization */
>>>> size_t size = sizeof(*pdn->m64_map) * PCI_SRIOV_NUM_BARS * num_of_max_VFs;
>>>> pdn->m64_map = kmalloc(size, GFP_KERNEL);
>>>> for (i = 0; i < PCI_SRIOV_NUM_BARS; i++)
>>>> for (j = 0; j < num_of_max_VFs; j++)
>>>> pdn->m64_map[i * PCI_SRIOV_NUM_BARS + j] = PNV_INVALID_M64;
>>>>
>>>> /* Destroy */
>>>> int step = 1;
>>>>
>>>> if (!pdn->m64_single_mode)
>>>> step = phb->ioda.total_pe;
>>>> for (i = 0; i < PCI_SRIOV_NUM_BARS * num_of_max_VFs; i += step)
>>>> if (pdn->m64_map[i] == PNV_INVALID_M64)
>>>> continue;
>>>>
>>>> /* Unmap the window */
>>>>
>>>
>>> The m64_map is a pointer to an array with 6 elements, which represents the 6
>>> M64 BAR index for the 6 VF BARs.
>>>
>>> When we use Shared Mode, one array is allocated. The six elements
>>> represents the six M64 BAR(at most) used to map the whole IOV BAR.
>>>
>>> When we use Single Mode, num_vfs array is allocate. Each array represents
>>> the map between one VF's BAR and M64 BAR index.
>>>
>>> During the map and un-map, M64 BAR is assigned one by one in VF BAR's order.
>>> So I think the code is explicit.
>>>
>>> In your code, you allocate a big one dimension array to hold the M64 BAR
>>> index. It works, while I don't think this is more explicit than original code.
>>>
>>
>> When M64 is in Single Mode, array with (num_vfs * 6) entries is allocated
>> because every VF BAR (6 at most) will have one corresponding PHB M64 BAR.
>> Anything I missed?
>>
>> The point in my code is you needn't worry about the mode (single vs shared)
>> As I said, not too much memory wasted. However, it's up to you.
>>
>
> If we don't want to save some memory, how about just define them static
> instead of dynamically allocate?
I like static and you can make it uint8_t[][] (or char[][]) as these
indexes are not going to be bigger than 255 anyway.
>> I'm not fan of "int (*m64_map)[PCI_SRIOV_NUM_BARS]". Instead, you can replace
>> it with "int *m64_map" and calculate its size using following formula:
>>
>> sizeof(*pdn->m64_map) * PCI_SRIOV_NUM_BARS;
>>
>> sizeof(*pdn->m64_map) * PCI_SRIOV_NUM_BARS * num_vfs;
--
Alexey
next prev parent reply other threads:[~2015-08-07 8:13 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-29 7:22 [PATCH] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR Wei Yang
2015-07-30 1:15 ` Gavin Shan
2015-07-30 5:43 ` Wei Yang
2015-07-31 0:13 ` Gavin Shan
2015-07-31 2:01 ` Wei Yang
2015-08-05 1:24 ` [PATCH V2 0/6] Redesign SR-IOV on PowerNV Wei Yang
2015-08-05 1:24 ` [PATCH V2 1/6] powerpc/powernv: don't enable SRIOV when VF BAR contains non M64 BAR Wei Yang
2015-08-06 4:35 ` Gavin Shan
2015-08-06 6:10 ` Alexey Kardashevskiy
2015-08-06 6:57 ` Gavin Shan
2015-08-06 7:47 ` Alexey Kardashevskiy
2015-08-06 11:07 ` Gavin Shan
2015-08-06 14:13 ` Wei Yang
2015-08-07 1:24 ` Alexey Kardashevskiy
2015-08-06 14:10 ` Wei Yang
2015-08-07 1:20 ` Gavin Shan
2015-08-07 2:24 ` Wei Yang
2015-08-07 3:50 ` Gavin Shan
2015-08-07 7:14 ` Alexey Kardashevskiy
2015-08-10 1:40 ` Wei Yang
2015-08-05 1:24 ` [PATCH V2 2/6] powerpc/powernv: simplify the calculation of iov resource Wei Yang
2015-08-06 4:51 ` Gavin Shan
2015-08-06 9:00 ` Alexey Kardashevskiy
2015-08-06 9:41 ` Wei Yang
2015-08-06 10:15 ` Alexey Kardashevskiy
2015-08-07 1:36 ` Wei Yang
2015-08-06 13:49 ` Wei Yang
2015-08-07 1:08 ` Gavin Shan
2015-08-05 1:25 ` [PATCH V2 3/6] powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR Wei Yang
2015-08-06 5:20 ` Gavin Shan
2015-08-06 9:36 ` Wei Yang
2015-08-06 10:07 ` Gavin Shan
2015-08-07 1:48 ` Wei Yang
2015-08-07 8:13 ` Alexey Kardashevskiy [this message]
2015-08-06 10:04 ` Alexey Kardashevskiy
2015-08-07 2:01 ` Wei Yang
2015-08-07 8:59 ` Alexey Kardashevskiy
2015-08-10 1:48 ` Wei Yang
2015-08-05 1:25 ` [PATCH V2 4/6] powerpc/powernv: replace the hard coded boundary with gate Wei Yang
2015-08-06 5:26 ` Gavin Shan
2015-08-07 9:11 ` Alexey Kardashevskiy
2015-08-05 1:25 ` [PATCH V2 5/6] powerpc/powernv: boundary the total vf bar size instead of the individual one Wei Yang
2015-08-06 5:28 ` Gavin Shan
2015-08-06 14:03 ` Wei Yang
2015-08-07 1:23 ` Gavin Shan
2015-08-07 2:25 ` Wei Yang
2015-08-05 1:25 ` [PATCH V2 6/6] powerpc/powernv: allocate discrete PE# when using M64 BAR in Single PE mode Wei Yang
2015-08-06 5:36 ` Gavin Shan
2015-08-06 13:41 ` Wei Yang
2015-08-07 1:36 ` Gavin Shan
2015-08-07 2:33 ` Wei Yang
2015-08-07 3:43 ` Gavin Shan
2015-08-07 5:44 ` Wei Yang
2015-08-07 5:54 ` Gavin Shan
2015-08-07 6:25 ` Wei Yang
2015-08-07 10:00 ` Alexey Kardashevskiy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55C4689A.6020805@ozlabs.ru \
--to=aik@ozlabs$(echo .)ru \
--cc=benh@kernel$(echo .)crashing.org \
--cc=gwshan@linux$(echo .)vnet.ibm.com \
--cc=linuxppc-dev@ozlabs$(echo .)org \
--cc=weiyang@linux$(echo .)vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox