From: marc.zyngier@arm•com (Marc Zyngier)
To: linux-arm-kernel@lists•infradead.org
Subject: [RFC PATCH 0/6] ARM64: KVM: PMU infrastructure support
Date: Thu, 27 Nov 2014 11:06:39 +0000 [thread overview]
Message-ID: <547705BF.5040609@arm.com> (raw)
In-Reply-To: <CAAhSdy3US4H9DZLDmwawR6D22GUpQ8H5NqrZ8XvYJjhsnPzzRQ@mail.gmail.com>
On 27/11/14 10:54, Anup Patel wrote:
> On Thu, Nov 27, 2014 at 4:10 PM, Marc Zyngier <marc.zyngier@arm•com> wrote:
>> On 27/11/14 10:22, Anup Patel wrote:
>>> On Tue, Nov 25, 2014 at 7:12 PM, Christoffer Dall
>>> <christoffer.dall@linaro•org> wrote:
>>>> On Tue, Nov 25, 2014 at 06:17:03PM +0530, Anup Patel wrote:
>>>>> Hi Christoffer,
>>>>>
>>>>> On Mon, Nov 24, 2014 at 8:07 PM, Christoffer Dall
>>>>> <christoffer.dall@linaro•org> wrote:
>>>>>> On Mon, Nov 24, 2014 at 02:14:48PM +0530, Anup Patel wrote:
>>>>>>> On Fri, Nov 21, 2014 at 5:19 PM, Christoffer Dall
>>>>>>> <christoffer.dall@linaro•org> wrote:
>>>>>>>> On Fri, Nov 21, 2014 at 04:06:05PM +0530, Anup Patel wrote:
>>>>>>>>> Hi Christoffer,
>>>>>>>>>
>>>>>>>>> On Fri, Nov 21, 2014 at 3:29 PM, Christoffer Dall
>>>>>>>>> <christoffer.dall@linaro•org> wrote:
>>>>>>>>>> On Thu, Nov 20, 2014 at 08:17:32PM +0530, Anup Patel wrote:
>>>>>>>>>>> On Wed, Nov 19, 2014 at 8:59 PM, Christoffer Dall
>>>>>>>>>>> <christoffer.dall@linaro•org> wrote:
>>>>>>>>>>>> On Tue, Nov 11, 2014 at 02:48:25PM +0530, Anup Patel wrote:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have second thoughts about rebasing KVM PMU patches
>>>>>>>>>>>>> to Marc's irq-forwarding patches.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The PMU IRQs (when virtualized by KVM) are not exactly
>>>>>>>>>>>>> forwarded IRQs because they are shared between Host
>>>>>>>>>>>>> and Guest.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scenario1
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> We might have perf running on Host and no KVM guest
>>>>>>>>>>>>> running. In this scenario, we wont get interrupts on Host
>>>>>>>>>>>>> because the kvm_pmu_hyp_init() (similar to the function
>>>>>>>>>>>>> kvm_timer_hyp_init() of Marc's IRQ-forwarding
>>>>>>>>>>>>> implementation) has put all host PMU IRQs in forwarding
>>>>>>>>>>>>> mode.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The only way solve this problem is to not set forwarding
>>>>>>>>>>>>> mode for PMU IRQs in kvm_pmu_hyp_init() and instead
>>>>>>>>>>>>> have special routines to turn on and turn off the forwarding
>>>>>>>>>>>>> mode of PMU IRQs. These routines will be called from
>>>>>>>>>>>>> kvm_arch_vcpu_ioctl_run() for toggling the PMU IRQ
>>>>>>>>>>>>> forwarding state.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Scenario2
>>>>>>>>>>>>> -------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> We might have perf running on Host and Guest simultaneously
>>>>>>>>>>>>> which means it is quite likely that PMU HW trigger IRQ meant
>>>>>>>>>>>>> for Host between "ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);"
>>>>>>>>>>>>> and "kvm_pmu_sync_hwstate(vcpu);" (similar to timer sync routine
>>>>>>>>>>>>> of Marc's patchset which is called before local_irq_enable()).
>>>>>>>>>>>>>
>>>>>>>>>>>>> In this scenario, the updated kvm_pmu_sync_hwstate(vcpu)
>>>>>>>>>>>>> will accidentally forward IRQ meant for Host to Guest unless
>>>>>>>>>>>>> we put additional checks to inspect VCPU PMU state.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am I missing any detail about IRQ forwarding for above
>>>>>>>>>>>>> scenarios?
>>>>>>>>>>>>>
>>>>>>>>>>>> Hi Anup,
>>>>>>>>>>>
>>>>>>>>>>> Hi Christoffer,
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I briefly discussed this with Marc. What I don't understand is how it
>>>>>>>>>>>> would be possible to get an interrupt for the host while running the
>>>>>>>>>>>> guest?
>>>>>>>>>>>>
>>>>>>>>>>>> The rationale behind my question is that whenever you're running the
>>>>>>>>>>>> guest, the PMU should be programmed exclusively with guest state, and
>>>>>>>>>>>> since the PMU is per core, any interrupts should be for the guest, where
>>>>>>>>>>>> it would always be pending.
>>>>>>>>>>>
>>>>>>>>>>> Yes, thats right PMU is programmed exclusively for guest when
>>>>>>>>>>> guest is running and for host when host is running.
>>>>>>>>>>>
>>>>>>>>>>> Let us assume a situation (Scenario2 mentioned previously)
>>>>>>>>>>> where both host and guest are using PMU. When the guest is
>>>>>>>>>>> running we come back to host mode due to variety of reasons
>>>>>>>>>>> (stage2 fault, guest IO, regular host interrupt, host interrupt
>>>>>>>>>>> meant for guest, ....) which means we will return from the
>>>>>>>>>>> "ret = kvm_call_hyp(__kvm_vcpu_run, vcpu);" statement in the
>>>>>>>>>>> kvm_arch_vcpu_ioctl_run() function with local IRQs disabled.
>>>>>>>>>>> At this point we would have restored back host PMU context and
>>>>>>>>>>> any PMU counter used by host can trigger PMU overflow interrup
>>>>>>>>>>> for host. Now we will be having "kvm_pmu_sync_hwstate(vcpu);"
>>>>>>>>>>> in the kvm_arch_vcpu_ioctl_run() function (similar to the
>>>>>>>>>>> kvm_timer_sync_hwstate() of Marc's IRQ forwarding patchset)
>>>>>>>>>>> which will try to detect PMU irq forwarding state in GIC hence it
>>>>>>>>>>> can accidentally discover PMU irq pending for guest while this
>>>>>>>>>>> PMU irq is actually meant for host.
>>>>>>>>>>>
>>>>>>>>>>> This above mentioned situation does not happen for timer
>>>>>>>>>>> because virtual timer interrupts are exclusively used for guest.
>>>>>>>>>>> The exclusive use of virtual timer interrupt for guest ensures that
>>>>>>>>>>> the function kvm_timer_sync_hwstate() will always see correct
>>>>>>>>>>> state of virtual timer IRQ from GIC.
>>>>>>>>>>>
>>>>>>>>>> I'm not quite following.
>>>>>>>>>>
>>>>>>>>>> When you call kvm_pmu_sync_hwstate(vcpu) in the non-preemtible section,
>>>>>>>>>> you would (1) capture the active state of the IRQ pertaining to the
>>>>>>>>>> guest and (2) deactive the IRQ on the host, then (3) switch the state of
>>>>>>>>>> the PMU to the host state, and finally (4) re-enable IRQs on the CPU
>>>>>>>>>> you're running on.
>>>>>>>>>>
>>>>>>>>>> If the host PMU state restored in (3) causes the PMU to raise an
>>>>>>>>>> interrupt, you'll take an interrupt after (4), which is for the host,
>>>>>>>>>> and you'll handle it on the host.
>>>>>>>>>>
>>>>>>>>> We only switch PMU state in assembly code using
>>>>>>>>> kvm_call_hyp(__kvm_vcpu_run, vcpu)
>>>>>>>>> so whenever we are in kvm_arch_vcpu_ioctl_run() (i.e. host mode)
>>>>>>>>> the current hardware PMU state is for host. This means whenever
>>>>>>>>> we are in host mode the host PMU can change state of PMU IRQ
>>>>>>>>> in GIC even if local IRQs are disabled.
>>>>>>>>>
>>>>>>>>> Whenever we inspect active state of PMU IRQ in the
>>>>>>>>> kvm_pmu_sync_hwstate() function using irq_get_fwd_state() API.
>>>>>>>>> Here we are not guaranteed that IRQ forward state returned by the
>>>>>>>>> irq_get_fwd_state() API is for guest only.
>>>>>>>>>
>>>>>>>>> The above situation does not manifest for virtual timer because
>>>>>>>>> virtual timer registers are exclusively accessed by Guest and
>>>>>>>>> virtual timer interrupt is only for Guest (never used by Host).
>>>>>>>>>
>>>>>>>>>> Whenever you schedule the guest VCPU again, you'll (a) disable
>>>>>>>>>> interrupts on the CPU, (b) restore the active state of the IRQ for the
>>>>>>>>>> guest, (c) restore the guest PMU state, (d) switch to the guest with
>>>>>>>>>> IRQs enabled on the CPU (potentially).
>>>>>>>>>
>>>>>>>>> Here too, while we are between step (a) and step (b) the PMU HW
>>>>>>>>> context is for host and any PMU counter can overflow. The step (b)
>>>>>>>>> can actually override the PMU IRQ meant for Host.
>>>>>>>>>
>>>>>>>> Can you not simply switch the state from C-code after capturing the IRQ
>>>>>>>> state then? Everything should be accessible from EL1, right?
>>>>>>>
>>>>>>> Yes, I think that would be the only option. This also means I will need
>>>>>>> to re-implement context switching for doing it in C-code.
>>>>>>
>>>>>> Yes, you'd add some inline assembly in the C-code to access the
>>>>>> registers I guess. Only thing I thought about after writing my original
>>>>>> mail is whether you'll be counting events while context-swtiching and
>>>>>> running on the host, which you actually don't want to. Not sure if
>>>>>> there's a better way to avoid that.
>>>>>>
>>>>>>>
>>>>>>> What about the scenario1 which I had mentioned?
>>>>>>>
>>>>>>
>>>>>> You have to consider enabling/disabling forwarding and setting/clearing
>>>>>> the active state is part of the guest PMU state and all of it has to be
>>>>>> context-switched.
>>>>>
>>>>> I found one more issue.
>>>>>
>>>>> If PMU irq is PPI then enabling/disabling forwarding will not
>>>>> work because irqd_set_irq_forwarded() function takes irq_data
>>>>> as argument which is member of irq_desc and irq_desc for PPIs
>>>>> is not per_cpu. This means we cannot call irqd_set_irq_forwarded()
>>>>> simultaneously from different host CPUs.
>>>>>
>>>
>>> Hi Marc,
>>>
>>>> I'll let Marc answer this one and if this still applies to his view of
>>>> how the next version of the forwarding series will look like.
>>
>> I'm looking at it at the moment.
>>
>> I'm inclined to say that we should fix the forwarding code to allow
>> individual PPIs to be forwarded. This is a bit harder than what we're
>> doing at the moment, but that's possible.
>>
>> Of course, that complicates the code a bit, as we have to make sure
>> we're not premptable at that time.
>>
>> What do you think?
>
> Currently, irqd_set_irq_forwarded() is lockless.
>
> It would be great if we can update irqd_set_irq_forwarded() for PPIs
> such that it remains irqd_set_irq_forwarded() lockless so that we
> dont have much overhead when we enable/disable forwarding
> state.
We probably need a separate API anyway, as you want to be able to
provide a cpumask to configure this. We can refine this as we go, and I
wouldn't worry about overhead just yet.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
next prev parent reply other threads:[~2014-11-27 11:06 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-08-05 9:24 [RFC PATCH 0/6] ARM64: KVM: PMU infrastructure support Anup Patel
2014-08-05 9:24 ` [RFC PATCH 1/6] ARM64: Move PMU register related defines to asm/pmu.h Anup Patel
2014-08-05 9:24 ` [RFC PATCH 2/6] ARM64: perf: Re-enable overflow interrupt from interrupt handler Anup Patel
2014-08-06 14:24 ` Will Deacon
2014-08-07 9:03 ` Anup Patel
2014-08-07 9:06 ` Will Deacon
2014-08-05 9:24 ` [RFC PATCH 3/6] ARM: " Anup Patel
2014-08-05 9:24 ` [RFC PATCH 4/6] ARM/ARM64: KVM: Add common code PMU IRQ routing Anup Patel
2014-08-05 9:24 ` [RFC PATCH 5/6] ARM64: KVM: Implement full context switch of PMU registers Anup Patel
2014-08-05 9:24 ` [RFC PATCH 6/6] ARM64: KVM: Upgrade to lazy " Anup Patel
2014-08-05 9:32 ` [RFC PATCH 0/6] ARM64: KVM: PMU infrastructure support Anup Patel
2014-08-05 9:35 ` Anup Patel
2014-11-07 20:23 ` Christoffer Dall
2014-11-07 20:25 ` Christoffer Dall
2014-11-08 9:36 ` Anup Patel
2014-11-08 12:39 ` Christoffer Dall
2014-11-11 9:18 ` Anup Patel
2014-11-18 3:24 ` Anup Patel
2014-11-19 15:29 ` Christoffer Dall
2014-11-20 14:47 ` Anup Patel
2014-11-21 9:59 ` Christoffer Dall
2014-11-21 10:36 ` Anup Patel
2014-11-21 11:49 ` Christoffer Dall
2014-11-24 8:44 ` Anup Patel
2014-11-24 14:37 ` Christoffer Dall
2014-11-25 12:47 ` Anup Patel
2014-11-25 13:42 ` Christoffer Dall
2014-11-27 10:22 ` Anup Patel
2014-11-27 10:40 ` Marc Zyngier
2014-11-27 10:54 ` Anup Patel
2014-11-27 11:06 ` Marc Zyngier [this message]
2014-12-30 5:49 ` Anup Patel
2015-01-08 4:02 ` Anup Patel
2015-01-11 19:11 ` Christoffer Dall
2015-01-12 4:19 ` Anup Patel
2015-02-15 15:33 ` Christoffer Dall
2015-02-16 12:16 ` Anup Patel
2015-02-16 12:23 ` Christoffer Dall
2015-01-14 4:28 ` Anup Patel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=547705BF.5040609@arm.com \
--to=marc.zyngier@arm$(echo .)com \
--cc=linux-arm-kernel@lists$(echo .)infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox