From: Michael Ellerman <mpe@ellerman•id.au>
To: "Cédric Le Goater" <clg@kaod•org>,
linuxppc-dev@lists•ozlabs.org,
"Nicholas Piggin" <npiggin@gmail•com>
Subject: Re: [PATCH 11/11] powerpc/smp: Add a doorbell=off kernel parameter
Date: Thu, 18 Nov 2021 20:24:48 +1100 [thread overview]
Message-ID: <87sfvtlstb.fsf@mpe.ellerman.id.au> (raw)
In-Reply-To: <7341ab0d-c12b-6d5f-76d3-5927a2734f02@kaod.org>
Cédric Le Goater <clg@kaod•org> writes:
> On 11/11/21 11:41, Michael Ellerman wrote:
>> Cédric Le Goater <clg@kaod•org> writes:
>>> On processors with a XIVE interrupt controller (POWER9 and above), the
>>> kernel can use either doorbells or XIVE to generate CPU IPIs. Sending
>>> doorbell is generally preferred to using the XIVE IC because it is
>>> faster. There are cases where we want to avoid doorbells and use XIVE
>>> only, for debug or performance. Only useful on POWER9 and above.
>>
>> How much do we want this?
>
> Yes. Thanks for asking. It is a recent need.
>
> Here is some background I should have added in the first place. May be
> for a v2.
>
> We have different ways of doing IPIs on POWER9 and above processors,
> depending on the platform and the underlying hypervisor.
>
> - PowerNV uses global doorbells
>
> - pSeries/KVM uses XIVE only because local doorbells are not
> efficient, as there are emulated in the KVM hypervisor
>
> - pSeries/PowerVM uses XIVE for remote cores and local doorbells for
> threads on same core (SMT4 or 8)
>
> This recent commit 5b06d1679f2f ("powerpc/pseries: Use doorbells even
> if XIVE is available") introduced the optimization for PowerVM and
> commit 107c55005fbd ("powerpc/pseries: Add KVM guest doorbell
> restrictions") restricted the optimization.
>
> We would like a way to turn off the optimization.
Just for test/debug though?
>> Kernel command line args are a bit of a pain, they tend to be poorly
>> tested, because someone has to explicitly enable them at boot time,
>> and then reboot to test the other case.
>
> True. The "xive=off" parameter was poorly tested initially.
>
>> When would we want to enable this?
>
> For bring-up, for debug, for tests. I have been using a similar switch
> to compare the XIVE interrupt controller performance with doorbells on
> POWER9 and P0WER10.
>
> A new need arises with PowerVM, some configurations will behave as KVM
> (local doorbell are unsupported) and the doorbell=off parameter is a
> simple way to handle this case today.
That's the first I've heard of that, what PowerVM configurations have
non-working doorbells?
>> Can we make the kernel smarter about when to use doorbells and make
>> it automated?
>
> I don't think we want to probe all IPI methods to detect how well
> local doorbells are supported on the platform. Do we ?
We don't *want to*, but sounds like we might have to if they are not
working in some configurations as you mentioned above.
cheers
next prev parent reply other threads:[~2021-11-18 9:25 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-05 10:26 [PATCH 00/11] powerpc/xive: Improve diagnostic and activate StoreEOI on P10 PowerNV Cédric Le Goater
2021-11-05 10:26 ` [PATCH 01/11] powerpc/xive: Replace pr_devel() by pr_debug() to ease debug Cédric Le Goater
2021-11-05 10:26 ` [PATCH 02/11] powerpc/xive: Introduce an helper to print out interrupt characteristics Cédric Le Goater
2021-11-05 10:26 ` [PATCH 03/11] powerpc/xive: Activate StoreEOI on P10 Cédric Le Goater
2021-11-05 10:26 ` [PATCH 04/11] powerpc/xive: Introduce xive_core_debugfs_create() Cédric Le Goater
2021-11-18 9:21 ` Michael Ellerman
2021-11-18 15:22 ` Cédric Le Goater
2021-11-05 10:26 ` [PATCH 05/11] powerpc/xive: Change the debugfs file 'xive' into a directory Cédric Le Goater
2021-11-05 10:26 ` [PATCH 06/11] powerpc/xive: Rename the 'cpus' debugfs file to 'ipis' Cédric Le Goater
2021-11-05 10:26 ` [PATCH 07/11] powerpc/xive: Add a debugfs file to dump EQs Cédric Le Goater
2021-11-05 10:26 ` [PATCH 08/11] powerpc/xive: Add a debugfs toggle for StoreEOI Cédric Le Goater
2021-11-05 10:26 ` [PATCH 09/11] powerpc/xive: Add a kernel parameter " Cédric Le Goater
2021-11-05 10:26 ` [PATCH 10/11] powerpc/xive: Add a debugfs toggle for save-restore Cédric Le Goater
2021-11-05 10:26 ` [PATCH 11/11] powerpc/smp: Add a doorbell=off kernel parameter Cédric Le Goater
2021-11-11 10:41 ` Michael Ellerman
2021-11-11 16:01 ` Cédric Le Goater
2021-11-18 9:24 ` Michael Ellerman [this message]
2021-11-18 15:26 ` Cédric Le Goater
2021-11-25 9:36 ` [PATCH 00/11] powerpc/xive: Improve diagnostic and activate StoreEOI on P10 PowerNV Michael Ellerman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87sfvtlstb.fsf@mpe.ellerman.id.au \
--to=mpe@ellerman$(echo .)id.au \
--cc=clg@kaod$(echo .)org \
--cc=linuxppc-dev@lists$(echo .)ozlabs.org \
--cc=npiggin@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox