public inbox for linuxppc-dev@ozlabs.org 
 help / color / mirror / Atom feed

From: Alistair Popple <alistair@popple•id.au>
To: linuxppc-dev@lists•ozlabs.org
Cc: mpe@ellerman•id.au, mhairgrove@nvidia•com, arbab@linux•ibm.com
Subject: Re: [PATCH 1/2] powernv/npu: Do a PID GPU TLB flush when invalidating a large address range
Date: Fri, 20 Apr 2018 13:51:58 +1000	[thread overview]
Message-ID: <3716532.LRmDtzLHtY@new-mexico> (raw)
In-Reply-To: <20180417091129.23069-1-alistair@popple.id.au>

Sorry, forgot to include:

Fixes: 1ab66d1fbada ("powerpc/powernv: Introduce address translation services for Nvlink2")

Thanks

On Tuesday, 17 April 2018 7:11:28 PM AEST Alistair Popple wrote:
> The NPU has a limited number of address translation shootdown (ATSD)
> registers and the GPU has limited bandwidth to process ATSDs. This can
> result in contention of ATSD registers leading to soft lockups on some
> threads, particularly when invalidating a large address range in
> pnv_npu2_mn_invalidate_range().
> 
> At some threshold it becomes more efficient to flush the entire GPU TLB for
> the given MM context (PID) than individually flushing each address in the
> range. This patch will result in ranges greater than 2MB being converted
> from 32+ ATSDs into a single ATSD which will flush the TLB for the given
> PID on each GPU.
> 
> Signed-off-by: Alistair Popple <alistair@popple•id.au>
> ---
>  arch/powerpc/platforms/powernv/npu-dma.c | 23 +++++++++++++++++++----
>  1 file changed, 19 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
> index 94801d8e7894..dc34662e9df9 100644
> --- a/arch/powerpc/platforms/powernv/npu-dma.c
> +++ b/arch/powerpc/platforms/powernv/npu-dma.c
> @@ -40,6 +40,13 @@
>  DEFINE_SPINLOCK(npu_context_lock);
>  
>  /*
> + * When an address shootdown range exceeds this threshold we invalidate the
> + * entire TLB on the GPU for the given PID rather than each specific address in
> + * the range.
> + */
> +#define ATSD_THRESHOLD (2*1024*1024)
> +
> +/*
>   * Other types of TCE cache invalidation are not functional in the
>   * hardware.
>   */
> @@ -675,11 +682,19 @@ static void pnv_npu2_mn_invalidate_range(struct mmu_notifier *mn,
>  	struct npu_context *npu_context = mn_to_npu_context(mn);
>  	unsigned long address;
>  
> -	for (address = start; address < end; address += PAGE_SIZE)
> -		mmio_invalidate(npu_context, 1, address, false);
> +	if (end - start > ATSD_THRESHOLD) {
> +		/*
> +		 * Just invalidate the entire PID if the address range is too
> +		 * large.
> +		 */
> +		mmio_invalidate(npu_context, 0, 0, true);
> +	} else {
> +		for (address = start; address < end; address += PAGE_SIZE)
> +			mmio_invalidate(npu_context, 1, address, false);
>  
> -	/* Do the flush only on the final addess == end */
> -	mmio_invalidate(npu_context, 1, address, true);
> +		/* Do the flush only on the final addess == end */
> +		mmio_invalidate(npu_context, 1, address, true);
> +	}
>  }
>  
>  static const struct mmu_notifier_ops nv_nmmu_notifier_ops = {
> 

next prev parent reply	other threads:[~2018-04-20  3:51 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-17  9:11 [PATCH 1/2] powernv/npu: Do a PID GPU TLB flush when invalidating a large address range Alistair Popple
2018-04-17  9:11 ` [PATCH 2/2] powernv/npu: Add a debugfs setting to change ATSD threshold Alistair Popple
2018-04-17 21:45   ` Balbir Singh
2018-07-23 15:11   ` [2/2] " Michael Ellerman
2018-04-17  9:17 ` [PATCH 1/2] powernv/npu: Do a PID GPU TLB flush when invalidating a large address range Balbir Singh
2018-04-17 22:25   ` Balbir Singh
2018-04-20  3:51 ` Alistair Popple [this message]
2018-04-24  3:48 ` [1/2] " Michael Ellerman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3716532.LRmDtzLHtY@new-mexico \
    --to=alistair@popple$(echo .)id.au \
    --cc=arbab@linux$(echo .)ibm.com \
    --cc=linuxppc-dev@lists$(echo .)ozlabs.org \
    --cc=mhairgrove@nvidia$(echo .)com \
    --cc=mpe@ellerman$(echo .)id.au \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox