From: will.deacon@arm•com (Will Deacon)
To: linux-arm-kernel@lists•infradead.org
Subject: [PATCH] ARM: mm: dma: Update coherent streaming apis with missing memory barrier
Date: Thu, 24 Apr 2014 15:09:13 +0100 [thread overview]
Message-ID: <20140424140913.GB14110@arm.com> (raw)
In-Reply-To: <535913D4.6020401@ti.com>
On Thu, Apr 24, 2014 at 02:38:28PM +0100, Santosh Shilimkar wrote:
> On Thursday 24 April 2014 07:21 AM, Will Deacon wrote:
> > On Thu, Apr 24, 2014 at 12:15:47PM +0100, Russell King - ARM Linux wrote:
> >> Yes, the hardware /is/ broken, but if you want to get it working in a
> >> way that's acceptable in upstream kernels, adding that barrier to rmb()
> >> is probably the only acceptable solution - especially if you have other
> >> stuff going in between the rmb() and the DMA unmap.
> >
> > The problem then is that the additional barrier may well require
> > bus-specific knowledge and access to parts of bus driver code which we can't
> > use inside the rmb() macro. To solve this properly, the bus topology topic
> > once again rears its ugly head, as I think you'd need a callback from the
> > device driver to the bus on which it resides in order to provide the
> > appropriate barrier (which isn't something that can be done sanely for
> > the platform_bus).
> >
> Not exactly against the bus notifier point but we can't afford to have such
> notifier calls in hot paths. Especially gigabit network drivers per packet
> processing paths where even 10 cycle cost makes huge impact on the throughput.
I don't think anybody is suggesting that you do this per-packet. This is a
per-DMA-transfer barrier, which is required anyway. The details of the
barrier are what varies, and are likely bus-specific.
> Interconnect barriers are really needed for completion. I think CPUs within at
> least same clusters will be ordered with rmb(). But same is not true when you
> have multiple clusters and then further down coherent interconnect comes into
> picture where all other non-CPU coherent masters are participating.
You're making a lot of rash generalisations here. The architected barrier
instructions as used by Linux will work perfectly well within the
inner-shareable domain. That means you don't need to worry about
multiple-clusters of CPUs.
However, you can't read *anything* into how a barrier instruction executed
on the CPU affects writes from another master; there is inherently a race
there which must be dealt with by either the external master or some
implementation-specific action by the CPU. This is the real problem.
> If rmb() has to reach all the way to coherent masters(non-CPU), then I suspect
> most of the ARM coherent architectures are broken. If you take any typical SOC,
> ARM CPUs are bolted with other coherent masters at AXI boundary or may be with
> ACP interfaces. At this level rmb() isn't good enough and you at least
> need a dsb() for completion.
An rmb() expands to dsb, neither of which give you anything in this scenario
as described by the architecture.
> So in my view unless and until you have features like DVM in hardware, dsb() is
> needed to guarantee even the ordering within CPUs sitting across clusters.
Firstly, you can only have multiple clusters of CPUs running with a single
Linux image if hardware coherency is supported between them. In this case,
all the CPUs will live in the same inner-shareable domain and dmb ish is
sufficient to enforce ordering between them.
Secondly, a dsb executed by a CPU is irrelevant to ordering of accesses by
an external peripheral, regardless of whether that peripheral is cache
coherent. If you think about this as a producer/consumer problem, you need
ordering at *both* ends to make any guarantees.
Will
next prev parent reply other threads:[~2014-04-24 14:09 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-21 18:03 [PATCH] ARM: mm: dma: Update coherent streaming apis with missing memory barrier Santosh Shilimkar
2014-04-22 10:28 ` Will Deacon
2014-04-22 13:49 ` Santosh Shilimkar
2014-04-22 14:08 ` Arnd Bergmann
2014-04-22 14:36 ` Santosh Shilimkar
2014-04-22 19:53 ` Arnd Bergmann
2014-04-22 19:58 ` Santosh Shilimkar
2014-04-22 20:23 ` Arnd Bergmann
2014-04-22 20:30 ` Santosh Shilimkar
2014-04-23 9:02 ` Will Deacon
2014-04-23 16:02 ` Catalin Marinas
2014-04-23 17:17 ` Will Deacon
2014-04-23 18:37 ` Russell King - ARM Linux
2014-04-23 18:58 ` Arnd Bergmann
2014-04-23 19:04 ` Russell King - ARM Linux
2014-04-24 10:47 ` Catalin Marinas
2014-04-24 11:15 ` Russell King - ARM Linux
2014-04-24 11:21 ` Will Deacon
2014-04-24 13:38 ` Santosh Shilimkar
2014-04-24 14:09 ` Will Deacon [this message]
2014-04-24 14:44 ` Santosh Shilimkar
2014-04-24 19:12 ` Russell King - ARM Linux
2014-04-23 19:34 ` Jason Gunthorpe
2014-04-24 10:58 ` Will Deacon
2014-04-24 12:12 ` Arnd Bergmann
2014-04-24 12:37 ` Will Deacon
2014-04-24 9:54 ` Catalin Marinas
2014-04-24 11:13 ` Russell King - ARM Linux
2014-04-24 9:09 ` Catalin Marinas
2014-04-24 9:16 ` Russell King - ARM Linux
2014-04-24 10:13 ` Catalin Marinas
2014-05-02 21:33 ` Joel Fernandes
2014-05-06 10:01 ` Will Deacon
2014-04-22 15:07 ` Catalin Marinas
2014-04-22 15:18 ` Santosh Shilimkar
2014-04-22 15:30 ` Catalin Marinas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140424140913.GB14110@arm.com \
--to=will.deacon@arm$(echo .)com \
--cc=linux-arm-kernel@lists$(echo .)infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox