From: Benjamin Herrenschmidt <benh@kernel•crashing.org>
To: Sinan Kaya <okaya@codeaurora•org>,
Linus Torvalds <torvalds@linux-foundation•org>
Cc: Alexander Duyck <alexander.duyck@gmail•com>,
Will Deacon <will.deacon@arm•com>, Arnd Bergmann <arnd@arndb•de>,
Jason Gunthorpe <jgg@ziepe•ca>,
David Laight <David.Laight@aculab•com>, Oliver <oohall@gmail•com>,
"open list:LINUX FOR POWERPC (32-BIT AND 64-BIT)"
<linuxppc-dev@lists•ozlabs.org>,
"linux-rdma@vger•kernel.org" <linux-rdma@vger•kernel.org>,
Alexander Duyck <alexander.h.duyck@redhat•com>,
"Paul E. McKenney" <paulmck@linux•vnet.ibm.com>,
"netdev@vger•kernel.org" <netdev@vger•kernel.org>
Subject: Re: RFC on writel and writel_relaxed
Date: Wed, 28 Mar 2018 15:41:45 +1100 [thread overview]
Message-ID: <1522212105.7364.101.camel@kernel.crashing.org> (raw)
In-Reply-To: <bf77be28-051d-79c8-9b65-948c8a254d6d@codeaurora.org>
On Tue, 2018-03-27 at 23:24 -0400, Sinan Kaya wrote:
> On 3/27/2018 10:51 PM, Linus Torvalds wrote:
> > > The discussion at hand is about
> > >
> > > dma_buffer->foo = 1; /* WB */
> > > writel(KICK, DMA_KICK_REGISTER); /* UC */
> >
> > Yes. That certainly is ordered on x86. In fact, afaik it's ordered
> > even if that writel() might be of type WC, because that only delays
> > writes, it doesn't move them earlier.
>
> Now that we clarified x86 myth, Is this guaranteed on all architectures?
If not we need to fix it. It's guaranteed on the "main" ones (arm,
arm64, powerpc, i386, x86_64). We might need to check with other arch
maintainers for the rest.
We really want Linux to provide well defined "sane" semantics for the
basic writel accessors.
Note: We still have open questions about how readl() relates to
surrounding memory accesses. It looks like ARM and powerpc do different
things here.
> We keep getting IA64 exception example. Maybe, this is also corrected since
> then.
I would think ia64 fixed it back when it was all discussed. I was under
the impression all ia64 had "special" was the writel vs. spin_unlock
which requires mmiowb, but maybe that was never completely fixed ?
> Jose Abreu says "I don't know about x86 but arc architecture doesn't
> have a wmb() in the writel() function (in some configs)".
Well, it probably should then.
> As long as we have these exceptions, these wmb() in common drivers is not
> going anywhere and relaxed-arches will continue paying performance penalty.
Well, let's fix them or leave them broken, at this point, it doesn't
matter. We can give all arch maintainers a wakeup call and start making
drivers work based on the documented assumptions.
> I see 15% performance loss on ARM64 servers using Intel i40e network
> drivers and an XL710 adapter due to CPU keeping itself busy doing barriers
> most of the time rather than real work because of sequences like this all over
> the place.
>
> dma_buffer->foo = 1; /* WB */
> wmb()
> writel(KICK, DMA_KICK_REGISTER); /* UC */
>
> I posted several patches last week to remove duplicate barriers on ARM while
> trying to make the code friendly with other architectures.
>
> Basically changing it to
>
> dma_buffer->foo = 1; /* WB */
> wmb()
> writel_relaxed(KICK, DMA_KICK_REGISTER); /* UC */
> mmiowb()
>
> This is a small step in the performance direction until we remove all exceptions.
>
> https://www.spinics.net/lists/netdev/msg491842.html
> https://www.spinics.net/lists/linux-rdma/msg62434.html
> https://www.spinics.net/lists/arm-kernel/msg642336.html
>
> Discussion started to move around the need for relaxed API on PPC and then
> why wmb() question came up.
I'm working on the problem of relaxed APIs for powerpc, but we can keep
that orthogonal. As is, today, a wmb() + writel() and a wmb() +
writel_relaxed() on powerpc are identical. So changing them will not
break us.
But I don't see the point of doing that transformation if we can just
get the straying archs fixed. It's not like any of them has a
significant market presence these days anyway.
Cheers,
Ben.
> Sinan
>
next prev parent reply other threads:[~2018-03-28 4:43 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1521692689.16434.293.camel@kernel.crashing.org>
[not found] ` <CAOSf1CFv1HHCL3YOVhRn2U=grdNjaQ=A4m3xwxN2Rek1-_TySg@mail.gmail.com>
[not found] ` <1521726722.16434.312.camel@kernel.crashing.org>
[not found] ` <20180323163510.GC13033@ziepe.ca>
[not found] ` <1521854626.16434.359.camel@kernel.crashing.org>
[not found] ` <58ce5b83f40f4775bec1be8db66adb0d@AcuMS.aculab.com>
[not found] ` <20180326165425.GA15554@ziepe.ca>
[not found] ` <CAK8P3a1zeMyj+Z-y4ER4moY6Zip9EWNOinf+VnboGOrgiwbBZA@mail.gmail.com>
[not found] ` <20180326202545.GB15554@ziepe.ca>
[not found] ` <CAK8P3a3fc43ZcW626hmsd3DVcLw7hGkdUMxp7s4Rn3mdkziwMQ@mail.gmail.com>
[not found] ` <20180326210951.GD15554@ziepe.ca>
[not found] ` <CAK8P3a2UU1xAM0NLo7Q4-Xgo1SzY3De1uqpFudr+2ZW7nHEPmA@mail.gmail.com>
[not found] ` <1522101616.7364.13.camel@kernel.crashing.org>
2018-03-27 14:46 ` RFC on writel and writel_relaxed Sinan Kaya
2018-03-27 15:01 ` Jose Abreu
2018-03-27 15:10 ` Will Deacon
2018-03-27 18:54 ` Alexander Duyck
2018-03-27 19:54 ` Arnd Bergmann
2018-03-27 20:46 ` Arnd Bergmann
2018-03-27 21:33 ` Benjamin Herrenschmidt
2018-03-28 0:39 ` Linus Torvalds
2018-03-28 1:03 ` Benjamin Herrenschmidt
2018-03-28 2:51 ` Linus Torvalds
2018-03-28 3:24 ` Sinan Kaya
2018-03-28 4:41 ` Benjamin Herrenschmidt [this message]
2018-03-28 6:14 ` Linus Torvalds
2018-03-28 11:41 ` okaya
2018-03-28 15:13 ` Benjamin Herrenschmidt
2018-03-28 15:55 ` David Miller
2018-03-28 16:23 ` Nicholas Piggin
2018-03-28 21:31 ` Benjamin Herrenschmidt
2018-03-28 22:09 ` Nicholas Piggin
2018-03-29 9:20 ` Will Deacon
2018-03-29 13:56 ` Sinan Kaya
2018-03-29 14:04 ` David Miller
2018-03-29 16:29 ` Arnd Bergmann
2018-03-29 16:59 ` Sinan Kaya
2018-03-30 1:40 ` Benjamin Herrenschmidt
2018-04-02 13:01 ` Sinan Kaya
2018-03-28 4:33 ` Benjamin Herrenschmidt
2018-03-28 6:26 ` Linus Torvalds
2018-03-28 6:42 ` Benjamin Herrenschmidt
2018-03-28 6:53 ` Linus Torvalds
2018-03-28 6:56 ` Benjamin Herrenschmidt
2018-03-28 7:11 ` Arnd Bergmann
2018-03-28 7:42 ` Benjamin Herrenschmidt
2018-03-28 9:07 ` Will Deacon
2018-03-28 9:56 ` Benjamin Herrenschmidt
2018-03-28 10:13 ` Aw: " Lino Sanfilippo
2018-03-28 10:20 ` Benjamin Herrenschmidt
2018-03-28 11:30 ` David Laight
2018-03-28 15:12 ` Benjamin Herrenschmidt
2018-03-28 16:16 ` David Laight
2018-03-28 1:21 ` Benjamin Herrenschmidt
2018-03-27 21:35 ` Benjamin Herrenschmidt
2018-03-27 21:54 Alexander Duyck
2018-03-27 22:35 ` Sinan Kaya
2018-03-27 23:43 ` Benjamin Herrenschmidt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1522212105.7364.101.camel@kernel.crashing.org \
--to=benh@kernel$(echo .)crashing.org \
--cc=David.Laight@aculab$(echo .)com \
--cc=alexander.duyck@gmail$(echo .)com \
--cc=alexander.h.duyck@redhat$(echo .)com \
--cc=arnd@arndb$(echo .)de \
--cc=jgg@ziepe$(echo .)ca \
--cc=linux-rdma@vger$(echo .)kernel.org \
--cc=linuxppc-dev@lists$(echo .)ozlabs.org \
--cc=netdev@vger$(echo .)kernel.org \
--cc=okaya@codeaurora$(echo .)org \
--cc=oohall@gmail$(echo .)com \
--cc=paulmck@linux$(echo .)vnet.ibm.com \
--cc=torvalds@linux-foundation$(echo .)org \
--cc=will.deacon@arm$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox