From: Mikhail Zolotaryov <lebon@lebon•org.ua>
To: Prodyut Hazarika <phazarika@amcc•com>
Cc: Tom Burns <tburns@datacast•com>,
Andrea Zypchen <azypchen@intldata•ca>,
linuxppc-dev@lists•ozlabs.org, azilkie@datacast•com
Subject: Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)
Date: Wed, 09 Sep 2009 16:28:05 +0300 [thread overview]
Message-ID: <4AA7AD65.7070403@lebon.org.ua> (raw)
In-Reply-To: <0CA0A16855646F4FA96D25A158E299D606F60B70@SDCEXCHANGE01.ad.amcc.com>
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
> Hi Adam,
>
>
>> Yes, I am using the 440EPx (same as the sequoia board).
>> Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
>>
> (using
>
>> the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
>> end up being partially corrupted when we try to parse the data in the
>> virtual page. We have confirmed the data is good before the PCI-IDE
>> bridge. We are creating two 8K pages and map them to physical DMA
>>
> memory
>
>> using single-entry scatter/gather structs. When a DMA block is
>> corrupted, we see a random portion of it (always a multiple of 16byte
>> cache lines) is overwritten with old data from the last time the
>>
> buffer
>
>> was used.
>>
>
> This looks like a cache coherency problem.
> Can you ensure that the TLB entries corresponding to the DMA region has
> the CacheInhibit bit set.
> You will need a BDI connected to your system.
>
> Also, you will need to invalidate and flush the lines appropriately,
> since in 440 cores,
> L1Cache coherency is managed entirely by software.
> Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
> how to do it.
>
> Thanks
> Prodyut
>
> On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
>
>> Hi Adam,
>>
>>
>>> Are you sure there is L2 cache on the 440?
>>>
>> It depends on the SoC you are using. SoC like 460EX (Canyonlands
>>
> board)
>
>> have L2Cache.
>> It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
>> has a 440 cpu core, but no L2Cache.
>> Could you please tell me which SoC you are using?
>> You can also refer to the appropriate dts file to see if there is L2C.
>> For example, in canyonlands.dts (460EX based board), we have the L2C
>> entry.
>> L2C0: l2c {
>> ...
>> }
>>
>>
>>> I am seeing this problem with our custom IDE driver which is based on
>>>
>
>
>>> pretty old code. Our driver uses pci_alloc_consistent() to allocate
>>>
> the
>
>>> physical DMA memory and alloc_pages() to allocate a virtual page. It
>>> then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
>>> should convert these to the DMA API calls as you suggest.
>>>
>> Could you give more details on the consistency problem? It is a good
>> idea to change to the new DMA APIs, but pci_alloc_consistent() should
>> work too
>>
>> Thanks
>> Prodyut
>>
>> On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
>>
>>> On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
>>>
>>>> Hi Adam,
>>>>
>>>> If you have a look in include/asm-ppc/pgtable.h for the following
>>>>
>> section:
>>
>>>> #ifdef CONFIG_44x
>>>> #define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED |
>>>>
>> _PAGE_GUARDED)
>>
>>>> #else
>>>> #define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
>>>> #endif
>>>>
>>>> Try adding _PAGE_COHERENT to the appropriate line above and see if
>>>>
>> that
>>
>>>> fixes your issue - this causes the 'M' bit to be set on the page
>>>>
>> which
>>
>>>> sure enforce cache coherency. If it doesn't, you'll need to check
>>>>
>> the
>>
>>>> 'M' bit isn't being masked out in head_44x.S (it was originally
>>>>
>> masked
>>
>>>> out on arch/powerpc, but was fixed in later kernels when the cache
>>>>
>
>
>>>> coherency issues with non-SMP systems were resolved).
>>>>
>>> I have some doubts about the usefulness of doing that for 4xx.
>>>
> AFAIK,
>
>>> the 440 core just ignores M.
>>>
>>> The problem lies probably elsewhere. Maybe the L2 cache coherency
>>>
>> isn't
>>
>>> enabled or not working ?
>>>
>>> The L1 cache on 440 is simply not coherent, so drivers have to make
>>>
>> sure
>>
>>> they use the appropriate DMA APIs which will do cache flushing when
>>> needed.
>>>
>>> Adam, what driver is causing you that sort of problems ?
>>>
>>> Cheers,
>>> Ben.
>>>
>>>
>>>
next prev parent reply other threads:[~2009-09-09 13:37 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-09-02 21:22 AW: PowerPC PCI DMA issues (prefetch/coherency?) Adam Zilkie
2009-09-03 8:05 ` Chris Pringle
2009-09-03 9:57 ` Benjamin Herrenschmidt
2009-09-03 16:04 ` Adam Zilkie
2009-09-03 16:21 ` Josh Boyer
2009-09-03 20:27 ` Prodyut Hazarika
2009-09-08 18:01 ` Adam Zilkie
2009-09-08 18:59 ` Prodyut Hazarika
2009-09-08 19:30 ` Adam Zilkie
2009-09-08 19:56 ` Prodyut Hazarika
2009-09-08 20:00 ` Adam Zilkie
2009-09-09 1:34 ` Benjamin Herrenschmidt
2009-09-08 21:34 ` Benjamin Herrenschmidt
2009-09-09 13:28 ` Mikhail Zolotaryov [this message]
2009-09-09 13:43 ` Tom Burns
2009-09-09 14:12 ` Mikhail Zolotaryov
2009-09-09 14:10 ` Tom Burns
2009-09-09 14:40 ` Mikhail Zolotaryov
2009-09-11 1:57 ` Benjamin Herrenschmidt
2009-09-11 7:17 ` Mikhail Zolotaryov
2009-09-11 7:31 ` Benjamin Herrenschmidt
2009-09-11 1:57 ` Benjamin Herrenschmidt
2009-09-10 19:53 ` Tom Burns
2009-09-10 20:30 ` Pravin Bathija
2009-09-11 2:44 ` Benjamin Herrenschmidt
2009-09-11 5:12 ` Stefan Roese
2009-09-11 5:17 ` Benjamin Herrenschmidt
2009-09-11 5:25 ` Stefan Roese
2009-09-11 5:35 ` Pravin Bathija
2009-09-11 5:40 ` Benjamin Herrenschmidt
2009-09-11 9:23 ` Pravin Bathija
2009-09-11 1:59 ` Benjamin Herrenschmidt
2009-09-11 16:05 ` Prodyut Hazarika
2009-09-11 1:55 ` Benjamin Herrenschmidt
2009-09-11 13:51 ` Tom Burns
2009-09-08 21:29 ` Benjamin Herrenschmidt
2009-09-03 12:20 ` Wrobel Heinz-R39252
2009-09-03 12:43 ` Chris Pringle
2009-09-06 21:32 ` Benjamin Herrenschmidt
2009-09-03 15:54 ` Adam Zilkie
[not found] <4A37A503.3030209@oxtel.com>
[not found] ` <20090616162114.GA5051@loki.buserror.net>
[not found] ` <4A37C97A.5050508@oxtel.com>
2009-06-16 16:46 ` Scott Wood
2009-06-16 16:57 ` Chris Pringle
2009-06-16 17:03 ` Scott Wood
2009-06-17 7:58 ` Chris Pringle
2009-06-17 13:18 ` Chris Pringle
2009-06-18 11:24 ` Chris Pringle
2009-06-22 14:31 ` AW: " Sergej.Stepanov
2009-06-29 8:11 ` Chris Pringle
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4AA7AD65.7070403@lebon.org.ua \
--to=lebon@lebon$(echo .)org.ua \
--cc=azilkie@datacast$(echo .)com \
--cc=azypchen@intldata$(echo .)ca \
--cc=linuxppc-dev@lists$(echo .)ozlabs.org \
--cc=phazarika@amcc$(echo .)com \
--cc=tburns@datacast$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox