Subject: Re: Machine Check in P2010(e500v2)

From: Joakim Tjernlund <[email-masked]>
Date: Sat, 9 Sep 2017 12:45:43 +0000

On Fri, 2017-09-08 at 22:27 +0000, Leo Li wrote:
> > -----Original Message-----
> > From: Joakim Tjernlund [mailto:[email-masked]]
> > Sent: Friday, September 08, 2017 7:51 AM
> > To: [email-masked]; Leo Li <[email-masked]>; York Sun
> > <[email-masked]>
> > Subject: Re: Machine Check in P2010(e500v2)
> > 
> > On Fri, 2017-09-08 at 11:54 +0200, Joakim Tjernlund wrote:
> > > On Thu, 2017-09-07 at 18:54 +0000, Leo Li wrote:
> > > > > -----Original Message-----
> > > > > From: Joakim Tjernlund [mailto:[email-masked]]
> > > > > Sent: Thursday, September 07, 2017 3:41 AM
> > > > > To: [email-masked]; Leo Li <[email-masked]>;
> > > > > York Sun <[email-masked]>
> > > > > Subject: Re: Machine Check in P2010(e500v2)
> > > > > 
> > > > > On Thu, 2017-09-07 at 00:50 +0200, Joakim Tjernlund wrote:
> > > > > > On Wed, 2017-09-06 at 21:13 +0000, Leo Li wrote:
> > > > > > > > -----Original Message-----
> > > > > > > > From: Joakim Tjernlund
> > > > > > > > [mailto:[email-masked]]
> > > > > > > > Sent: Wednesday, September 06, 2017 3:54 PM
> > > > > > > > To: [email-masked]; Leo Li
> > > > > > > > <[email-masked]>; York Sun <[email-masked]>
> > > > > > > > Subject: Re: Machine Check in P2010(e500v2)
> > > > > > > > 
> > > > > > > > On Wed, 2017-09-06 at 20:28 +0000, Leo Li wrote:
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Joakim Tjernlund
> > > > > > > > > > [mailto:[email-masked]]
> > > > > > > > > > Sent: Wednesday, September 06, 2017 3:17 PM
> > > > > > > > > > To: [email-masked]; Leo Li
> > > > > > > > > > <[email-masked]>; York Sun <[email-masked]>
> > > > > > > > > > Subject: Re: Machine Check in P2010(e500v2)
> > > > > > > > > > 
> > > > > > > > > > On Wed, 2017-09-06 at 19:31 +0000, Leo Li wrote:
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: York Sun
> > > > > > > > > > > > Sent: Wednesday, September 06, 2017 10:38 AM
> > > > > > > > > > > > To: Joakim Tjernlund
> > > > > > > > > > > > <[email-masked]>;
> > > > > > > > > > > > linuxppc- [email-masked]; Leo Li
> > > > > > > > > > > > <[email-masked]>
> > > > > > > > > > > > Subject: Re: Machine Check in P2010(e500v2)
> > > > > > > > > > > > 
> > > > > > > > > > > > Scott is no longer with Freescale/NXP. Adding Leo.
> > > > > > > > > > > > 
> > > > > > > > > > > > On 09/05/2017 01:40 AM, Joakim Tjernlund wrote:
> > > > > > > > > > > > > So after some debugging I found this bug:
> > > > > > > > > > > > > @@ -996,7 +998,7 @@ int
> > > > > > > > > > > > > fsl_pci_mcheck_exception(struct pt_regs
> > > > > > > > 
> > > > > > > > *regs)
> > > > > > > > > > > > >          if (is_in_pci_mem_space(addr)) {
> > > > > > > > > > > > >                  if (user_mode(regs)) {
> > > > > > > > > > > > >                          pagefault_disable();
> > > > > > > > > > > > > -                       ret = get_user(regs->nip, &inst);
> > > > > > > > > > > > > +                       ret = get_user(inst,
> > > > > > > > > > > > > + (__u32 __user *)regs->nip);
> > > > > > > > > > > > >                          pagefault_enable();
> > > > > > > > > > > > >                  } else {
> > > > > > > > > > > > >                          ret =
> > > > > > > > > > > > > probe_kernel_address(regs->nip, inst);
> > > > > > > > > > > > > 
> > > > > > > > > > > > > However, the kernel still locked up after fixing that.
> > > > > > > > > > > > > Now I wonder why this fixup is there in the first place?
> > > > > > > > > > > > > The routine will not really fixup the insn, just
> > > > > > > > > > > > > return 0xffffffff for the failing read and then advance the
> > 
> > process NIP.
> > > > > > > > > > > 
> > > > > > > > > > > You are right.  The code here only gives 0xffffffff to
> > > > > > > > > > > the load instructions and
> > > > > > > > > > 
> > > > > > > > > > continue with the next instruction when the load
> > > > > > > > > > instruction is causing the machine check.  This will
> > > > > > > > > > prevent a system lockup when reading from PCI/RapidIO device
> > 
> > which is link down.
> > > > > > > > > > > 
> > > > > > > > > > > I don't know what is actual problem in your case.
> > > > > > > > > > > Maybe it is a write
> > > > > > > > > > 
> > > > > > > > > > instruction instead of read?   Or the code is in a infinite loop
> > 
> > waiting for
> > > > > 
> > > > > a
> > > > > > > > 
> > > > > > > > valid
> > > > > > > > > > read result?  Are you able to do some further debugging
> > > > > > > > > > with the NIP correctly printed?
> > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > According to the MC it is a Read and the NIP also leads
> > > > > > > > > > to a read in the
> > > > > > > > 
> > > > > > > > program.
> > > > > > > > > > ATM, I have disabled the fixup but I will enable that again.
> > > > > > > > > > Question, is it safe add a small printk when this MC
> > > > > > > > > > happens(after fixing up)? I need to see that it has
> > > > > > > > > > happened as the error is somewhat
> > > > > > > > 
> > > > > > > > random.
> > > > > > > > > 
> > > > > > > > > I think it is safe to add printk as the current machine
> > > > > > > > > check handlers are also
> > > > > > > > 
> > > > > > > > using printk.
> > > > > > > > 
> > > > > > > > I hope so, but if the fixup fires there is no printk at all so I was a bit
> > 
> > unsure.
> > > > > > > > Don't like this fixup though, is there not a better way than
> > > > > > > > faking a read to user space(or kernel for that matter) ?
> > > > > > > 
> > > > > > > I don't have a better idea.  Without the fixup, the offending
> > > > > > > load instruction
> > > > > 
> > > > > will never finish if there is anything wrong with the backing
> > > > > device and freeze the whole system.  Do you have any suggestion in mind?
> > > > > > > 
> > > > > > 
> > > > > > But it never finishes the load, it just fakes a load of
> > > > > > 0xfffffffff, for user space I rather have it signal a SIGBUS but
> > > > > > that does not seem to work either, at least not for us but that
> > > > > > could be a bug in general MC code
> > > > > 
> > > > > maybe.
> > > > > > This fixup might be valid for kernel only as it has never worked
> > > > > > for user space
> > > > > 
> > > > > due to the bug I found.
> > > > > > 
> > > > > > Where can I read about this errata ?
> > > > > 
> > > > > I have look high and low an cannot find an errata which maps to this fixup.
> > > > > The closest I get is A-005125 which seems to have another
> > > > > workaround, I cannot find any evidence that this workaround has been
> > 
> > applied in Linux, can you?
> > > > 
> > > > This is not A-005125.  There was an erratum for this issue with older silicons
> > 
> > (e.g. erratum PCI-ex 3 for MPC8572).
> > > > " When its link goes down, the PCI Express controller clears all
> > > > outstanding transactions with an error indicator and sends a link
> > > > down exception to the interrupt controller if PEX_PME_MES_DISR[LDDD]
> > > > = 0. If, however, any transactions are sent to the controller after
> > > > the link down event, they are accepted by the controller and wait
> > > > for the link to come back up before starting any timeout counters (for
> > 
> > example, completion timeout). There is no mechanism to cancel the new
> > transactions short of a device HRESET. "
> > > > 
> > > > But it was removed in newer silicon like P2020/P2010 probably because a
> > 
> > Machine Check will be triggered in this situation to deal with the stalled
> > instruction and no longer considered it as a hardware issue.
> > > > 
> > > 
> > > Maybe this fixup should be configurable then?
> 
> No.  My point is that the problem was no longer considered a hardware issue because of the machine check mechanism is in place to handle it.  If there is no handling of this special case, we would still experience a system hang if this situation really occurs.
> 
> > > 
> > > > The A-005125 is dealt with in u-boot.
> > 
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.de
> > nx.de%2Fpipermail%2Fu-boot%2F2013-
> > August%2F161185.html&data=01%7C01%7Cleoyang.li%40nxp.com%7Ccb8a93e
> > 0090e48eb53a008d4f6b84235%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0&
> > sdata=8sR4yoXA4adqMHz6TY%2BvmYpfCBTcYEZHjPuANjz%2F1EQ%3D&reserve
> > d=0
> > > 
> > > Yes, I found it eventually :)
> > > 
> > > However, I cannot return to normal execution. I can follow the code to
> > > returning from
> > > machine_check_exception() and moving into ASM handler for returning
> > > from a ME but then I am a bit lost. It does not seem to be any problem
> > > executing, it feels more like a SW bug dealing with machine checks. Don't
> > 
> > known how to diagnose this further and could use some pointers.
> 
> Is the execution returned to the user application?  I doubt the system hang is caused by the machine check handling.
> You can try to comment out the machine check handling code and check if there is any improvement and see if
> this is related to the machine check handling.

It tries to return to user app but I cannot see what happens as the system lock up when the
MC returns.
How do you mean comment out MC handling? The simplest path is the PCI fixup which will
just do regs->nip += 4; and then return to user space. That still does not work as
as soon MC handling returns, the system is locked up.

> 
> Machine check is a serious situation and not always possible to be recovered from. 

This one should at least not kill the whole system. It is a simple bus error in user space and
the app should get SIGBUS and the the system should carry on. 

> I would focus more on debugging why the machine check is triggered by the user space application.
> Can you locate what code is causing this machine check from user space?  
> Is it accessing some hardware related space which is not ready? 
> Or is it accessing address that it shouldn't have accessed?

of course, this is ongoing and getting closer a solution. The MC looking the machine completely
does not make this any easier though.
These are 2 separate things, fixing the cause and not having a simple bus error lock up the machine.
I am focusing on fixing the lockup.

I have been following the execution in the kernel and I always end up in the ASM returning
from the MC.
The other day we got a similar PCI MC(bus error) on T1042 CPU(e5500/e500mc) and there
the system survived. The one thing I see different there is that MSR RI is set
when entering MC, why is that?

 Jocke