Subject: Re: spurious mce Hardware Error messages in next-20250912

From: Yazen Ghannam <[email-masked]>
Date: Wed, 17 Sep 2025 15:26:52 -0400

On Wed, Sep 17, 2025 at 05:33:29PM +0200, Bert Karwatzki wrote:
> Am Mittwoch, dem 17.09.2025 um 10:41 -0400 schrieb Yazen Ghannam:
> > On Wed, Sep 17, 2025 at 09:13:11AM +0200, Bert Karwatzki wrote:
> > > Am Dienstag, dem 16.09.2025 um 22:27 +0200 schrieb Bert Karwatzki:
> > [...]
> > > 
> > > I ran a test for 10h and got one real deferred error, I also looked through
> > > older logs (which only go back to 2025-08-17) and they do not contain any
> > > mce Hardware errors. Here's the output of
> > > 
> > > $ dmesg | grep -E "mce|Hardware Error"
> > > [...]
> > > [10163.739261] [   T9326] mce: [Hardware Error]: Machine check events logged
> > > [10163.739265] [   T9326] [Hardware Error]: Deferred error, no action required.
> > > [10163.739267] [   T9326] [Hardware Error]: CPU:0 (19:50:0) MC14_STATUS[-|-|-|AddrV|PCC|-|-|Deferred|-|-]: 0x8700900800000000
> > > [10163.739275] [   T9326] [Hardware Error]: Error Addr: 0x0095464100000020
> > > [10163.739276] [   T9326] [Hardware Error]: IPID: 0x000700b040000000
> > > [10163.739278] [   T9326] [Hardware Error]: L3 Cache Ext. Error Code: 0
> > > [10163.739279] [   T9326] [Hardware Error]: cache level: RESV, tx: INSN
> > > [...]
> 
> This seems to be a real deferred errror.

The "Deferred" status bit is set, but that seems to be coincidence. This
error code shouldn't have this bit set. Likewise, in previous examples
we saw the "Poison" status bit set when it shouldn't be.

> 
> > 
> > Summary so far:
> > 1) Errors are found on CPU0 banks 11 and 14.
> > 2) Errors are found during MCA timer-based polling.
> > 3) The data is coming from MCA_DESTAT register.
> > 4) The status bits are not consistent with documentation.
> > 5) Likely these errors are not generating a deferred error interrupt.
> > 
> > Bert, can you please collecting the following data?
> > 
> > 1) Output of "/proc/interrupts".
> >   a) The MCE, MCP, THR, and DFR lines are of interest.
> >   b) We should verify if any other notification types occur besides
> >      "MCP" (MCA polling).
> 
> This is from next-20250916 (without the debug patch), unfortunately I've
> already rebooted after the testrun with next-20250912 and your debug patch.
> 
> $ cat /proc/interrupts | grep -E "DFR|THR|MCE|MCP"
>  THR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0
> 0          0   Threshold APIC interrupts
>  DFR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0
> 0          0   Deferred Error APIC interrupts
>  MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0
> 0          0   Machine check exceptions
>  MCP:         39         39         39         39         39         39         39         39         39         39         39         39         39         39
> 39         39   Machine check polls
> 
> 
> 
> > 2) Using an older kernel, read the MCA_DESTAT registers for L3 cache.
> >   a) CPU0 bank 11: "sudo rdmsr -p 0 0xC00020b8"
> >   b) CPU0 bank 14: "sudo rdmsr -p 0 0xC00020e8"
> >   c) If these are non-zero, then I think we can confirm that the
> >      spurious data was always there.
> > 
> > Thanks,
> > Yazen
> 
> This is from 6.12.43+deb13-amd64 (the stock debian trixie kernel, currently the
> oldest version I have installed):
> 
> # rdmsr -p 0 0xC00020b8
> 8700aa0800000000
> # rdmsr -p 0 0xC00020e8
> 8700a28800000000
> 

Right, so it seems we have bogus data logged in these registers. And
this is unrelated to the recent patches.

We have some combination of bits set in MCA_DESTAT registers. The
deferred error interrupt hasn't fired (at least from the latest
example).

There does seem to be some combination of bits that are always set and
others flip between examples.

I'll highlight this to our hardware folks. But I don't think there's
much we can do other than filter these out somehow.

I can add two checks to the patch to make it more like the current
behavior.

1) Check for 'Deferred' status bit when logging from the MCA_DESTAT.
This was in the debug patch I shared.
2) Only check MCA_DESTAT when we are notified by the deferred error
interrupt.

Technically, both of these shouldn't be necessary based on the
architecture.

So there's a third option: add this error signature to our filter_mce()
function.

As I write this out, I feel more inclined to option #3. I think it would
be better to stick to the architecture. We may get error reports like
this. But that may be preferable so that any potential hardware issues
can be investigated sooner. If there's a real problem, better to get it
fixed in future products rather than implicitly mask it by our code
flow.

Any thoughts from others?

Thanks,
Yazen