* [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
@ 2026-05-18 13:23 Yury Murashka
2026-05-18 20:29 ` Bjorn Helgaas
2026-05-19 9:53 ` Lukas Wunner
0 siblings, 2 replies; 7+ messages in thread
From: Yury Murashka @ 2026-05-18 13:23 UTC (permalink / raw)
To: bhelgaas, mahesh
Cc: oohall, corbet, skhan, linux-pci, linux-doc, linux-kernel,
linuxppc-dev, Yury Murashka
pci_aer_clear_nonfatal_status() is not called when AER recovery fails.
If a new AER error is subsequently reported, the AER driver calls
find_source_device() to find the source of the error. It rescans the
whole bus and picks the first device reporting an AER error. Because the
previous error was never cleared, the error is attributed to the wrong
device and AER recovery is started for the wrong device.
Add a kernel boot parameter pci=aer_clear_on_recovery_failure to clear
AER error status even when recovery fails, preventing stale errors from
causing incorrect device identification on subsequent AER events.
Signed-off-by: Yury Murashka <yurypm@arista•com>
---
Documentation/admin-guide/kernel-parameters.txt | 5 +++++
drivers/pci/pci.c | 2 ++
drivers/pci/pci.h | 2 ++
drivers/pci/pcie/err.c | 13 +++++++++++++
4 files changed, 22 insertions(+)
diff --git a/Documentation/admin-guide/kernel-parameters.txt
b/Documentation/admin-guide/kernel-parameters.txt
index 4d0f545fb..5a9e266f5 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5301,6 +5301,11 @@ Kernel parameters
nomio [S390] Do not use MIO instructions.
norid [S390] ignore the RID field and force use of
one PCI domain per PCI function
+ aer_clear_on_recovery_failure
+ [PCIE] If the PCIEAER kernel config parameter is
+ enabled, this kernel boot option can be used to
+ enable AER errors cleanup even if error recovery
+ failed.
notph [PCIE] If the PCIE_TPH kernel config parameter
is enabled, this kernel boot option can be used
to disable PCIe TLP Processing Hints support
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index d34266651..701459c62 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -6769,6 +6769,8 @@ static int __init pci_setup(char *str)
disable_acs_redir_param = str + 18;
} else if (!strncmp(str, "config_acs=", 11)) {
config_acs_param = str + 11;
+ } else if (!strncmp(str,
"aer_clear_on_recovery_failure", 29)) {
+ pci_enable_aer_clear_on_recovery_failure();
} else {
pr_err("PCI: Unknown option `%s'\n", str);
}
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 4a14f88e5..093a7c896 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -1292,6 +1292,7 @@ int pci_aer_clear_status(struct pci_dev *dev);
int pci_aer_raw_clear_status(struct pci_dev *dev);
void pci_save_aer_state(struct pci_dev *dev);
void pci_restore_aer_state(struct pci_dev *dev);
+void pci_enable_aer_clear_on_recovery_failure(void);
#else
static inline void pci_no_aer(void) { }
static inline void pci_aer_init(struct pci_dev *d) { }
@@ -1301,6 +1302,7 @@ static inline int pci_aer_clear_status(struct
pci_dev *dev) { return -EINVAL; }
static inline int pci_aer_raw_clear_status(struct pci_dev *dev) {
return -EINVAL; }
static inline void pci_save_aer_state(struct pci_dev *dev) { }
static inline void pci_restore_aer_state(struct pci_dev *dev) { }
+static inline void pci_enable_aer_clear_on_recovery_failure(void) { }
#endif
#ifdef CONFIG_ACPI
diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c
index bebe4bc11..29d655a34 100644
--- a/drivers/pci/pcie/err.c
+++ b/drivers/pci/pcie/err.c
@@ -21,6 +21,13 @@
#include "portdrv.h"
#include "../pci.h"
+static int enable_aer_clear_on_recovery_failure;
+
+void pci_enable_aer_clear_on_recovery_failure(void)
+{
+ enable_aer_clear_on_recovery_failure = 1;
+}
+
static pci_ers_result_t merge_result(enum pci_ers_result orig,
enum pci_ers_result new)
{
@@ -289,6 +296,12 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev,
return status;
failed:
+ if (enable_aer_clear_on_recovery_failure &&
+ (host->native_aer || pcie_ports_native)) {
+ pcie_clear_device_status(dev);
+ pci_aer_clear_nonfatal_status(dev);
+ }
+
pci_walk_bridge(bridge, pci_pm_runtime_put, NULL);
pci_walk_bridge(bridge, report_perm_failure_detected, NULL);
--
2.51.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
2026-05-18 13:23 [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure Yury Murashka
@ 2026-05-18 20:29 ` Bjorn Helgaas
2026-05-18 20:49 ` Yury M.
2026-05-19 9:53 ` Lukas Wunner
1 sibling, 1 reply; 7+ messages in thread
From: Bjorn Helgaas @ 2026-05-18 20:29 UTC (permalink / raw)
To: Yury Murashka
Cc: bhelgaas, mahesh, oohall, corbet, skhan, linux-pci, linux-doc,
linux-kernel, linuxppc-dev, Lukas Wunner
[+cc Lukas]
On Mon, May 18, 2026 at 02:23:36PM +0100, Yury Murashka wrote:
> pci_aer_clear_nonfatal_status() is not called when AER recovery fails.
> If a new AER error is subsequently reported, the AER driver calls
> find_source_device() to find the source of the error. It rescans the
> whole bus and picks the first device reporting an AER error. Because the
> previous error was never cleared, the error is attributed to the wrong
> device and AER recovery is started for the wrong device.
>
> Add a kernel boot parameter pci=aer_clear_on_recovery_failure to clear
> AER error status even when recovery fails, preventing stale errors from
> causing incorrect device identification on subsequent AER events.
Why should we add a kernel parameter for this? How would a user
decide whether to use the parameter? Are there cases where we
find the source of the first error, but we *wouldn't* want to clear
it if recovery fails?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
2026-05-18 20:29 ` Bjorn Helgaas
@ 2026-05-18 20:49 ` Yury M.
0 siblings, 0 replies; 7+ messages in thread
From: Yury M. @ 2026-05-18 20:49 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: bhelgaas, mahesh, oohall, corbet, skhan, linux-pci, linux-doc,
linux-kernel, linuxppc-dev, Lukas Wunner
Current behavior has existed for a long time and I could easily imagine
that there is software which relies on the fact that the system is in a
non-modified state if AER recovery failed. The software can analyze the
system and do cleanup afterwards. Sometimes, if something fails in the
system, it is better to have it in a non-modified state.
In short, I just wanted to preserve the current logic by default because
there is a chance that we have software which relies on the current
behavior.
On 5/18/26 21:29, Bjorn Helgaas wrote:
> [+cc Lukas]
>
> On Mon, May 18, 2026 at 02:23:36PM +0100, Yury Murashka wrote:
>> pci_aer_clear_nonfatal_status() is not called when AER recovery fails.
>> If a new AER error is subsequently reported, the AER driver calls
>> find_source_device() to find the source of the error. It rescans the
>> whole bus and picks the first device reporting an AER error. Because the
>> previous error was never cleared, the error is attributed to the wrong
>> device and AER recovery is started for the wrong device.
>>
>> Add a kernel boot parameter pci=aer_clear_on_recovery_failure to clear
>> AER error status even when recovery fails, preventing stale errors from
>> causing incorrect device identification on subsequent AER events.
> Why should we add a kernel parameter for this? How would a user
> decide whether to use the parameter? Are there cases where we
> find the source of the first error, but we *wouldn't* want to clear
> it if recovery fails?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
2026-05-18 13:23 [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure Yury Murashka
2026-05-18 20:29 ` Bjorn Helgaas
@ 2026-05-19 9:53 ` Lukas Wunner
[not found] ` <3633b587-5782-4cfa-b967-997de86866bb@arista.com>
1 sibling, 1 reply; 7+ messages in thread
From: Lukas Wunner @ 2026-05-19 9:53 UTC (permalink / raw)
To: Yury Murashka
Cc: bhelgaas, mahesh, oohall, linux-pci, linux-kernel, linuxppc-dev
On Mon, May 18, 2026 at 02:23:36PM +0100, Yury Murashka wrote:
> pci_aer_clear_nonfatal_status() is not called when AER recovery fails.
> If a new AER error is subsequently reported, the AER driver calls
> find_source_device() to find the source of the error. It rescans the
> whole bus and picks the first device reporting an AER error. Because the
> previous error was never cleared, the error is attributed to the wrong
> device and AER recovery is started for the wrong device.
I guess the rationale of the current behavior is that the devices
affected by the failed error recovery are basically in a broken
state once error recovery failed and so user intervention is
required, e.g. a remove/rescan via sysfs.
My question is, why is error recovery failing for the devices
in the first place?
And what does the hierarchy look like?
(lspci -tv and lspci -vvv output please)
I also don't quite follow your assertion that (only) the first device
reporting an error is picked. The algorithm tries to collect *all*
error-reporting devices in the affected portion of the hierarchy.
Thanks,
Lukas
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
[not found] ` <3633b587-5782-4cfa-b967-997de86866bb@arista.com>
@ 2026-05-20 8:43 ` Lukas Wunner
2026-05-20 10:00 ` Yury M.
2026-05-20 9:02 ` Lukas Wunner
1 sibling, 1 reply; 7+ messages in thread
From: Lukas Wunner @ 2026-05-20 8:43 UTC (permalink / raw)
To: Yury M.; +Cc: bhelgaas, mahesh, oohall, linux-pci, linux-kernel, linuxppc-dev
On Tue, May 19, 2026 at 05:05:20PM +0100, Yury M. wrote:
> Root port can detect AER error with source 0000:00:00.0.
>
> In this case, we call find_source_device -> find_device_iter. The
> 'multi-error' flag is not set, and we are looking for the first error (not
> all). This means that for any error with the 0000:00:00.0 source on the root
> port, we will report the error for the first device on the bus.
No, is_error_source() considers bus number 0 as a bogus number
and will iterate over all devices on the bus.
> In my case, an AER error reported by 0000:06:08.0 will be logged as an error
> reported by 0000:06:07.0 if AER recovery constantly fails.
The problem is that 0000:06:08.0 reports an Advisory Non-Fatal Error,
i.e. it sets the ANFE bit in the Correctable Error Status Register
and signals (only) a Correctable Error, even though it also sets bits
in the Uncorrectable Error Status Register.
The kernel lacks support for ANFE handling and will only clear the bits
in the Correctable Error Status Register. It neglects to also clear
(and report) the bits in the Uncorrectable Error Status Register.
There was an effort two years back to bring up ANFE support but it
fizzled out. I talked to the submitter and he's now busy with other
things:
https://lore.kernel.org/r/20240620025857.206647-1-zhenzhong.duan@intel.com/
It's on my todo list to respin his series but I can't promise when
I'll get to it.
Thanks,
Lukas
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
[not found] ` <3633b587-5782-4cfa-b967-997de86866bb@arista.com>
2026-05-20 8:43 ` Lukas Wunner
@ 2026-05-20 9:02 ` Lukas Wunner
1 sibling, 0 replies; 7+ messages in thread
From: Lukas Wunner @ 2026-05-20 9:02 UTC (permalink / raw)
To: Yury M.; +Cc: bhelgaas, mahesh, oohall, linux-pci, linux-kernel, linuxppc-dev
On Tue, May 19, 2026 at 05:05:20PM +0100, Yury M. wrote:
> [ 476.224325] pci 0000:46:00.1: AER: can't recover (no error_detected callback)
> [ 476.224333] pci 0000:46:00.2: AER: can't recover (no error_detected callback)
> [ 476.224335] pci 0000:46:00.3: AER: can't recover (no error_detected callback)
> [ 476.224338] pci 0000:46:00.4: AER: can't recover (no error_detected callback)
> [ 476.224371] pcieport 0000:06:07.0: AER: device recovery failed
One more thing, recovery is failing here because those four devices
on bus 46 are unbound. I've got two patches under development to
allow error recovery for unbound devices. You may want to try those
and see if error recovery succeeds:
https://github.com/l1k/linux/commits/aer_unbound
Thanks,
Lukas
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure
2026-05-20 8:43 ` Lukas Wunner
@ 2026-05-20 10:00 ` Yury M.
0 siblings, 0 replies; 7+ messages in thread
From: Yury M. @ 2026-05-20 10:00 UTC (permalink / raw)
To: Lukas Wunner
Cc: bhelgaas, mahesh, oohall, linux-pci, linux-kernel, linuxppc-dev
On 5/20/26 09:43, Lukas Wunner wrote:
> No, is_error_source() considers bus number 0 as a bogus number
> and will iterate over all devices on the bus.
Right, if address is 0, is_error_source accepts any device with AER
error and returns 'true' for any device on a bus. But
when is_error_source reports 'true' for the first device on a bus, we
stop iterating in find_device_iter. We stop iterating if
"e_info->multi_error_valid == 0", that is our case. So, the error is
reported only for the first device on a bus.
On 5/20/26 10:02, Lukas Wunner wrote:
> One more thing, recovery is failing here because those four devices
> on bus 46 are unbound. I've got two patches under development to
> allow error recovery for unbound devices. You may want to try those
> and see if error recovery succeeds:
>
> https://github.com/l1k/linux/commits/aer_unbound
I believe that aer_unbound will fix our issue. I need some time to
confirm that the commit fixes our particular issue.
But I think that we still have an issue in the pcie_do_recovery
function. It is still possible that AER recovery will fail. The reason
could be different, and most of these cases should be investigated and
fixed. But until these issues are fixed and there is possibility that
AER recovery could fail, users should not experience the kind of issues
I described in this patch. For me, it looks better to do an AER error
cleanup even if AER recovery failed. We can add an error log with a
warning that we did the AER errors cleanup although AER recovery failed.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-05-20 10:00 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-18 13:23 [PATCH] PCI/AER: Clear non-fatal errors on AER recovery failure Yury Murashka
2026-05-18 20:29 ` Bjorn Helgaas
2026-05-18 20:49 ` Yury M.
2026-05-19 9:53 ` Lukas Wunner
[not found] ` <3633b587-5782-4cfa-b967-997de86866bb@arista.com>
2026-05-20 8:43 ` Lukas Wunner
2026-05-20 10:00 ` Yury M.
2026-05-20 9:02 ` Lukas Wunner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox