* [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support
@ 2026-05-18 8:49 Ruidong Tian
2026-05-18 8:49 ` [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user() Ruidong Tian
` (8 more replies)
0 siblings, 9 replies; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
This series continues Tong Tiangen's work on arm64 ARCH_HAS_COPY_MC
support. We encounter the same problem, and from a forward-looking
perspective, large-memory ARM machines such as Grace and Vera will suffer
more from this class of issues, which motivates us to push this feature
upstream.
Problem
=========
With the increase of memory capacity and density, the probability of memory
error also increases. The increasing size and density of server RAM in data
centers and clouds have shown increased uncorrectable memory errors.
Currently, more and more scenarios that can tolerate memory errors, such as
COW[1,2], KSM copy[3], coredump copy[4], khugepaged[5,6], uaccess copy[7],
etc.
Solution
=========
This patchset introduces a new processing framework on ARM64, which enables
ARM64 to support error recovery in the above scenarios, and more scenarios
can be expanded based on this in the future.
In arm64, memory error handling in do_sea(), which is divided into two cases:
1. If the user state consumed the memory errors, the solution is to kill
the user process and isolate the error page.
2. If the kernel state consumed the memory errors, the solution is to
panic.
For case 2, Undifferentiated panic may not be the optimal choice, as it can
be handled better. In some scenarios, we can avoid panic, such as uaccess,
if the uaccess fails due to memory error, only the user process will be
affected, returning an error to the caller and isolating the user page with
hardware memory errors is a better choice.
[1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
[2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
[3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
[4] commit 245f09226893 ("mm: hwpoison: coredump: support recovery from dump_user_range()")
[5] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
[6] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
[7] commit 278b917f8cb9 ("x86/mce: Add _ASM_EXTABLE_CPY for copy user access")
------------------
Test result:
Tested on Kunpeng 920.
1. copy_page(), copy_mc_page() basic function test pass, and the disassembly
contents remains the same before and after refactor.
2. copy_to/from_user() access kernel NULL pointer raise translation fault
and dump error message then die(), test pass.
3. Test following scenarios: copy_from_user(), get_user(), COW.
Before patched: trigger a hardware memory error then panic.
After patched: trigger a hardware memory error without panic.
Testing step:
step1. start an user-process.
step2. poison(einj) the user-process's page.
step3: user-process access the poison page in kernel mode, then trigger SEA.
step4: the kernel will not panic, only the user process is killed, the poison
page is isolated. (before patched, the kernel will panic in do_sea())
The above tests can also be reproduced using ras-tools, which provides
einj-based injection and validation for uaccess and COW scenarios.
Example usage:
einj_mem_uc futex # get_user
einj_mem_uc copyin # copy_to_user
einj_mem_uc copy-on-write # COW
Link: https://git.kernel.org/pub/scm/linux/kernel/git/aegl/ras-tools.git
------------------
Benefits
=========
According to Huawei's statistics from their storage products, memory errors
triggered in kernel-mode by COW and page cache read (uaccess) scenarios
account for more than 50%. With this patchset deployed, all kernel panics
caused by COW and page cache memory errors are eliminated.
Alibaba Cloud has also observed memory errors occurring in uaccess contexts.
Since V13:
1. Changed MC-safe functions to return an error rather than kill the user
process. When a user program invokes a syscall and the kernel encounters
a memory error during uaccess, killing the process is unexpected; the
syscall should return an error.
2. Added FEAT_MOPS support for the copy_page_mc paths.
3. Refactored copy_page() and memcpy() on top of the shared memcpy_template,
reducing duplicated assembly code.
Since v12:
Thanks to the suggestions of Jonathan, Mark, and Mauro, the following modifications
are made:
1. Rebase to latest kernel version.
2. Patch1, add Jonathan's and Mauro's review-by.
3. Patch2, modified do_apei_claim_sea() according to Mark's and Jonathan's suggestions,
and optimized the commit message according to Mark's suggestions(Added description of
the impact on regular copy_to_user()).
4. Patch3, optimized the commit message according to Mauro's suggestions and add Jonathan's
review-by.
5. Patch4, modified copy_mc_user_highpage() and Optimized the commit message according to
Jonathan's suggestions(no functional changes).
6. Patch5, optimized the commit message according to Mauro's suggestions.
7. Patch4/5, FEAT_MOPS is added to the code logic. Currently, the fixup is not performed
on the MOPS instruction.
8. Remove patch6 in v12 according to Jonathan's suggestions.
Since v11:
1. Rebase to latest kernel version 6.9-rc1.
2. Add patch 5, Since the problem described in "Since V10 Besides 3" has
been solved in a50026bdb867 ('iov_iter: get rid of 'copy_mc' flag').
3. Add the benefit of applying the patch set to our company to the description of patch0.
Since V10:
Accroding Mark's suggestion:
1. Merge V10's patch2 and patch3 to V11's patch2.
2. Patch2(V11): use new fixup_type for ld* in copy_to_user(), fix fatal
issues (NULL kernel pointeraccess) been fixup incorrectly.
3. Patch2(V11): refactoring the logic of do_sea().
4. Patch4(V11): Remove duplicate assembly logic and remove do_mte().
Besides:
1. Patch2(V11): remove st* insn's fixup, st* generally not trigger memory error.
2. Split a part of the logic of patch2(V11) to patch5(V11), for detail,
see patch5(V11)'s commit msg.
3. Remove patch6(v10) “arm64: introduce copy_mc_to_kernel() implementation”.
During modification, some problems that cannot be solved in a short
period are found. The patch will be released after the problems are
solved.
4. Add test result in this patch.
5. Modify patchset title, do not use machine check and remove "-next".
Since V9:
1. Rebase to latest kernel version 6.8-rc2.
2. Add patch 6/6 to support copy_mc_to_kernel().
Since V8:
1. Rebase to latest kernel version and fix topo in some of the patches.
2. According to the suggestion of Catalin, I attempted to modify the
return value of function copy_mc_[user]_highpage() to bytes not copied.
During the modification process, I found that it would be more
reasonable to return -EFAULT when copy error occurs (referring to the
newly added patch 4).
For ARM64, the implementation of copy_mc_[user]_highpage() needs to
consider MTE. Considering the scenario where data copying is successful
but the MTE tag copying fails, it is also not reasonable to return
bytes not copied.
3. Considering the recent addition of machine check safe support for
multiple scenarios, modify commit message for patch 5 (patch 4 for V8).
Since V7:
Currently, there are patches supporting recover from poison
consumption for the cow scenario[1]. Therefore, Supporting cow
scenario under the arm64 architecture only needs to modify the relevant
code under the arch/.
[1]https://lore.kernel.org/lkml/20221031201029.102123-1-tony.luck@intel.com/
Since V6:
Resend patches that are not merged into the mainline in V6.
Since V5:
1. Add patch2/3 to add uaccess assembly helpers.
2. Optimize the implementation logic of arm64_do_kernel_sea() in patch8.
3. Remove kernel access fixup in patch9.
All suggestion are from Mark.
Since V4:
1. According Michael's suggestion, add patch5.
2. According Mark's suggestiog, do some restructuring to arm64
extable, then a new adaptation of machine check safe support is made based
on this.
3. According Mark's suggestion, support machine check safe in do_mte() in
cow scene.
4. In V4, two patches have been merged into -next, so V5 not send these
two patches.
Since V3:
1. According to Robin's suggestion, direct modify user_ldst and
user_ldp in asm-uaccess.h and modify mte.S.
2. Add new macro USER_MC in asm-uaccess.h, used in copy_from_user.S
and copy_to_user.S.
3. According to Robin's suggestion, using micro in copy_page_mc.S to
simplify code.
4. According to KeFeng's suggestion, modify powerpc code in patch1.
5. According to KeFeng's suggestion, modify mm/extable.c and some code
optimization.
Since V2:
1. According to Mark's suggestion, all uaccess can be recovered due to
memory error.
2. Scenario pagecache reading is also supported as part of uaccess
(copy_to_user()) and duplication code problem is also solved.
Thanks for Robin's suggestion.
3. According Mark's suggestion, update commit message of patch 2/5.
4. According Borisllav's suggestion, update commit message of patch 1/5.
Since V1:
1.Consistent with PPC/x86, Using CONFIG_ARCH_HAS_COPY_MC instead of
ARM64_UCE_KERNEL_RECOVERY.
2.Add two new scenes, cow and pagecache reading.
3.Fix two small bug(the first two patch).
V1 in here:
https://lore.kernel.org/lkml/20220323033705.3966643-1-tongtiangen@huawei.com/
Ruidong Tian (3):
ACPI: APEI: GHES: use exception context to gate SIGBUS on poison
consumption
lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
Tong Tiangen (5):
uaccess: add generic fallback version of copy_mc_to_user()
arm64: add support for ARCH_HAS_COPY_MC
mm/hwpoison: return -EFAULT when copy fail in
copy_mc_[user]_highpage()
arm64: support copy_mc_[user]_highpage()
arm64: introduce copy_mc_to_kernel() implementation
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/asm-extable.h | 22 ++-
arch/arm64/include/asm/asm-uaccess.h | 4 +
arch/arm64/include/asm/extable.h | 1 +
arch/arm64/include/asm/mte.h | 9 +
arch/arm64/include/asm/page.h | 10 ++
arch/arm64/include/asm/string.h | 5 +
arch/arm64/include/asm/uaccess.h | 17 ++
arch/arm64/kernel/acpi.c | 2 +-
arch/arm64/lib/Makefile | 2 +
arch/arm64/lib/copy_mc_page.S | 44 +++++
arch/arm64/lib/copy_page.S | 62 +------
arch/arm64/lib/copy_page_template.S | 71 ++++++++
arch/arm64/lib/copy_to_user.S | 10 +-
arch/arm64/lib/memcpy.S | 253 ++-------------------------
arch/arm64/lib/memcpy_mc.S | 56 ++++++
arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++
arch/arm64/lib/mte.S | 29 +++
arch/arm64/mm/copypage.c | 75 ++++++++
arch/arm64/mm/extable.c | 21 +++
arch/arm64/mm/fault.c | 30 +++-
arch/powerpc/include/asm/uaccess.h | 1 +
arch/x86/include/asm/uaccess.h | 1 +
drivers/acpi/apei/ghes.c | 36 ++--
include/acpi/ghes.h | 6 +-
include/linux/highmem.h | 16 +-
include/linux/uaccess.h | 8 +
lib/tests/memcpy_kunit.c | 178 ++++++++++++++++++-
mm/kasan/shadow.c | 12 ++
mm/khugepaged.c | 4 +-
30 files changed, 904 insertions(+), 331 deletions(-)
create mode 100644 arch/arm64/lib/copy_mc_page.S
create mode 100644 arch/arm64/lib/copy_page_template.S
create mode 100644 arch/arm64/lib/memcpy_mc.S
create mode 100644 arch/arm64/lib/memcpy_template.S
--
2.39.3
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user()
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-27 9:16 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption Ruidong Tian
` (7 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong, Mauro Carvalho Chehab, Jonathan Cameron
From: Tong Tiangen <tongtiangen@huawei•com>
x86/powerpc has it's implementation of copy_mc_to_user(), we add generic
fallback in include/linux/uaccess.h prepare for other architechures to
enable CONFIG_ARCH_HAS_COPY_MC.
Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
Acked-by: Michael Ellerman <mpe@ellerman•id.au>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel•org>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei•com>
---
arch/powerpc/include/asm/uaccess.h | 1 +
arch/x86/include/asm/uaccess.h | 1 +
include/linux/uaccess.h | 8 ++++++++
3 files changed, 10 insertions(+)
diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h
index e98c628e3899..073de098d45a 100644
--- a/arch/powerpc/include/asm/uaccess.h
+++ b/arch/powerpc/include/asm/uaccess.h
@@ -432,6 +432,7 @@ copy_mc_to_user(void __user *to, const void *from, unsigned long n)
return n;
}
+#define copy_mc_to_user copy_mc_to_user
#endif
extern size_t copy_from_user_flushcache(void *dst, const void __user *src, size_t size);
diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index 3a0dd3c2b233..308b0854d1d5 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -496,6 +496,7 @@ copy_mc_to_kernel(void *to, const void *from, unsigned len);
unsigned long __must_check
copy_mc_to_user(void __user *to, const void *from, unsigned len);
+#define copy_mc_to_user copy_mc_to_user
#endif
/*
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index 56328601218c..c53a65394f80 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -250,6 +250,14 @@ copy_mc_to_kernel(void *dst, const void *src, size_t cnt)
}
#endif
+#ifndef copy_mc_to_user
+static inline unsigned long __must_check
+copy_mc_to_user(void *dst, const void *src, size_t cnt)
+{
+ return copy_to_user(dst, src, cnt);
+}
+#endif
+
static __always_inline void pagefault_disabled_inc(void)
{
current->pagefault_disabled++;
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
2026-05-18 8:49 ` [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user() Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-27 9:34 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC Ruidong Tian
` (6 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
When a GHES SEA (Synchronous External Abort) fires while the CPU
was executing in kernel mode, it typically means that kernel code
itself consumed a poisoned memory location -- e.g. copy_from_user()
/ copy_to_user() invoked from a ioctl() or write() syscall touched
a poisoned user page or page-cache page on behalf of the task.
The expected behaviour in that case is that the faulting kernel
helper returns via its extable fixup and the syscall returns an
error (e.g. -EFAULT) to user space. It is NOT appropriate to deliver
SIGBUS to the current task: the task did not directly dereference
the poisoned address, the kernel did on its behalf, and the kernel
is able to recover.
Up to now ghes_handle_memory_failure() unconditionally promoted any
synchronous recoverable memory error to MF_ACTION_REQUIRED, which
ends up SIGBUS on current -- regardless of whether the poison was
consumed from user space or from inside the kernel on the task's
behalf. That kills tasks that should instead have seen a plain
syscall error.
To fix this, the execution mode in which the exception was taken
must be captured at the arch-level entry point, where pt_regs (and
hence user_mode(regs)) are still available. The estatus node that
later drains the error in IRQ / process context no longer has
access to the original regs.
Introduce:
enum context { NO_USE = -1, IN_KERNEL = 0, IN_USER = 1 };
and plumb the value all the way down to the queued estatus node:
* Add an 'enum context context' field to struct ghes_estatus_node
and record it in ghes_in_nmi_queue_one_entry().
* Extend ghes_notify_sea() and the internal
ghes_in_nmi_spool_from_list() with an enum context parameter.
Then consume the recorded context in ghes_handle_memory_failure()
for the GHES_SEV_RECOVERABLE / sync path:
flags = sync && context == IN_USER ? MF_ACTION_REQUIRED : 0;
i.e. MF_ACTION_REQUIRED (and thus SIGBUS via the task_work path) is
only raised for user-mode poison consumption. Synchronous errors
taken in kernel mode fall back to memory_failure_queue() with
flags=0, asynchronously isolating the poisoned page while letting
the faulting kernel helper's extable fixup return -EFAULT
to user space.
Paths that pass NO_USE are unaffected:
sync is false for them, so flags stays 0 as before.
Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
---
arch/arm64/kernel/acpi.c | 2 +-
drivers/acpi/apei/ghes.c | 36 ++++++++++++++++++++----------------
include/acpi/ghes.h | 6 ++++--
3 files changed, 25 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 5891f92c2035..40d4a2913d51 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -409,7 +409,7 @@ int apei_claim_sea(struct pt_regs *regs)
*/
local_daif_restore(DAIF_ERRCTX);
nmi_enter();
- err = ghes_notify_sea();
+ err = ghes_notify_sea(user_mode(regs));
nmi_exit();
/*
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index 3236a3ce79d6..6f265893cddf 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -529,7 +529,7 @@ static bool ghes_do_memory_failure(u64 physical_addr, int flags)
}
static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
- int sev, bool sync)
+ int sev, bool sync, enum context context)
{
int flags = -1;
int sec_sev = ghes_severity(gdata->error_severity);
@@ -543,7 +543,7 @@ static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
(gdata->flags & CPER_SEC_ERROR_THRESHOLD_EXCEEDED))
flags = MF_SOFT_OFFLINE;
if (sev == GHES_SEV_RECOVERABLE && sec_sev == GHES_SEV_RECOVERABLE)
- flags = sync ? MF_ACTION_REQUIRED : 0;
+ flags = sync && context == IN_USER ? MF_ACTION_REQUIRED : 0;
if (flags != -1)
return ghes_do_memory_failure(mem_err->physical_addr, flags);
@@ -552,10 +552,10 @@ static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
}
static bool ghes_handle_arm_hw_error(struct acpi_hest_generic_data *gdata,
- int sev, bool sync)
+ int sev, bool sync, enum context context)
{
struct cper_sec_proc_arm *err = acpi_hest_get_payload(gdata);
- int flags = sync ? MF_ACTION_REQUIRED : 0;
+ int flags = sync && context == IN_USER ? MF_ACTION_REQUIRED : 0;
int length = gdata->error_data_length;
char error_type[120];
bool queued = false;
@@ -910,7 +910,8 @@ static void ghes_log_hwerr(int sev, guid_t *sec_type)
}
static void ghes_do_proc(struct ghes *ghes,
- const struct acpi_hest_generic_status *estatus)
+ const struct acpi_hest_generic_status *estatus,
+ enum context context)
{
int sev, sec_sev;
struct acpi_hest_generic_data *gdata;
@@ -937,11 +938,11 @@ static void ghes_do_proc(struct ghes *ghes,
atomic_notifier_call_chain(&ghes_report_chain, sev, mem_err);
arch_apei_report_mem_error(sev, mem_err);
- queued = ghes_handle_memory_failure(gdata, sev, sync);
+ queued = ghes_handle_memory_failure(gdata, sev, sync, context);
} else if (guid_equal(sec_type, &CPER_SEC_PCIE)) {
ghes_handle_aer(gdata);
} else if (guid_equal(sec_type, &CPER_SEC_PROC_ARM)) {
- queued = ghes_handle_arm_hw_error(gdata, sev, sync);
+ queued = ghes_handle_arm_hw_error(gdata, sev, sync, context);
} else if (guid_equal(sec_type, &CPER_SEC_CXL_PROT_ERR)) {
struct cxl_cper_sec_prot_err *prot_err = acpi_hest_get_payload(gdata);
@@ -1190,7 +1191,7 @@ static int ghes_proc(struct ghes *ghes)
if (ghes_print_estatus(NULL, ghes->generic, estatus))
ghes_estatus_cache_add(ghes->generic, estatus);
}
- ghes_do_proc(ghes, estatus);
+ ghes_do_proc(ghes, estatus, NO_USE);
out:
ghes_clear_estatus(ghes, estatus, buf_paddr, FIX_APEI_GHES_IRQ);
@@ -1297,7 +1298,7 @@ static void ghes_proc_in_irq(struct irq_work *irq_work)
len = cper_estatus_len(estatus);
node_len = GHES_ESTATUS_NODE_LEN(len);
- ghes_do_proc(estatus_node->ghes, estatus);
+ ghes_do_proc(estatus_node->ghes, estatus, estatus_node->context);
if (!ghes_estatus_cached(estatus)) {
generic = estatus_node->generic;
@@ -1335,7 +1336,8 @@ static void ghes_print_queued_estatus(void)
}
static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
- enum fixed_addresses fixmap_idx)
+ enum fixed_addresses fixmap_idx,
+ enum context context)
{
struct acpi_hest_generic_status *estatus, tmp_header;
struct ghes_estatus_node *estatus_node;
@@ -1364,6 +1366,7 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
if (!estatus_node)
return -ENOMEM;
+ estatus_node->context = context;
estatus_node->ghes = ghes;
estatus_node->generic = ghes->generic;
estatus = GHES_ESTATUS_FROM_NODE(estatus_node);
@@ -1398,14 +1401,15 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
}
static int ghes_in_nmi_spool_from_list(struct list_head *rcu_list,
- enum fixed_addresses fixmap_idx)
+ enum fixed_addresses fixmap_idx,
+ enum context context)
{
int ret = -ENOENT;
struct ghes *ghes;
rcu_read_lock();
list_for_each_entry_rcu(ghes, rcu_list, list) {
- if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx))
+ if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, context))
ret = 0;
}
rcu_read_unlock();
@@ -1488,7 +1492,7 @@ static LIST_HEAD(ghes_sea);
* Return 0 only if one of the SEA error sources successfully reported an error
* record sent from the firmware.
*/
-int ghes_notify_sea(void)
+int ghes_notify_sea(enum context context)
{
static DEFINE_RAW_SPINLOCK(ghes_notify_lock_sea);
int rv;
@@ -1497,7 +1501,7 @@ int ghes_notify_sea(void)
return -ENOENT;
raw_spin_lock(&ghes_notify_lock_sea);
- rv = ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA);
+ rv = ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA, context);
raw_spin_unlock(&ghes_notify_lock_sea);
return rv;
@@ -1552,7 +1556,7 @@ static int ghes_notify_nmi(unsigned int cmd, struct pt_regs *regs)
return ret;
raw_spin_lock(&ghes_notify_lock_nmi);
- if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI))
+ if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI, NO_USE))
ret = NMI_HANDLED;
raw_spin_unlock(&ghes_notify_lock_nmi);
@@ -1606,7 +1610,7 @@ static void ghes_nmi_init_cxt(void)
static int __ghes_sdei_callback(struct ghes *ghes,
enum fixed_addresses fixmap_idx)
{
- if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx)) {
+ if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, NO_USE)) {
irq_work_queue(&ghes_proc_irq_work);
return 0;
diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h
index 8d7e5caef3f1..646cd5c3c0ca 100644
--- a/include/acpi/ghes.h
+++ b/include/acpi/ghes.h
@@ -33,10 +33,12 @@ struct ghes {
void __iomem *error_status_vaddr;
};
+enum context {NO_USE = -1, IN_KERNEL = 0, IN_USER = 1};
struct ghes_estatus_node {
struct llist_node llnode;
struct acpi_hest_generic *generic;
struct ghes *ghes;
+ enum context context;
};
struct ghes_estatus_cache {
@@ -135,9 +137,9 @@ static inline void *acpi_hest_get_next(struct acpi_hest_generic_data *gdata)
section = acpi_hest_get_next(section))
#ifdef CONFIG_ACPI_APEI_SEA
-int ghes_notify_sea(void);
+int ghes_notify_sea(enum context context);
#else
-static inline int ghes_notify_sea(void) { return -ENOENT; }
+static inline int ghes_notify_sea(enum context context) { return -ENOENT; }
#endif
struct notifier_block;
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
2026-05-18 8:49 ` [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user() Ruidong Tian
2026-05-18 8:49 ` [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-27 11:35 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage() Ruidong Tian
` (5 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
From: Tong Tiangen <tongtiangen@huawei•com>
For the arm64 kernel, when it processes hardware memory errors for
synchronize notifications(do_sea()), if the errors is consumed within the
kernel, the current processing is panic. However, it is not optimal.
Take copy_from/to_user for example, If ld* triggers a memory error, even in
kernel mode, only the associated process is affected. Killing the user
process and isolating the corrupt page is a better choice.
Add new fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR to identify insn
that can recover from memory errors triggered by access to kernel memory,
and this fixup type is used in __arch_copy_to_user(), This make the regular
copy_to_user() will handle kernel memory errors.
[Ruidong: handle EX_TYPE_UACCESS_CPY in fixup_exception_me()]
Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/asm-extable.h | 22 +++++++++++++++++++-
arch/arm64/include/asm/asm-uaccess.h | 4 ++++
arch/arm64/include/asm/extable.h | 1 +
arch/arm64/lib/copy_to_user.S | 10 +++++-----
arch/arm64/mm/extable.c | 21 +++++++++++++++++++
arch/arm64/mm/fault.c | 30 ++++++++++++++++++++--------
7 files changed, 75 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943..831b20d45893 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -21,6 +21,7 @@ config ARM64
select ARCH_HAS_CACHE_LINE_SIZE
select ARCH_HAS_CC_PLATFORM
select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
+ select ARCH_HAS_COPY_MC if ACPI_APEI_GHES
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE
diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/asm-extable.h
index d67e2fdd1aee..4980023f2fbd 100644
--- a/arch/arm64/include/asm/asm-extable.h
+++ b/arch/arm64/include/asm/asm-extable.h
@@ -11,6 +11,8 @@
#define EX_TYPE_KACCESS_ERR_ZERO 3
#define EX_TYPE_UACCESS_CPY 4
#define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 5
+/* kernel access memory error safe */
+#define EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR 6
/* Data fields for EX_TYPE_UACCESS_ERR_ZERO */
#define EX_DATA_REG_ERR_SHIFT 0
@@ -42,7 +44,7 @@
(.L__gpr_num_##gpr << EX_DATA_REG_##reg##_SHIFT)
#define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \
- __ASM_EXTABLE_RAW(insn, fixup, \
+ __ASM_EXTABLE_RAW(insn, fixup, \
EX_TYPE_UACCESS_ERR_ZERO, \
( \
EX_DATA_REG(ERR, err) | \
@@ -55,6 +57,17 @@
#define _ASM_EXTABLE_UACCESS(insn, fixup) \
_ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, wzr, wzr)
+#define _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, err, zero) \
+ __ASM_EXTABLE_RAW(insn, fixup, \
+ EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, \
+ ( \
+ EX_DATA_REG(ERR, err) | \
+ EX_DATA_REG(ZERO, zero) \
+ ))
+
+#define _ASM_EXTABLE_KACCESS_MEM_ERR(insn, fixup) \
+ _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, wzr, wzr)
+
/*
* Create an exception table entry for uaccess `insn`, which will branch to `fixup`
* when an unhandled fault is taken.
@@ -76,6 +89,13 @@
.macro _asm_extable_uaccess_cpy, insn, fixup, uaccess_is_write
__ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_UACCESS_CPY, \uaccess_is_write)
.endm
+/*
+ * Create an exception table entry for kaccess `insn`, which will branch to
+ * `fixup` when an unhandled fault is taken.
+ */
+ .macro _asm_extable_kaccess_mem_err, insn, fixup
+ _ASM_EXTABLE_KACCESS_MEM_ERR(\insn, \fixup)
+ .endm
#else /* __ASSEMBLER__ */
diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h
index 12aa6a283249..c8f0af5fde63 100644
--- a/arch/arm64/include/asm/asm-uaccess.h
+++ b/arch/arm64/include/asm/asm-uaccess.h
@@ -57,6 +57,10 @@ alternative_else_nop_endif
.endm
#endif
+#define KERNEL_MEM_ERR(l, x...) \
+9999: x; \
+ _asm_extable_kaccess_mem_err 9999b, l
+
#define USER(l, x...) \
9999: x; \
_asm_extable_uaccess 9999b, l
diff --git a/arch/arm64/include/asm/extable.h b/arch/arm64/include/asm/extable.h
index 9dc39612bdf5..47c851d7df4f 100644
--- a/arch/arm64/include/asm/extable.h
+++ b/arch/arm64/include/asm/extable.h
@@ -48,4 +48,5 @@ bool ex_handler_bpf(const struct exception_table_entry *ex,
#endif /* !CONFIG_BPF_JIT */
bool fixup_exception(struct pt_regs *regs, unsigned long esr);
+bool fixup_exception_me(struct pt_regs *regs);
#endif
diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S
index 819f2e3fc7a9..991d94ecc1a8 100644
--- a/arch/arm64/lib/copy_to_user.S
+++ b/arch/arm64/lib/copy_to_user.S
@@ -20,7 +20,7 @@
* x0 - bytes not copied
*/
.macro ldrb1 reg, ptr, val
- ldrb \reg, [\ptr], \val
+ KERNEL_MEM_ERR(9998f, ldrb \reg, [\ptr], \val)
.endm
.macro strb1 reg, ptr, val
@@ -28,7 +28,7 @@
.endm
.macro ldrh1 reg, ptr, val
- ldrh \reg, [\ptr], \val
+ KERNEL_MEM_ERR(9998f, ldrh \reg, [\ptr], \val)
.endm
.macro strh1 reg, ptr, val
@@ -36,7 +36,7 @@
.endm
.macro ldr1 reg, ptr, val
- ldr \reg, [\ptr], \val
+ KERNEL_MEM_ERR(9998f, ldr \reg, [\ptr], \val)
.endm
.macro str1 reg, ptr, val
@@ -44,7 +44,7 @@
.endm
.macro ldp1 reg1, reg2, ptr, val
- ldp \reg1, \reg2, [\ptr], \val
+ KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr], \val)
.endm
.macro stp1 reg1, reg2, ptr, val
@@ -74,7 +74,7 @@ SYM_FUNC_START(__arch_copy_to_user)
9997: cmp dst, dstin
b.ne 9998f
// Before being absolutely sure we couldn't copy anything, try harder
- ldrb tmp1w, [srcin]
+KERNEL_MEM_ERR(9998f, ldrb tmp1w, [srcin])
USER(9998f, sttrb tmp1w, [dst])
add dst, dst, #1
9998: sub x0, end, dst // bytes not copied
diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c
index 6e0528831cd3..f78ac7e92845 100644
--- a/arch/arm64/mm/extable.c
+++ b/arch/arm64/mm/extable.c
@@ -110,7 +110,28 @@ bool fixup_exception(struct pt_regs *regs, unsigned long esr)
return ex_handler_uaccess_cpy(ex, regs, esr);
case EX_TYPE_LOAD_UNALIGNED_ZEROPAD:
return ex_handler_load_unaligned_zeropad(ex, regs);
+ case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR:
+ return false;
}
BUG();
}
+
+bool fixup_exception_me(struct pt_regs *regs)
+{
+ const struct exception_table_entry *ex;
+
+ ex = search_exception_tables(instruction_pointer(regs));
+ if (!ex)
+ return false;
+
+ switch (ex->type) {
+ case EX_TYPE_UACCESS_CPY:
+ return ex_handler_uaccess_cpy(ex, regs, 0);
+ case EX_TYPE_UACCESS_ERR_ZERO:
+ case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR:
+ return ex_handler_uaccess_err_zero(ex, regs);
+ }
+
+ return false;
+}
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0f3c5c7ca054..efbda54770be 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -858,21 +858,35 @@ static int do_bad(unsigned long far, unsigned long esr, struct pt_regs *regs)
return 1; /* "fault" */
}
+/*
+ * APEI claimed this as a firmware-first notification.
+ * Some processing deferred to task_work before ret_to_user().
+ */
+static int do_apei_claim_sea(struct pt_regs *regs)
+{
+ int ret;
+
+ ret = apei_claim_sea(regs);
+ if (ret)
+ return ret;
+
+ if (!user_mode(regs) && IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC)) {
+ if (!fixup_exception_me(regs))
+ return -ENOENT;
+ }
+
+ return ret;
+}
+
static int do_sea(unsigned long far, unsigned long esr, struct pt_regs *regs)
{
const struct fault_info *inf;
unsigned long siaddr;
- inf = esr_to_fault_info(esr);
-
- if (user_mode(regs) && apei_claim_sea(regs) == 0) {
- /*
- * APEI claimed this as a firmware-first notification.
- * Some processing deferred to task_work before ret_to_user().
- */
+ if (do_apei_claim_sea(regs) == 0)
return 0;
- }
+ inf = esr_to_fault_info(esr);
if (esr & ESR_ELx_FnV) {
siaddr = 0;
} else {
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage()
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
` (2 preceding siblings ...)
2026-05-18 8:49 ` [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-27 11:44 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage() Ruidong Tian
` (4 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong, Jonathan Cameron, Mauro Carvalho Chehab
From: Tong Tiangen <tongtiangen@huawei•com>
Currently, copy_mc_[user]_highpage() returns zero on success, or in case
of failures, the number of bytes that weren't copied.
While tracking the number of not copied works fine for x86 and PPC, There
are some difficulties in doing the same thing on ARM64 because there is no
available caller-saved register in copy_page()(lib/copy_page.S) to save
"bytes not copied", and the following copy_mc_page() will also encounter
the same problem.
Consider the caller of copy_mc_[user]_highpage() cannot do any processing
on the remaining data(The page has hardware errors), they only check if
copy was succeeded or not, make the interface more generic by using an
error code when copy fails (-EFAULT) or return zero on success.
Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei•com>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel•org>
---
include/linux/highmem.h | 8 ++++----
mm/khugepaged.c | 4 ++--
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index af03db851a1d..18dc4aca4aa1 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -427,8 +427,8 @@ static inline void copy_highpage(struct page *to, struct page *from)
/*
* If architecture supports machine check exception handling, define the
* #MC versions of copy_user_highpage and copy_highpage. They copy a memory
- * page with #MC in source page (@from) handled, and return the number
- * of bytes not copied if there was a #MC, otherwise 0 for success.
+ * page with #MC in source page (@from) handled, and return -EFAULT if there
+ * was a #MC, otherwise 0 for success.
*/
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
@@ -447,7 +447,7 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from,
if (ret)
memory_failure_queue(page_to_pfn(from), 0);
- return ret;
+ return ret ? -EFAULT : 0;
}
static inline int copy_mc_highpage(struct page *to, struct page *from)
@@ -466,7 +466,7 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
if (ret)
memory_failure_queue(page_to_pfn(from), 0);
- return ret;
+ return ret ? -EFAULT : 0;
}
#else
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index b8452dbdb043..cf1b78eed3c3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -810,7 +810,7 @@ static enum scan_result __collapse_huge_page_copy(pte_t *pte, struct folio *foli
continue;
}
src_page = pte_page(pteval);
- if (copy_mc_user_highpage(page, src_page, src_addr, vma) > 0) {
+ if (copy_mc_user_highpage(page, src_page, src_addr, vma)) {
result = SCAN_COPY_MC;
break;
}
@@ -2143,7 +2143,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
}
for (i = 0; i < nr_pages; i++) {
- if (copy_mc_highpage(dst, folio_page(folio, i)) > 0) {
+ if (copy_mc_highpage(dst, folio_page(folio, i))) {
result = SCAN_COPY_MC;
goto rollback;
}
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage()
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
` (3 preceding siblings ...)
2026-05-18 8:49 ` [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage() Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-27 12:11 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests Ruidong Tian
` (3 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
From: Tong Tiangen <tongtiangen@huawei•com>
Currently, many scenarios that can tolerate memory errors when copying page
have been supported in the kernel[1~5], all of which are implemented by
copy_mc_[user]_highpage(). arm64 should also support this mechanism.
Due to mte, arm64 needs to have its own copy_mc_[user]_highpage()
architecture implementation, macros __HAVE_ARCH_COPY_MC_HIGHPAGE and
__HAVE_ARCH_COPY_MC_USER_HIGHPAGE have been added to control it.
Add new helper copy_mc_page() which provide a page copy implementation with
hardware memory error safe. The code logic of copy_mc_page() is the same as
copy_page(), the main difference is that the ldp insn of copy_mc_page()
contains the fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, therefore, the
main logic is extracted to copy_page_template.S. In addition, the fixup of
MOPS insn is not considered at present.
[Ruidong: add FEAT_MOPS support]
[1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
[2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
[3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
[4] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
[5] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
---
arch/arm64/include/asm/mte.h | 9 ++++
arch/arm64/include/asm/page.h | 10 ++++
arch/arm64/lib/Makefile | 2 +
arch/arm64/lib/copy_mc_page.S | 44 +++++++++++++++++
arch/arm64/lib/copy_page.S | 62 ++----------------------
arch/arm64/lib/copy_page_template.S | 71 +++++++++++++++++++++++++++
arch/arm64/lib/mte.S | 29 +++++++++++
arch/arm64/mm/copypage.c | 75 +++++++++++++++++++++++++++++
include/linux/highmem.h | 8 +++
9 files changed, 253 insertions(+), 57 deletions(-)
create mode 100644 arch/arm64/lib/copy_mc_page.S
create mode 100644 arch/arm64/lib/copy_page_template.S
diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
index 7f7b97e09996..a0b1757f4847 100644
--- a/arch/arm64/include/asm/mte.h
+++ b/arch/arm64/include/asm/mte.h
@@ -98,6 +98,11 @@ static inline bool try_page_mte_tagging(struct page *page)
void mte_zero_clear_page_tags(void *addr);
void mte_sync_tags(pte_t pte, unsigned int nr_pages);
void mte_copy_page_tags(void *kto, const void *kfrom);
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+int mte_copy_mc_page_tags(void *kto, const void *kfrom);
+#endif
+
void mte_thread_init_user(void);
void mte_thread_switch(struct task_struct *next);
void mte_cpu_setup(void);
@@ -134,6 +139,10 @@ static inline void mte_sync_tags(pte_t pte, unsigned int nr_pages)
static inline void mte_copy_page_tags(void *kto, const void *kfrom)
{
}
+static inline int mte_copy_mc_page_tags(void *kto, const void *kfrom)
+{
+ return 0;
+}
static inline void mte_thread_init_user(void)
{
}
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index e25d0d18f6d7..f65818ee614a 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -29,6 +29,16 @@ void copy_user_highpage(struct page *to, struct page *from,
void copy_highpage(struct page *to, struct page *from);
#define __HAVE_ARCH_COPY_HIGHPAGE
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+int copy_mc_page(void *to, const void *from);
+int copy_mc_highpage(struct page *to, struct page *from);
+#define __HAVE_ARCH_COPY_MC_HIGHPAGE
+
+int copy_mc_user_highpage(struct page *to, struct page *from,
+ unsigned long vaddr, struct vm_area_struct *vma);
+#define __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
+#endif
+
struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
unsigned long vaddr);
#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 448c917494f3..1f4c3f743a20 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -7,6 +7,8 @@ lib-y := clear_user.o delay.o copy_from_user.o \
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
+lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
+
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
obj-$(CONFIG_ARM64_MTE) += mte.o
diff --git a/arch/arm64/lib/copy_mc_page.S b/arch/arm64/lib/copy_mc_page.S
new file mode 100644
index 000000000000..ad1371e9e687
--- /dev/null
+++ b/arch/arm64/lib/copy_mc_page.S
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#include <linux/linkage.h>
+#include <linux/const.h>
+#include <asm/assembler.h>
+#include <asm/page.h>
+#include <asm/cpufeature.h>
+#include <asm/alternative.h>
+#include <asm/asm-extable.h>
+#include <asm/asm-uaccess.h>
+
+/*
+ * Copy a page from src to dest (both are page aligned) with memory error safe
+ *
+ * Parameters:
+ * x0 - dest
+ * x1 - src
+ * Returns:
+ * x0 - Return 0 if copy success, or -EFAULT if anything goes wrong
+ * while copying.
+ */
+ .macro ldp1 reg1, reg2, ptr, val
+ KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr, \val])
+ .endm
+
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ USER_CPY(9998f, 0, cpyfprt [\dst]!, [\src]!, \count!)
+ USER_CPY(9998f, 0, cpyfmrt [\dst]!, [\src]!, \count!)
+ USER_CPY(9998f, 0, cpyfert [\dst]!, [\src]!, \count!)
+ .endm
+
+SYM_FUNC_START(__pi_copy_mc_page)
+#include "copy_page_template.S"
+
+ mov x0, #0
+ ret
+
+9998: mov x0, #-EFAULT
+ ret
+
+SYM_FUNC_END(__pi_copy_mc_page)
+SYM_FUNC_ALIAS(copy_mc_page, __pi_copy_mc_page)
+EXPORT_SYMBOL(copy_mc_page)
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index e6374e7e5511..d0186bbf99f1 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -17,65 +17,13 @@
* x0 - dest
* x1 - src
*/
-SYM_FUNC_START(__pi_copy_page)
-#ifdef CONFIG_AS_HAS_MOPS
- .arch_extension mops
-alternative_if_not ARM64_HAS_MOPS
- b .Lno_mops
-alternative_else_nop_endif
-
- mov x2, #PAGE_SIZE
- cpypwn [x0]!, [x1]!, x2!
- cpymwn [x0]!, [x1]!, x2!
- cpyewn [x0]!, [x1]!, x2!
- ret
-.Lno_mops:
-#endif
- ldp x2, x3, [x1]
- ldp x4, x5, [x1, #16]
- ldp x6, x7, [x1, #32]
- ldp x8, x9, [x1, #48]
- ldp x10, x11, [x1, #64]
- ldp x12, x13, [x1, #80]
- ldp x14, x15, [x1, #96]
- ldp x16, x17, [x1, #112]
-
- add x0, x0, #256
- add x1, x1, #128
-1:
- tst x0, #(PAGE_SIZE - 1)
- stnp x2, x3, [x0, #-256]
- ldp x2, x3, [x1]
- stnp x4, x5, [x0, #16 - 256]
- ldp x4, x5, [x1, #16]
- stnp x6, x7, [x0, #32 - 256]
- ldp x6, x7, [x1, #32]
- stnp x8, x9, [x0, #48 - 256]
- ldp x8, x9, [x1, #48]
- stnp x10, x11, [x0, #64 - 256]
- ldp x10, x11, [x1, #64]
- stnp x12, x13, [x0, #80 - 256]
- ldp x12, x13, [x1, #80]
- stnp x14, x15, [x0, #96 - 256]
- ldp x14, x15, [x1, #96]
- stnp x16, x17, [x0, #112 - 256]
- ldp x16, x17, [x1, #112]
-
- add x0, x0, #128
- add x1, x1, #128
-
- b.ne 1b
-
- stnp x2, x3, [x0, #-256]
- stnp x4, x5, [x0, #16 - 256]
- stnp x6, x7, [x0, #32 - 256]
- stnp x8, x9, [x0, #48 - 256]
- stnp x10, x11, [x0, #64 - 256]
- stnp x12, x13, [x0, #80 - 256]
- stnp x14, x15, [x0, #96 - 256]
- stnp x16, x17, [x0, #112 - 256]
+ .macro ldp1 reg1, reg2, ptr, val
+ ldp \reg1, \reg2, [\ptr, \val]
+ .endm
+SYM_FUNC_START(__pi_copy_page)
+#include "copy_page_template.S"
ret
SYM_FUNC_END(__pi_copy_page)
SYM_FUNC_ALIAS(copy_page, __pi_copy_page)
diff --git a/arch/arm64/lib/copy_page_template.S b/arch/arm64/lib/copy_page_template.S
new file mode 100644
index 000000000000..d466b51c8ed9
--- /dev/null
+++ b/arch/arm64/lib/copy_page_template.S
@@ -0,0 +1,71 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ */
+
+/*
+ * Copy a page from src to dest (both are page aligned)
+ *
+ * Parameters:
+ * x0 - dest
+ * x1 - src
+ */
+dstin .req x0
+src .req x1
+
+#ifdef CONFIG_AS_HAS_MOPS
+ .arch_extension mops
+alternative_if_not ARM64_HAS_MOPS
+ b .Lno_mops
+alternative_else_nop_endif
+ mov x2, #PAGE_SIZE
+ cpy1 dst, src, x2
+ b .Lexitfunc
+.Lno_mops:
+#endif
+
+ ldp1 x2, x3, x1, #0
+ ldp1 x4, x5, x1, #16
+ ldp1 x6, x7, x1, #32
+ ldp1 x8, x9, x1, #48
+ ldp1 x10, x11, x1, #64
+ ldp1 x12, x13, x1, #80
+ ldp1 x14, x15, x1, #96
+ ldp1 x16, x17, x1, #112
+
+ add x0, x0, #256
+ add x1, x1, #128
+1:
+ tst x0, #(PAGE_SIZE - 1)
+
+ stnp x2, x3, [x0, #-256]
+ ldp1 x2, x3, x1, #0
+ stnp x4, x5, [x0, #16 - 256]
+ ldp1 x4, x5, x1, #16
+ stnp x6, x7, [x0, #32 - 256]
+ ldp1 x6, x7, x1, #32
+ stnp x8, x9, [x0, #48 - 256]
+ ldp1 x8, x9, x1, #48
+ stnp x10, x11, [x0, #64 - 256]
+ ldp1 x10, x11, x1, #64
+ stnp x12, x13, [x0, #80 - 256]
+ ldp1 x12, x13, x1, #80
+ stnp x14, x15, [x0, #96 - 256]
+ ldp1 x14, x15, x1, #96
+ stnp x16, x17, [x0, #112 - 256]
+ ldp1 x16, x17, x1, #112
+
+ add x0, x0, #128
+ add x1, x1, #128
+
+ b.ne 1b
+
+ stnp x2, x3, [x0, #-256]
+ stnp x4, x5, [x0, #16 - 256]
+ stnp x6, x7, [x0, #32 - 256]
+ stnp x8, x9, [x0, #48 - 256]
+ stnp x10, x11, [x0, #64 - 256]
+ stnp x12, x13, [x0, #80 - 256]
+ stnp x14, x15, [x0, #96 - 256]
+ stnp x16, x17, [x0, #112 - 256]
+.Lexitfunc:
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index 5018ac03b6bf..9d4eeb76a838 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -80,6 +80,35 @@ SYM_FUNC_START(mte_copy_page_tags)
ret
SYM_FUNC_END(mte_copy_page_tags)
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/*
+ * Copy the tags from the source page to the destination one with memory error safe
+ * x0 - address of the destination page
+ * x1 - address of the source page
+ * Returns:
+ * x0 - Return 0 if copy success, or
+ * -EFAULT if anything goes wrong while copying.
+ */
+SYM_FUNC_START(mte_copy_mc_page_tags)
+ mov x2, x0
+ mov x3, x1
+ multitag_transfer_size x5, x6
+1:
+KERNEL_MEM_ERR(2f, ldgm x4, [x3])
+ stgm x4, [x2]
+ add x2, x2, x5
+ add x3, x3, x5
+ tst x2, #(PAGE_SIZE - 1)
+ b.ne 1b
+
+ mov x0, #0
+ ret
+
+2: mov x0, #-EFAULT
+ ret
+SYM_FUNC_END(mte_copy_mc_page_tags)
+#endif
+
/*
* Read tags from a user buffer (one tag per byte) and set the corresponding
* tags at the given kernel address. Used by PTRACE_POKEMTETAGS.
diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c
index cd5912ba617b..9fd773baf17b 100644
--- a/arch/arm64/mm/copypage.c
+++ b/arch/arm64/mm/copypage.c
@@ -72,3 +72,78 @@ void copy_user_highpage(struct page *to, struct page *from,
flush_dcache_page(to);
}
EXPORT_SYMBOL_GPL(copy_user_highpage);
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/*
+ * Return -EFAULT if anything goes wrong while copying page or mte.
+ */
+int copy_mc_highpage(struct page *to, struct page *from)
+{
+ void *kto = page_address(to);
+ void *kfrom = page_address(from);
+ struct folio *src = page_folio(from);
+ struct folio *dst = page_folio(to);
+ unsigned int i, nr_pages;
+ int ret;
+
+ ret = copy_mc_page(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+
+ if (kasan_hw_tags_enabled())
+ page_kasan_tag_reset(to);
+
+ if (!system_supports_mte())
+ return 0;
+
+ if (folio_test_hugetlb(src)) {
+ if (!folio_test_hugetlb_mte_tagged(src) ||
+ from != folio_page(src, 0))
+ return 0;
+
+ WARN_ON_ONCE(!folio_try_hugetlb_mte_tagging(dst));
+
+ /*
+ * Populate tags for all subpages.
+ *
+ * Don't assume the first page is head page since
+ * huge page copy may start from any subpage.
+ */
+ nr_pages = folio_nr_pages(src);
+ for (i = 0; i < nr_pages; i++) {
+ kfrom = page_address(folio_page(src, i));
+ kto = page_address(folio_page(dst, i));
+ ret = mte_copy_mc_page_tags(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+ }
+ folio_set_hugetlb_mte_tagged(dst);
+ } else if (page_mte_tagged(from)) {
+ /* It's a new page, shouldn't have been tagged yet */
+ WARN_ON_ONCE(!try_page_mte_tagging(to));
+
+ ret = mte_copy_mc_page_tags(kto, kfrom);
+ if (ret)
+ return -EFAULT;
+ set_page_mte_tagged(to);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(copy_mc_highpage);
+
+int copy_mc_user_highpage(struct page *to, struct page *from,
+ unsigned long vaddr, struct vm_area_struct *vma)
+{
+ int ret;
+
+ ret = copy_mc_highpage(to, from);
+ if (ret)
+ return ret;
+
+ flush_dcache_page(to);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(copy_mc_user_highpage);
+#endif
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index 18dc4aca4aa1..f168c9d4ad0e 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -424,6 +424,7 @@ static inline void copy_highpage(struct page *to, struct page *from)
#endif
#ifdef copy_mc_to_kernel
+#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
/*
* If architecture supports machine check exception handling, define the
* #MC versions of copy_user_highpage and copy_highpage. They copy a memory
@@ -449,7 +450,9 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from,
return ret ? -EFAULT : 0;
}
+#endif
+#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE
static inline int copy_mc_highpage(struct page *to, struct page *from)
{
unsigned long ret;
@@ -468,20 +471,25 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
return ret ? -EFAULT : 0;
}
+#endif
#else
+#ifndef __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
static inline int copy_mc_user_highpage(struct page *to, struct page *from,
unsigned long vaddr, struct vm_area_struct *vma)
{
copy_user_highpage(to, from, vaddr, vma);
return 0;
}
+#endif
+#ifndef __HAVE_ARCH_COPY_MC_HIGHPAGE
static inline int copy_mc_highpage(struct page *to, struct page *from)
{
copy_highpage(to, from);
return 0;
}
#endif
+#endif
static inline void memcpy_page(struct page *dst_page, size_t dst_off,
struct page *src_page, size_t src_off,
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
` (4 preceding siblings ...)
2026-05-18 8:49 ` [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage() Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-27 13:43 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation Ruidong Tian
` (2 subsequent siblings)
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
Add KUnit tests for copy_page() and copy_mc_page(), modeled after
the existing memcpy_test() style: a static page-aligned src and a
two-page dst, filled with random bytes plus non-zero edges, then
verify byte-for-byte equality and that the adjacent page is
untouched. The copy_mc_page() case additionally checks the return
value is 0 on clean memory and is gated on CONFIG_ARCH_HAS_COPY_MC.
Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
---
lib/tests/memcpy_kunit.c | 67 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 66 insertions(+), 1 deletion(-)
diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
index d36933554e46..85df53ccfb0c 100644
--- a/lib/tests/memcpy_kunit.c
+++ b/lib/tests/memcpy_kunit.c
@@ -493,6 +493,67 @@ static void memmove_overlap_test(struct kunit *test)
}
}
+/* --- Page-sized copy tests --- */
+
+static u8 page_src[PAGE_SIZE] __aligned(PAGE_SIZE);
+static u8 page_dst[PAGE_SIZE * 2] __aligned(PAGE_SIZE);
+static const u8 page_zero[PAGE_SIZE] __aligned(PAGE_SIZE);
+
+static void init_page(struct kunit *test)
+{
+ /* Get many bit patterns. */
+ get_random_bytes(page_src, PAGE_SIZE);
+
+ /* Make sure we have non-zero edges. */
+ set_random_nonzero(test, &page_src[0]);
+ set_random_nonzero(test, &page_src[PAGE_SIZE - 1]);
+
+ /* Explicitly zero the entire destination. */
+ memset(page_dst, 0, ARRAY_SIZE(page_dst));
+}
+
+static void copy_page_test(struct kunit *test)
+{
+ init_page(test);
+
+ /* Copy. */
+ copy_page(page_dst, page_src);
+
+ /* Verify byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst, page_src, PAGE_SIZE), 0,
+ "copy_page content mismatch with random data");
+
+ /* Verify no overflow into second page. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0,
+ "copy_page overflow into adjacent page");
+}
+
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+static void copy_mc_page_test(struct kunit *test)
+{
+ int ret;
+
+ init_page(test);
+
+ /* Copy and check return value. */
+ ret = copy_mc_page(page_dst, page_src);
+ KUNIT_ASSERT_EQ_MSG(test, ret, 0,
+ "copy_mc_page returned %d on clean memory", ret);
+
+ /* Verify byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst, page_src, PAGE_SIZE), 0,
+ "copy_mc_page content mismatch with random data");
+
+ /* Verify no overflow into second page. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0,
+ "copy_mc_page overflow into adjacent page");
+}
+#endif /* CONFIG_ARCH_HAS_COPY_MC */
+
static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(memset_test),
KUNIT_CASE(memcpy_test),
@@ -500,6 +561,10 @@ static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE_SLOW(memmove_test),
KUNIT_CASE_SLOW(memmove_large_test),
KUNIT_CASE_SLOW(memmove_overlap_test),
+ KUNIT_CASE(copy_page_test),
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+ KUNIT_CASE(copy_mc_page_test),
+#endif
{}
};
@@ -510,5 +575,5 @@ static struct kunit_suite memcpy_test_suite = {
kunit_test_suite(memcpy_test_suite);
-MODULE_DESCRIPTION("test cases for memcpy(), memmove(), and memset()");
+MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_page()");
MODULE_LICENSE("GPL");
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
` (5 preceding siblings ...)
2026-05-18 8:49 ` [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-28 3:10 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test Ruidong Tian
2026-05-18 15:05 ` [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Kefeng Wang
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
From: Tong Tiangen <tongtiangen@huawei•com>
The copy_mc_to_kernel() helper is memory copy implementation that handles
source exceptions. It can be used in memory copy scenarios that tolerate
hardware memory errors(e.g: pmem_read/dax_copy_to_iter).
Currently, only x86 and ppc support this helper, Add this for ARM64 as
well, if ARCH_HAS_COPY_MC is defined, by implementing copy_mc_to_kernel()
and memcpy_mc() functions.
Because there is no caller-saved GPR is available for saving "bytes not
copied" in memcpy(), the memcpy_mc() is referenced to the implementation
of copy_from_user(). In addition, the fixup of MOPS insn is not considered
at present.
[Ruidong: refactor memcpy_mc on top of the new memcpy implementation.]
Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
---
arch/arm64/include/asm/string.h | 5 +
arch/arm64/include/asm/uaccess.h | 17 +++
arch/arm64/lib/Makefile | 2 +-
arch/arm64/lib/memcpy.S | 253 +++----------------------------
arch/arm64/lib/memcpy_mc.S | 56 +++++++
arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++++++
mm/kasan/shadow.c | 12 ++
7 files changed, 359 insertions(+), 235 deletions(-)
create mode 100644 arch/arm64/lib/memcpy_mc.S
create mode 100644 arch/arm64/lib/memcpy_template.S
diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/string.h
index 3a3264ff47b9..23eca4fb24fa 100644
--- a/arch/arm64/include/asm/string.h
+++ b/arch/arm64/include/asm/string.h
@@ -35,6 +35,10 @@ extern void *memchr(const void *, int, __kernel_size_t);
extern void *memcpy(void *, const void *, __kernel_size_t);
extern void *__memcpy(void *, const void *, __kernel_size_t);
+#define __HAVE_ARCH_MEMCPY_MC
+extern int memcpy_mc(void *, const void *, __kernel_size_t);
+extern int __memcpy_mc(void *, const void *, __kernel_size_t);
+
#define __HAVE_ARCH_MEMMOVE
extern void *memmove(void *, const void *, __kernel_size_t);
extern void *__memmove(void *, const void *, __kernel_size_t);
@@ -57,6 +61,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t cnt);
*/
#define memcpy(dst, src, len) __memcpy(dst, src, len)
+#define memcpy_mc(dst, src, len) __memcpy_mc(dst, src, len)
#define memmove(dst, src, len) __memmove(dst, src, len)
#define memset(s, c, n) __memset(s, c, n)
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index b0c83a08dda9..93277eca2268 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -499,5 +499,22 @@ static inline size_t probe_subpage_writeable(const char __user *uaddr,
}
#endif /* CONFIG_ARCH_HAS_SUBPAGE_FAULTS */
+#ifdef CONFIG_ARCH_HAS_COPY_MC
+/**
+ * copy_mc_to_kernel - memory copy that handles source exceptions
+ *
+ * @to: destination address
+ * @from: source address
+ * @size: number of bytes to copy
+ *
+ * Return 0 for success, or bytes not copied.
+ */
+static inline unsigned long __must_check
+copy_mc_to_kernel(void *to, const void *from, unsigned long size)
+{
+ return memcpy_mc(to, from, size);
+}
+#define copy_mc_to_kernel copy_mc_to_kernel
+#endif
#endif /* __ASM_UACCESS_H */
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 1f4c3f743a20..a5820e6c33d4 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -7,7 +7,7 @@ lib-y := clear_user.o delay.o copy_from_user.o \
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
-lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
+lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o memcpy_mc.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S
index 9b99106fb95f..ef6aea2de9b4 100644
--- a/arch/arm64/lib/memcpy.S
+++ b/arch/arm64/lib/memcpy.S
@@ -15,247 +15,32 @@
*
*/
-#define L(label) .L ## label
+ .macro ldrb1 reg, addr:vararg
+ ldrb \reg, \addr
+ .endm
-#define dstin x0
-#define src x1
-#define count x2
-#define dst x3
-#define srcend x4
-#define dstend x5
-#define A_l x6
-#define A_lw w6
-#define A_h x7
-#define B_l x8
-#define B_lw w8
-#define B_h x9
-#define C_l x10
-#define C_lw w10
-#define C_h x11
-#define D_l x12
-#define D_h x13
-#define E_l x14
-#define E_h x15
-#define F_l x16
-#define F_h x17
-#define G_l count
-#define G_h dst
-#define H_l src
-#define H_h srcend
-#define tmp1 x14
+ .macro ldr1 reg, addr:vararg
+ ldr \reg, \addr
+ .endm
-/* This implementation handles overlaps and supports both memcpy and memmove
- from a single entry point. It uses unaligned accesses and branchless
- sequences to keep the code small, simple and improve performance.
+ .macro ldp1 reg1, reg2, addr:vararg
+ ldp \reg1, \reg2, \addr
+ .endm
- Copies are split into 3 main cases: small copies of up to 32 bytes, medium
- copies of up to 128 bytes, and large copies. The overhead of the overlap
- check is negligible since it is only required for large copies.
+ .macro ret1
+ ret
+ .endm
- Large copies use a software pipelined loop processing 64 bytes per iteration.
- The destination pointer is 16-byte aligned to minimize unaligned accesses.
- The loop tail is handled by always copying 64 bytes from the end.
-*/
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ cpyp [\dst]!, [\src]!, \count!
+ cpym [\dst]!, [\src]!, \count!
+ cpye [\dst]!, [\src]!, \count!
+ .endm
-SYM_FUNC_START_LOCAL(__pi_memcpy_generic)
- add srcend, src, count
- add dstend, dstin, count
- cmp count, 128
- b.hi L(copy_long)
- cmp count, 32
- b.hi L(copy32_128)
-
- /* Small copies: 0..32 bytes. */
- cmp count, 16
- b.lo L(copy16)
- ldp A_l, A_h, [src]
- ldp D_l, D_h, [srcend, -16]
- stp A_l, A_h, [dstin]
- stp D_l, D_h, [dstend, -16]
- ret
-
- /* Copy 8-15 bytes. */
-L(copy16):
- tbz count, 3, L(copy8)
- ldr A_l, [src]
- ldr A_h, [srcend, -8]
- str A_l, [dstin]
- str A_h, [dstend, -8]
- ret
-
- .p2align 3
- /* Copy 4-7 bytes. */
-L(copy8):
- tbz count, 2, L(copy4)
- ldr A_lw, [src]
- ldr B_lw, [srcend, -4]
- str A_lw, [dstin]
- str B_lw, [dstend, -4]
- ret
-
- /* Copy 0..3 bytes using a branchless sequence. */
-L(copy4):
- cbz count, L(copy0)
- lsr tmp1, count, 1
- ldrb A_lw, [src]
- ldrb C_lw, [srcend, -1]
- ldrb B_lw, [src, tmp1]
- strb A_lw, [dstin]
- strb B_lw, [dstin, tmp1]
- strb C_lw, [dstend, -1]
-L(copy0):
- ret
-
- .p2align 4
- /* Medium copies: 33..128 bytes. */
-L(copy32_128):
- ldp A_l, A_h, [src]
- ldp B_l, B_h, [src, 16]
- ldp C_l, C_h, [srcend, -32]
- ldp D_l, D_h, [srcend, -16]
- cmp count, 64
- b.hi L(copy128)
- stp A_l, A_h, [dstin]
- stp B_l, B_h, [dstin, 16]
- stp C_l, C_h, [dstend, -32]
- stp D_l, D_h, [dstend, -16]
- ret
-
- .p2align 4
- /* Copy 65..128 bytes. */
-L(copy128):
- ldp E_l, E_h, [src, 32]
- ldp F_l, F_h, [src, 48]
- cmp count, 96
- b.ls L(copy96)
- ldp G_l, G_h, [srcend, -64]
- ldp H_l, H_h, [srcend, -48]
- stp G_l, G_h, [dstend, -64]
- stp H_l, H_h, [dstend, -48]
-L(copy96):
- stp A_l, A_h, [dstin]
- stp B_l, B_h, [dstin, 16]
- stp E_l, E_h, [dstin, 32]
- stp F_l, F_h, [dstin, 48]
- stp C_l, C_h, [dstend, -32]
- stp D_l, D_h, [dstend, -16]
- ret
-
- .p2align 4
- /* Copy more than 128 bytes. */
-L(copy_long):
- /* Use backwards copy if there is an overlap. */
- sub tmp1, dstin, src
- cbz tmp1, L(copy0)
- cmp tmp1, count
- b.lo L(copy_long_backwards)
-
- /* Copy 16 bytes and then align dst to 16-byte alignment. */
-
- ldp D_l, D_h, [src]
- and tmp1, dstin, 15
- bic dst, dstin, 15
- sub src, src, tmp1
- add count, count, tmp1 /* Count is now 16 too large. */
- ldp A_l, A_h, [src, 16]
- stp D_l, D_h, [dstin]
- ldp B_l, B_h, [src, 32]
- ldp C_l, C_h, [src, 48]
- ldp D_l, D_h, [src, 64]!
- subs count, count, 128 + 16 /* Test and readjust count. */
- b.ls L(copy64_from_end)
-
-L(loop64):
- stp A_l, A_h, [dst, 16]
- ldp A_l, A_h, [src, 16]
- stp B_l, B_h, [dst, 32]
- ldp B_l, B_h, [src, 32]
- stp C_l, C_h, [dst, 48]
- ldp C_l, C_h, [src, 48]
- stp D_l, D_h, [dst, 64]!
- ldp D_l, D_h, [src, 64]!
- subs count, count, 64
- b.hi L(loop64)
-
- /* Write the last iteration and copy 64 bytes from the end. */
-L(copy64_from_end):
- ldp E_l, E_h, [srcend, -64]
- stp A_l, A_h, [dst, 16]
- ldp A_l, A_h, [srcend, -48]
- stp B_l, B_h, [dst, 32]
- ldp B_l, B_h, [srcend, -32]
- stp C_l, C_h, [dst, 48]
- ldp C_l, C_h, [srcend, -16]
- stp D_l, D_h, [dst, 64]
- stp E_l, E_h, [dstend, -64]
- stp A_l, A_h, [dstend, -48]
- stp B_l, B_h, [dstend, -32]
- stp C_l, C_h, [dstend, -16]
- ret
-
- .p2align 4
-
- /* Large backwards copy for overlapping copies.
- Copy 16 bytes and then align dst to 16-byte alignment. */
-L(copy_long_backwards):
- ldp D_l, D_h, [srcend, -16]
- and tmp1, dstend, 15
- sub srcend, srcend, tmp1
- sub count, count, tmp1
- ldp A_l, A_h, [srcend, -16]
- stp D_l, D_h, [dstend, -16]
- ldp B_l, B_h, [srcend, -32]
- ldp C_l, C_h, [srcend, -48]
- ldp D_l, D_h, [srcend, -64]!
- sub dstend, dstend, tmp1
- subs count, count, 128
- b.ls L(copy64_from_start)
-
-L(loop64_backwards):
- stp A_l, A_h, [dstend, -16]
- ldp A_l, A_h, [srcend, -16]
- stp B_l, B_h, [dstend, -32]
- ldp B_l, B_h, [srcend, -32]
- stp C_l, C_h, [dstend, -48]
- ldp C_l, C_h, [srcend, -48]
- stp D_l, D_h, [dstend, -64]!
- ldp D_l, D_h, [srcend, -64]!
- subs count, count, 64
- b.hi L(loop64_backwards)
-
- /* Write the last iteration and copy 64 bytes from the start. */
-L(copy64_from_start):
- ldp G_l, G_h, [src, 48]
- stp A_l, A_h, [dstend, -16]
- ldp A_l, A_h, [src, 32]
- stp B_l, B_h, [dstend, -32]
- ldp B_l, B_h, [src, 16]
- stp C_l, C_h, [dstend, -48]
- ldp C_l, C_h, [src]
- stp D_l, D_h, [dstend, -64]
- stp G_l, G_h, [dstin, 48]
- stp A_l, A_h, [dstin, 32]
- stp B_l, B_h, [dstin, 16]
- stp C_l, C_h, [dstin]
- ret
-SYM_FUNC_END(__pi_memcpy_generic)
-
-#ifdef CONFIG_AS_HAS_MOPS
- .arch_extension mops
SYM_FUNC_START(__pi_memcpy)
-alternative_if_not ARM64_HAS_MOPS
- b __pi_memcpy_generic
-alternative_else_nop_endif
-
- mov dst, dstin
- cpyp [dst]!, [src]!, count!
- cpym [dst]!, [src]!, count!
- cpye [dst]!, [src]!, count!
- ret
+#include "memcpy_template.S"
SYM_FUNC_END(__pi_memcpy)
-#else
-SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic)
-#endif
SYM_FUNC_ALIAS(__memcpy, __pi_memcpy)
EXPORT_SYMBOL(__memcpy)
diff --git a/arch/arm64/lib/memcpy_mc.S b/arch/arm64/lib/memcpy_mc.S
new file mode 100644
index 000000000000..90624d35af4b
--- /dev/null
+++ b/arch/arm64/lib/memcpy_mc.S
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2021, Arm Limited.
+ *
+ * Adapted from the original at:
+ * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+#include <asm/asm-uaccess.h>
+
+/* Assumptions:
+ *
+ * ARMv8-a, AArch64, unaligned accesses.
+ *
+ */
+
+ .macro ldrb1 reg, addr:vararg
+ KERNEL_MEM_ERR(9998f, ldrb \reg, \addr)
+ .endm
+
+ .macro ldr1 reg, addr:vararg
+ KERNEL_MEM_ERR(9998f, ldr \reg, \addr)
+ .endm
+
+ .macro ldp1 reg1, reg2, addr:vararg
+ KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, \addr)
+ .endm
+
+ .macro ret1
+ mov x0, #0
+ ret
+ .endm
+
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ USER_CPY(9998f, 0, cpyp [\dst]!, [\src]!, \count!)
+ USER_CPY(9996f, 0, cpym [\dst]!, [\src]!, \count!)
+ USER_CPY(9996f, 0, cpye [\dst]!, [\src]!, \count!)
+ .endm
+
+SYM_FUNC_START(__memcpy_mc)
+#include "memcpy_template.S"
+
+ // Exception fixups
+9996: b.cs 9998f
+ // Registers are in Option A format
+ add dst, dst, count
+9998: sub x0, dstend, dstin // bytes not copied
+ ret
+SYM_FUNC_END(__memcpy_mc)
+
+EXPORT_SYMBOL(__memcpy_mc)
+SYM_FUNC_ALIAS_WEAK(memcpy_mc, __memcpy_mc)
+EXPORT_SYMBOL(memcpy_mc)
diff --git a/arch/arm64/lib/memcpy_template.S b/arch/arm64/lib/memcpy_template.S
new file mode 100644
index 000000000000..205516c6e076
--- /dev/null
+++ b/arch/arm64/lib/memcpy_template.S
@@ -0,0 +1,249 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2012-2021, Arm Limited.
+ *
+ * Adapted from the original at:
+ * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
+ */
+
+#include <linux/linkage.h>
+#include <asm/assembler.h>
+
+/* Assumptions:
+ *
+ * ARMv8-a, AArch64, unaligned accesses.
+ *
+ */
+
+#define L(label) .L ## label
+
+#define dstin x0
+#define src x1
+#define count x2
+#define dst x3
+#define srcend x4
+#define dstend x5
+#define A_l x6
+#define A_lw w6
+#define A_h x7
+#define B_l x8
+#define B_lw w8
+#define B_h x9
+#define C_l x10
+#define C_lw w10
+#define C_h x11
+#define D_l x12
+#define D_h x13
+#define E_l x14
+#define E_h x15
+#define F_l x16
+#define F_h x17
+#define G_l count
+#define G_h dst
+#define H_l src
+#define H_h srcend
+#define tmp1 x14
+
+/* This implementation handles overlaps and supports both memcpy and memmove
+ from a single entry point. It uses unaligned accesses and branchless
+ sequences to keep the code small, simple and improve performance.
+
+ Copies are split into 3 main cases: small copies of up to 32 bytes, medium
+ copies of up to 128 bytes, and large copies. The overhead of the overlap
+ check is negligible since it is only required for large copies.
+
+ Large copies use a software pipelined loop processing 64 bytes per iteration.
+ The destination pointer is 16-byte aligned to minimize unaligned accesses.
+ The loop tail is handled by always copying 64 bytes from the end.
+*/
+
+#ifdef CONFIG_AS_HAS_MOPS
+alternative_if_not ARM64_HAS_MOPS
+ b L(no_mops):
+alternative_else_nop_endif
+
+ cpy1 dst, src, count
+ ret1
+#endif
+
+L(no_mops):
+ add srcend, src, count
+ add dstend, dstin, count
+ cmp count, 128
+ b.hi L(copy_long)
+ cmp count, 32
+ b.hi L(copy32_128)
+
+ /* Small copies: 0..32 bytes. */
+ cmp count, 16
+ b.lo L(copy16)
+ ldp1 A_l, A_h, [src]
+ ldp1 D_l, D_h, [srcend, -16]
+ stp A_l, A_h, [dstin]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ /* Copy 8-15 bytes. */
+L(copy16):
+ tbz count, 3, L(copy8)
+ ldr1 A_l, [src]
+ ldr1 A_h, [srcend, -8]
+ str A_l, [dstin]
+ str A_h, [dstend, -8]
+ ret1
+
+ .p2align 3
+ /* Copy 4-7 bytes. */
+L(copy8):
+ tbz count, 2, L(copy4)
+ ldr1 A_lw, [src]
+ ldr1 B_lw, [srcend, -4]
+ str A_lw, [dstin]
+ str B_lw, [dstend, -4]
+ ret1
+
+ /* Copy 0..3 bytes using a branchless sequence. */
+L(copy4):
+ cbz count, L(copy0)
+ lsr tmp1, count, 1
+ ldrb1 A_lw, [src]
+ ldrb1 C_lw, [srcend, -1]
+ ldrb1 B_lw, [src, tmp1]
+ strb A_lw, [dstin]
+ strb B_lw, [dstin, tmp1]
+ strb C_lw, [dstend, -1]
+L(copy0):
+ ret1
+
+ .p2align 4
+ /* Medium copies: 33..128 bytes. */
+L(copy32_128):
+ ldp1 A_l, A_h, [src]
+ ldp1 B_l, B_h, [src, 16]
+ ldp1 C_l, C_h, [srcend, -32]
+ ldp1 D_l, D_h, [srcend, -16]
+ cmp count, 64
+ b.hi L(copy128)
+ stp A_l, A_h, [dstin]
+ stp B_l, B_h, [dstin, 16]
+ stp C_l, C_h, [dstend, -32]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+ /* Copy 65..128 bytes. */
+L(copy128):
+ ldp1 E_l, E_h, [src, 32]
+ ldp1 F_l, F_h, [src, 48]
+ cmp count, 96
+ b.ls L(copy96)
+ ldp1 G_l, G_h, [srcend, -64]
+ ldp1 H_l, H_h, [srcend, -48]
+ stp G_l, G_h, [dstend, -64]
+ stp H_l, H_h, [dstend, -48]
+L(copy96):
+ stp A_l, A_h, [dstin]
+ stp B_l, B_h, [dstin, 16]
+ stp E_l, E_h, [dstin, 32]
+ stp F_l, F_h, [dstin, 48]
+ stp C_l, C_h, [dstend, -32]
+ stp D_l, D_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+ /* Copy more than 128 bytes. */
+L(copy_long):
+ /* Use backwards copy if there is an overlap. */
+ sub tmp1, dstin, src
+ cbz tmp1, L(copy0)
+ cmp tmp1, count
+ b.lo L(copy_long_backwards)
+
+ /* Copy 16 bytes and then align dst to 16-byte alignment. */
+
+ ldp1 D_l, D_h, [src]
+ and tmp1, dstin, 15
+ bic dst, dstin, 15
+ sub src, src, tmp1
+ add count, count, tmp1 /* Count is now 16 too large. */
+ ldp1 A_l, A_h, [src, 16]
+ stp D_l, D_h, [dstin]
+ ldp1 B_l, B_h, [src, 32]
+ ldp1 C_l, C_h, [src, 48]
+ ldp1 D_l, D_h, [src, 64]!
+ subs count, count, 128 + 16 /* Test and readjust count. */
+ b.ls L(copy64_from_end)
+
+L(loop64):
+ stp A_l, A_h, [dst, 16]
+ ldp1 A_l, A_h, [src, 16]
+ stp B_l, B_h, [dst, 32]
+ ldp1 B_l, B_h, [src, 32]
+ stp C_l, C_h, [dst, 48]
+ ldp1 C_l, C_h, [src, 48]
+ stp D_l, D_h, [dst, 64]!
+ ldp1 D_l, D_h, [src, 64]!
+ subs count, count, 64
+ b.hi L(loop64)
+
+ /* Write the last iteration and copy 64 bytes from the end. */
+L(copy64_from_end):
+ ldp1 E_l, E_h, [srcend, -64]
+ stp A_l, A_h, [dst, 16]
+ ldp1 A_l, A_h, [srcend, -48]
+ stp B_l, B_h, [dst, 32]
+ ldp1 B_l, B_h, [srcend, -32]
+ stp C_l, C_h, [dst, 48]
+ ldp1 C_l, C_h, [srcend, -16]
+ stp D_l, D_h, [dst, 64]
+ stp E_l, E_h, [dstend, -64]
+ stp A_l, A_h, [dstend, -48]
+ stp B_l, B_h, [dstend, -32]
+ stp C_l, C_h, [dstend, -16]
+ ret1
+
+ .p2align 4
+
+ /* Large backwards copy for overlapping copies.
+ Copy 16 bytes and then align dst to 16-byte alignment. */
+L(copy_long_backwards):
+ ldp1 D_l, D_h, [srcend, -16]
+ and tmp1, dstend, 15
+ sub srcend, srcend, tmp1
+ sub count, count, tmp1
+ ldp1 A_l, A_h, [srcend, -16]
+ stp D_l, D_h, [dstend, -16]
+ ldp1 B_l, B_h, [srcend, -32]
+ ldp1 C_l, C_h, [srcend, -48]
+ ldp1 D_l, D_h, [srcend, -64]!
+ sub dstend, dstend, tmp1
+ subs count, count, 128
+ b.ls L(copy64_from_start)
+
+L(loop64_backwards):
+ stp A_l, A_h, [dstend, -16]
+ ldp1 A_l, A_h, [srcend, -16]
+ stp B_l, B_h, [dstend, -32]
+ ldp1 B_l, B_h, [srcend, -32]
+ stp C_l, C_h, [dstend, -48]
+ ldp1 C_l, C_h, [srcend, -48]
+ stp D_l, D_h, [dstend, -64]!
+ ldp1 D_l, D_h, [srcend, -64]!
+ subs count, count, 64
+ b.hi L(loop64_backwards)
+
+ /* Write the last iteration and copy 64 bytes from the start. */
+L(copy64_from_start):
+ ldp1 G_l, G_h, [src, 48]
+ stp A_l, A_h, [dstend, -16]
+ ldp1 A_l, A_h, [src, 32]
+ stp B_l, B_h, [dstend, -32]
+ ldp1 B_l, B_h, [src, 16]
+ stp C_l, C_h, [dstend, -48]
+ ldp1 C_l, C_h, [src]
+ stp D_l, D_h, [dstend, -64]
+ stp G_l, G_h, [dstin, 48]
+ stp A_l, A_h, [dstin, 32]
+ stp B_l, B_h, [dstin, 16]
+ stp C_l, C_h, [dstin]
+ ret1
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index d286e0a04543..3128f0d9cc46 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -79,6 +79,18 @@ void *memcpy(void *dest, const void *src, size_t len)
}
#endif
+#ifdef __HAVE_ARCH_MEMCPY_MC
+#undef memcpy_mc
+int memcpy_mc(void *dest, const void *src, size_t len)
+{
+ if (!kasan_check_range(src, len, false, _RET_IP_) ||
+ !kasan_check_range(dest, len, true, _RET_IP_))
+ return (int)len;
+
+ return __memcpy_mc(dest, src, len);
+}
+#endif
+
void *__asan_memset(void *addr, int c, ssize_t len)
{
if (!kasan_check_range(addr, len, true, _RET_IP_))
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
` (6 preceding siblings ...)
2026-05-18 8:49 ` [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation Ruidong Tian
@ 2026-05-18 8:49 ` Ruidong Tian
2026-05-28 3:17 ` Shuai Xue
2026-05-18 15:05 ` [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Kefeng Wang
8 siblings, 1 reply; 20+ messages in thread
From: Ruidong Tian @ 2026-05-18 8:49 UTC (permalink / raw)
To: catalin.marinas, will, rafael, tony.luck, guohanjun, mchehab,
xueshuai, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
tianruidong
memcpy_mc() is the Machine-Check safe memcpy variant that returns the
number of bytes NOT copied on a hardware memory error, or 0 on success.
Add two test cases modeled after the existing memcpy_test() and
memcpy_large_test() implementations:
Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
---
lib/tests/memcpy_kunit.c | 113 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 112 insertions(+), 1 deletion(-)
diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
index 85df53ccfb0c..b4b2dafb50f1 100644
--- a/lib/tests/memcpy_kunit.c
+++ b/lib/tests/memcpy_kunit.c
@@ -552,6 +552,115 @@ static void copy_mc_page_test(struct kunit *test)
memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0,
"copy_mc_page overflow into adjacent page");
}
+/*
+ * memcpy_mc() is a Machine-Check safe memcpy variant.
+ * Signature: int memcpy_mc(void *dst, const void *src, size_t len)
+ * Returns: 0 on success, or number of bytes NOT copied on MC error.
+ *
+ * In the normal (no-poison) path it must behave identically to memcpy()
+ * and always return 0.
+ */
+static void memcpy_mc_test(struct kunit *test)
+{
+#define TEST_OP "memcpy_mc"
+ struct some_bytes control = {
+ .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes zero = { };
+ struct some_bytes middle = {
+ .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes three = {
+ .data = { 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ },
+ };
+ struct some_bytes dest = { };
+ int ret, count;
+ u8 *ptr;
+
+ /* Verify static initializers. */
+ check(control, 0x20);
+ check(zero, 0);
+ compare("static initializers", dest, zero);
+
+ /* Verify assignment. */
+ dest = control;
+ compare("direct assignment", dest, control);
+
+ /* Verify complete overwrite. */
+ ret = memcpy_mc(dest.data, control.data, sizeof(dest.data));
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("complete overwrite", dest, control);
+
+ /* Verify middle overwrite: 7 bytes at offset 12. */
+ dest = control;
+ ret = memcpy_mc(dest.data + 12, zero.data, 7);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("middle overwrite", dest, middle);
+
+ /* Verify zero-length copy is a no-op. */
+ dest = control;
+ ret = memcpy_mc(dest.data, zero.data, 0);
+ KUNIT_ASSERT_EQ(test, ret, 0);
+ compare("zero length", dest, control);
+
+ /* Verify argument side-effects aren't repeated. */
+ dest = control;
+ ptr = dest.data;
+ count = 1;
+ memcpy(ptr++, zero.data, count++);
+ ptr += 8;
+ memcpy(ptr++, zero.data, count++);
+ compare("argument side-effects", dest, three);
+#undef TEST_OP
+}
+
+static void memcpy_mc_large_test(struct kunit *test)
+{
+ init_large(test);
+
+ /* Sweep 1..1024 bytes x shifting offset to cover all template paths. */
+ for (int bytes = 1; bytes <= ARRAY_SIZE(large_src); bytes++) {
+ for (int offset = 0; offset < ARRAY_SIZE(large_src); offset++) {
+ int right_zero_pos = offset + bytes;
+ int right_zero_size = ARRAY_SIZE(large_dst) - right_zero_pos;
+ int ret;
+
+ ret = memcpy_mc(large_dst + offset, large_src, bytes);
+ KUNIT_ASSERT_EQ_MSG(test, ret, 0,
+ "memcpy_mc returned %d with size %d at offset %d",
+ ret, bytes, offset);
+
+ /* No write before copy area. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(large_dst, large_zero, offset), 0,
+ "with size %d at offset %d", bytes, offset);
+ /* No write after copy area. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(&large_dst[right_zero_pos], large_zero,
+ right_zero_size), 0,
+ "with size %d at offset %d", bytes, offset);
+ /* Byte-for-byte exact. */
+ KUNIT_ASSERT_EQ_MSG(test,
+ memcmp(large_dst + offset, large_src, bytes), 0,
+ "with size %d at offset %d", bytes, offset);
+
+ memset(large_dst + offset, 0, bytes);
+ }
+ cond_resched();
+ }
+}
#endif /* CONFIG_ARCH_HAS_COPY_MC */
static struct kunit_case memcpy_test_cases[] = {
@@ -564,6 +673,8 @@ static struct kunit_case memcpy_test_cases[] = {
KUNIT_CASE(copy_page_test),
#ifdef CONFIG_ARCH_HAS_COPY_MC
KUNIT_CASE(copy_mc_page_test),
+ KUNIT_CASE(memcpy_mc_test),
+ KUNIT_CASE_SLOW(memcpy_mc_large_test),
#endif
{}
};
@@ -575,5 +686,5 @@ static struct kunit_suite memcpy_test_suite = {
kunit_test_suite(memcpy_test_suite);
-MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset() and copy_page()");
+MODULE_DESCRIPTION("test cases for memcpy(), memmove(), memset(), copy_page() and memcpy_mc()");
MODULE_LICENSE("GPL");
--
2.39.3
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
` (7 preceding siblings ...)
2026-05-18 8:49 ` [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test Ruidong Tian
@ 2026-05-18 15:05 ` Kefeng Wang
2026-06-05 7:33 ` Ruidong Tian
8 siblings, 1 reply; 20+ messages in thread
From: Kefeng Wang @ 2026-05-18 15:05 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, xueshuai, tongtiangen, james.morse, robin.murphy,
andreyknvl, dvyukov, vincenzo.frascino, mpe, npiggin,
ryabinin.a.a, glider, christophe.leroy, aneesh.kumar,
naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/2026 4:49 PM, Ruidong Tian wrote:
> This series continues Tong Tiangen's work on arm64 ARCH_HAS_COPY_MC
> support. We encounter the same problem, and from a forward-looking
> perspective, large-memory ARM machines such as Grace and Vera will suffer
> more from this class of issues, which motivates us to push this feature
> upstream.
>
> Problem
> =========
> With the increase of memory capacity and density, the probability of memory
> error also increases. The increasing size and density of server RAM in data
> centers and clouds have shown increased uncorrectable memory errors.
>
> Currently, more and more scenarios that can tolerate memory errors, such as
> COW[1,2], KSM copy[3], coredump copy[4], khugepaged[5,6], uaccess copy[7],
> etc.
We have encountered more scenarios and have made more enhancements, eg,
658be46520ce mm: support poison recovery from copy_present_page()
aa549f923f5e mm: support poison recovery from do_cow_fault()
f00b295b9b61 fs: hugetlbfs: support poisoned recover from
hugetlbfs_migrate_folio()
060913999d7a mm: migrate: support poisoned recover from migrate folio
Hope that the architecture-related sections can receive relevant reviews
and responses.
Thanks.
> Solution
> =========
>
> This patchset introduces a new processing framework on ARM64, which enables
> ARM64 to support error recovery in the above scenarios, and more scenarios
> can be expanded based on this in the future.
>
> In arm64, memory error handling in do_sea(), which is divided into two cases:
> 1. If the user state consumed the memory errors, the solution is to kill
> the user process and isolate the error page.
> 2. If the kernel state consumed the memory errors, the solution is to
> panic.
>
> For case 2, Undifferentiated panic may not be the optimal choice, as it can
> be handled better. In some scenarios, we can avoid panic, such as uaccess,
> if the uaccess fails due to memory error, only the user process will be
> affected, returning an error to the caller and isolating the user page with
> hardware memory errors is a better choice.
>
> [1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
> [2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
> [3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
> [4] commit 245f09226893 ("mm: hwpoison: coredump: support recovery from dump_user_range()")
> [5] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
> [6] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
> [7] commit 278b917f8cb9 ("x86/mce: Add _ASM_EXTABLE_CPY for copy user access")
>
> ------------------
> Test result:
>
> Tested on Kunpeng 920.
>
> 1. copy_page(), copy_mc_page() basic function test pass, and the disassembly
> contents remains the same before and after refactor.
>
> 2. copy_to/from_user() access kernel NULL pointer raise translation fault
> and dump error message then die(), test pass.
>
> 3. Test following scenarios: copy_from_user(), get_user(), COW.
>
> Before patched: trigger a hardware memory error then panic.
> After patched: trigger a hardware memory error without panic.
>
> Testing step:
> step1. start an user-process.
> step2. poison(einj) the user-process's page.
> step3: user-process access the poison page in kernel mode, then trigger SEA.
> step4: the kernel will not panic, only the user process is killed, the poison
> page is isolated. (before patched, the kernel will panic in do_sea())
>
> The above tests can also be reproduced using ras-tools, which provides
> einj-based injection and validation for uaccess and COW scenarios.
> Example usage:
>
> einj_mem_uc futex # get_user
> einj_mem_uc copyin # copy_to_user
> einj_mem_uc copy-on-write # COW
>
> Link: https://git.kernel.org/pub/scm/linux/kernel/git/aegl/ras-tools.git
>
> ------------------
>
> Benefits
> =========
> According to Huawei's statistics from their storage products, memory errors
> triggered in kernel-mode by COW and page cache read (uaccess) scenarios
> account for more than 50%. With this patchset deployed, all kernel panics
> caused by COW and page cache memory errors are eliminated.
> Alibaba Cloud has also observed memory errors occurring in uaccess contexts.
>
> Since V13:
> 1. Changed MC-safe functions to return an error rather than kill the user
> process. When a user program invokes a syscall and the kernel encounters
> a memory error during uaccess, killing the process is unexpected; the
> syscall should return an error.
> 2. Added FEAT_MOPS support for the copy_page_mc paths.
> 3. Refactored copy_page() and memcpy() on top of the shared memcpy_template,
> reducing duplicated assembly code.
>
> Since v12:
> Thanks to the suggestions of Jonathan, Mark, and Mauro, the following modifications
> are made:
> 1. Rebase to latest kernel version.
> 2. Patch1, add Jonathan's and Mauro's review-by.
> 3. Patch2, modified do_apei_claim_sea() according to Mark's and Jonathan's suggestions,
> and optimized the commit message according to Mark's suggestions(Added description of
> the impact on regular copy_to_user()).
> 4. Patch3, optimized the commit message according to Mauro's suggestions and add Jonathan's
> review-by.
> 5. Patch4, modified copy_mc_user_highpage() and Optimized the commit message according to
> Jonathan's suggestions(no functional changes).
> 6. Patch5, optimized the commit message according to Mauro's suggestions.
> 7. Patch4/5, FEAT_MOPS is added to the code logic. Currently, the fixup is not performed
> on the MOPS instruction.
> 8. Remove patch6 in v12 according to Jonathan's suggestions.
>
> Since v11:
> 1. Rebase to latest kernel version 6.9-rc1.
> 2. Add patch 5, Since the problem described in "Since V10 Besides 3" has
> been solved in a50026bdb867 ('iov_iter: get rid of 'copy_mc' flag').
> 3. Add the benefit of applying the patch set to our company to the description of patch0.
>
> Since V10:
> Accroding Mark's suggestion:
> 1. Merge V10's patch2 and patch3 to V11's patch2.
> 2. Patch2(V11): use new fixup_type for ld* in copy_to_user(), fix fatal
> issues (NULL kernel pointeraccess) been fixup incorrectly.
> 3. Patch2(V11): refactoring the logic of do_sea().
> 4. Patch4(V11): Remove duplicate assembly logic and remove do_mte().
>
> Besides:
> 1. Patch2(V11): remove st* insn's fixup, st* generally not trigger memory error.
> 2. Split a part of the logic of patch2(V11) to patch5(V11), for detail,
> see patch5(V11)'s commit msg.
> 3. Remove patch6(v10) “arm64: introduce copy_mc_to_kernel() implementation”.
> During modification, some problems that cannot be solved in a short
> period are found. The patch will be released after the problems are
> solved.
> 4. Add test result in this patch.
> 5. Modify patchset title, do not use machine check and remove "-next".
>
> Since V9:
> 1. Rebase to latest kernel version 6.8-rc2.
> 2. Add patch 6/6 to support copy_mc_to_kernel().
>
> Since V8:
> 1. Rebase to latest kernel version and fix topo in some of the patches.
> 2. According to the suggestion of Catalin, I attempted to modify the
> return value of function copy_mc_[user]_highpage() to bytes not copied.
> During the modification process, I found that it would be more
> reasonable to return -EFAULT when copy error occurs (referring to the
> newly added patch 4).
>
> For ARM64, the implementation of copy_mc_[user]_highpage() needs to
> consider MTE. Considering the scenario where data copying is successful
> but the MTE tag copying fails, it is also not reasonable to return
> bytes not copied.
> 3. Considering the recent addition of machine check safe support for
> multiple scenarios, modify commit message for patch 5 (patch 4 for V8).
>
> Since V7:
> Currently, there are patches supporting recover from poison
> consumption for the cow scenario[1]. Therefore, Supporting cow
> scenario under the arm64 architecture only needs to modify the relevant
> code under the arch/.
> [1]https://lore.kernel.org/lkml/20221031201029.102123-1-tony.luck@intel.com/
>
> Since V6:
> Resend patches that are not merged into the mainline in V6.
>
> Since V5:
> 1. Add patch2/3 to add uaccess assembly helpers.
> 2. Optimize the implementation logic of arm64_do_kernel_sea() in patch8.
> 3. Remove kernel access fixup in patch9.
> All suggestion are from Mark.
>
> Since V4:
> 1. According Michael's suggestion, add patch5.
> 2. According Mark's suggestiog, do some restructuring to arm64
> extable, then a new adaptation of machine check safe support is made based
> on this.
> 3. According Mark's suggestion, support machine check safe in do_mte() in
> cow scene.
> 4. In V4, two patches have been merged into -next, so V5 not send these
> two patches.
>
> Since V3:
> 1. According to Robin's suggestion, direct modify user_ldst and
> user_ldp in asm-uaccess.h and modify mte.S.
> 2. Add new macro USER_MC in asm-uaccess.h, used in copy_from_user.S
> and copy_to_user.S.
> 3. According to Robin's suggestion, using micro in copy_page_mc.S to
> simplify code.
> 4. According to KeFeng's suggestion, modify powerpc code in patch1.
> 5. According to KeFeng's suggestion, modify mm/extable.c and some code
> optimization.
>
> Since V2:
> 1. According to Mark's suggestion, all uaccess can be recovered due to
> memory error.
> 2. Scenario pagecache reading is also supported as part of uaccess
> (copy_to_user()) and duplication code problem is also solved.
> Thanks for Robin's suggestion.
> 3. According Mark's suggestion, update commit message of patch 2/5.
> 4. According Borisllav's suggestion, update commit message of patch 1/5.
>
> Since V1:
> 1.Consistent with PPC/x86, Using CONFIG_ARCH_HAS_COPY_MC instead of
> ARM64_UCE_KERNEL_RECOVERY.
> 2.Add two new scenes, cow and pagecache reading.
> 3.Fix two small bug(the first two patch).
>
> V1 in here:
> https://lore.kernel.org/lkml/20220323033705.3966643-1-tongtiangen@huawei.com/
>
> Ruidong Tian (3):
> ACPI: APEI: GHES: use exception context to gate SIGBUS on poison
> consumption
> lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
> lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
>
> Tong Tiangen (5):
> uaccess: add generic fallback version of copy_mc_to_user()
> arm64: add support for ARCH_HAS_COPY_MC
> mm/hwpoison: return -EFAULT when copy fail in
> copy_mc_[user]_highpage()
> arm64: support copy_mc_[user]_highpage()
> arm64: introduce copy_mc_to_kernel() implementation
>
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/asm-extable.h | 22 ++-
> arch/arm64/include/asm/asm-uaccess.h | 4 +
> arch/arm64/include/asm/extable.h | 1 +
> arch/arm64/include/asm/mte.h | 9 +
> arch/arm64/include/asm/page.h | 10 ++
> arch/arm64/include/asm/string.h | 5 +
> arch/arm64/include/asm/uaccess.h | 17 ++
> arch/arm64/kernel/acpi.c | 2 +-
> arch/arm64/lib/Makefile | 2 +
> arch/arm64/lib/copy_mc_page.S | 44 +++++
> arch/arm64/lib/copy_page.S | 62 +------
> arch/arm64/lib/copy_page_template.S | 71 ++++++++
> arch/arm64/lib/copy_to_user.S | 10 +-
> arch/arm64/lib/memcpy.S | 253 ++-------------------------
> arch/arm64/lib/memcpy_mc.S | 56 ++++++
> arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++
> arch/arm64/lib/mte.S | 29 +++
> arch/arm64/mm/copypage.c | 75 ++++++++
> arch/arm64/mm/extable.c | 21 +++
> arch/arm64/mm/fault.c | 30 +++-
> arch/powerpc/include/asm/uaccess.h | 1 +
> arch/x86/include/asm/uaccess.h | 1 +
> drivers/acpi/apei/ghes.c | 36 ++--
> include/acpi/ghes.h | 6 +-
> include/linux/highmem.h | 16 +-
> include/linux/uaccess.h | 8 +
> lib/tests/memcpy_kunit.c | 178 ++++++++++++++++++-
> mm/kasan/shadow.c | 12 ++
> mm/khugepaged.c | 4 +-
> 30 files changed, 904 insertions(+), 331 deletions(-)
> create mode 100644 arch/arm64/lib/copy_mc_page.S
> create mode 100644 arch/arm64/lib/copy_page_template.S
> create mode 100644 arch/arm64/lib/memcpy_mc.S
> create mode 100644 arch/arm64/lib/memcpy_template.S
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user()
2026-05-18 8:49 ` [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user() Ruidong Tian
@ 2026-05-27 9:16 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-27 9:16 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
Mauro Carvalho Chehab, Jonathan Cameron
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> From: Tong Tiangen <tongtiangen@huawei•com>
>
> x86/powerpc has it's implementation of copy_mc_to_user(), we add generic
> fallback in include/linux/uaccess.h prepare for other architechures to
> enable CONFIG_ARCH_HAS_COPY_MC.
>
> Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
> Acked-by: Michael Ellerman <mpe@ellerman•id.au>
> Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel•org>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei•com>
> ---
> arch/powerpc/include/asm/uaccess.h | 1 +
> arch/x86/include/asm/uaccess.h | 1 +
> include/linux/uaccess.h | 8 ++++++++
> 3 files changed, 10 insertions(+)
>
> diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h
> index e98c628e3899..073de098d45a 100644
> --- a/arch/powerpc/include/asm/uaccess.h
> +++ b/arch/powerpc/include/asm/uaccess.h
> @@ -432,6 +432,7 @@ copy_mc_to_user(void __user *to, const void *from, unsigned long n)
>
> return n;
> }
> +#define copy_mc_to_user copy_mc_to_user
> #endif
>
> extern size_t copy_from_user_flushcache(void *dst, const void __user *src, size_t size);
> diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
> index 3a0dd3c2b233..308b0854d1d5 100644
> --- a/arch/x86/include/asm/uaccess.h
> +++ b/arch/x86/include/asm/uaccess.h
> @@ -496,6 +496,7 @@ copy_mc_to_kernel(void *to, const void *from, unsigned len);
>
> unsigned long __must_check
> copy_mc_to_user(void __user *to, const void *from, unsigned len);
> +#define copy_mc_to_user copy_mc_to_user
> #endif
>
> /*
> diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
> index 56328601218c..c53a65394f80 100644
> --- a/include/linux/uaccess.h
> +++ b/include/linux/uaccess.h
> @@ -250,6 +250,14 @@ copy_mc_to_kernel(void *dst, const void *src, size_t cnt)
> }
> #endif
>
> +#ifndef copy_mc_to_user
> +static inline unsigned long __must_check
> +copy_mc_to_user(void *dst, const void *src, size_t cnt)
> +{
The prototype of this generic fallback diverges from both copy_to_user()
and the existing arch implementations of copy_mc_to_user() in two
non-trivial ways:
1. The destination pointer drops the __user annotation.
copy_to_user() is declared as:
unsigned long copy_to_user(void __user *to,
const void *from,
unsigned long n);
and arch/x86/include/asm/uaccess.h declares the arch version as:
unsigned long copy_mc_to_user(void __user *to,
const void *from,
unsigned long len);
With this fallback typed as 'void *dst', sparse will silently
accept a kernel pointer being passed in, and the address-space
check is effectively lost for any architecture that ends up using
the fallback. The annotation is the whole point of __user; please
keep it.
2. The size parameter is 'size_t cnt' instead of 'unsigned long n'.
copy_to_user() and the x86/ppc copy_mc_to_user() implementations
all use 'unsigned long'. On all supported arches this happens to
be the same width as size_t, so it is not a functional bug, but
the inconsistent signature is a foot-gun: when an arch later
switches from the fallback to its own implementation, every
caller signature has to be re-checked.
Please align the fallback prototype exactly with the arch versions:
#ifndef copy_mc_to_user
static inline unsigned long __must_check
copy_mc_to_user(void __user *dst, const void *src, unsigned long cnt)
{
return copy_to_user(dst, src, cnt);
}
#endif
With that fixed:
Reviewed-by: Shuai Xue <xueshuai@linux•alibaba.com>
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption
2026-05-18 8:49 ` [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption Ruidong Tian
@ 2026-05-27 9:34 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-27 9:34 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> When a GHES SEA (Synchronous External Abort) fires while the CPU
> was executing in kernel mode, it typically means that kernel code
> itself consumed a poisoned memory location -- e.g. copy_from_user()
> / copy_to_user() invoked from a ioctl() or write() syscall touched
> a poisoned user page or page-cache page on behalf of the task.
>
> The expected behaviour in that case is that the faulting kernel
> helper returns via its extable fixup and the syscall returns an
> error (e.g. -EFAULT) to user space. It is NOT appropriate to deliver
> SIGBUS to the current task: the task did not directly dereference
> the poisoned address, the kernel did on its behalf, and the kernel
> is able to recover.
>
> Up to now ghes_handle_memory_failure() unconditionally promoted any
> synchronous recoverable memory error to MF_ACTION_REQUIRED, which
> ends up SIGBUS on current -- regardless of whether the poison was
> consumed from user space or from inside the kernel on the task's
> behalf. That kills tasks that should instead have seen a plain
> syscall error.
>
> To fix this, the execution mode in which the exception was taken
> must be captured at the arch-level entry point, where pt_regs (and
> hence user_mode(regs)) are still available. The estatus node that
> later drains the error in IRQ / process context no longer has
> access to the original regs.
>
> Introduce:
>
> enum context { NO_USE = -1, IN_KERNEL = 0, IN_USER = 1 };
>
> and plumb the value all the way down to the queued estatus node:
>
> * Add an 'enum context context' field to struct ghes_estatus_node
> and record it in ghes_in_nmi_queue_one_entry().
> * Extend ghes_notify_sea() and the internal
> ghes_in_nmi_spool_from_list() with an enum context parameter.
>
> Then consume the recorded context in ghes_handle_memory_failure()
> for the GHES_SEV_RECOVERABLE / sync path:
>
> flags = sync && context == IN_USER ? MF_ACTION_REQUIRED : 0;
>
> i.e. MF_ACTION_REQUIRED (and thus SIGBUS via the task_work path) is
> only raised for user-mode poison consumption. Synchronous errors
> taken in kernel mode fall back to memory_failure_queue() with
> flags=0, asynchronously isolating the poisoned page while letting
> the faulting kernel helper's extable fixup return -EFAULT
> to user space.
>
> Paths that pass NO_USE are unaffected:
> sync is false for them, so flags stays 0 as before.
>
> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
> ---
> arch/arm64/kernel/acpi.c | 2 +-
> drivers/acpi/apei/ghes.c | 36 ++++++++++++++++++++----------------
> include/acpi/ghes.h | 6 ++++--
> 3 files changed, 25 insertions(+), 19 deletions(-)
>
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 5891f92c2035..40d4a2913d51 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -409,7 +409,7 @@ int apei_claim_sea(struct pt_regs *regs)
> */
> local_daif_restore(DAIF_ERRCTX);
> nmi_enter();
> - err = ghes_notify_sea();
> + err = ghes_notify_sea(user_mode(regs));
apei_claim_sea() explicitly documents that @regs may be NULL when
called from process context, and arch/arm64/kvm/mmu.c does exactly
that:
if (apei_claim_sea(NULL) == 0)
return 1;
user_mode(regs) dereferences regs->pstate, so any SEA taken on a
KVM-enabled arm64 host with this patch applied will oops immediately.
This also relies on bool 1 == IN_USER == 1 and bool 0 == IN_KERNEL == 0
purely by coincidence. The day someone reorders the enumerators or
adds a new value before IN_USER, the policy silently flips (kernel-
mode poison would start raising SIGBUS again) with no warning from
the compiler. Please convert explicitly with a ternary as shown
above; problem #1 then falls out for free.
Please handle the NULL case explicitly, e.g.:
err = ghes_notify_sea(regs ? (user_mode(regs) ? GHES_CTX_USER
: GHES_CTX_KERNEL)
: GHES_CTX_NA);
> nmi_exit();
>
> /*
> diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
> index 3236a3ce79d6..6f265893cddf 100644
> --- a/drivers/acpi/apei/ghes.c
> +++ b/drivers/acpi/apei/ghes.c
> @@ -529,7 +529,7 @@ static bool ghes_do_memory_failure(u64 physical_addr, int flags)
> }
>
> static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
> - int sev, bool sync)
> + int sev, bool sync, enum context context)
> {
> int flags = -1;
> int sec_sev = ghes_severity(gdata->error_severity);
> @@ -543,7 +543,7 @@ static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
> (gdata->flags & CPER_SEC_ERROR_THRESHOLD_EXCEEDED))
> flags = MF_SOFT_OFFLINE;
> if (sev == GHES_SEV_RECOVERABLE && sec_sev == GHES_SEV_RECOVERABLE)
> - flags = sync ? MF_ACTION_REQUIRED : 0;
> + flags = sync && context == IN_USER ? MF_ACTION_REQUIRED : 0;
>
> if (flags != -1)
> return ghes_do_memory_failure(mem_err->physical_addr, flags);
> @@ -552,10 +552,10 @@ static bool ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata,
> }
>
> static bool ghes_handle_arm_hw_error(struct acpi_hest_generic_data *gdata,
> - int sev, bool sync)
> + int sev, bool sync, enum context context)
> {
> struct cper_sec_proc_arm *err = acpi_hest_get_payload(gdata);
> - int flags = sync ? MF_ACTION_REQUIRED : 0;
> + int flags = sync && context == IN_USER ? MF_ACTION_REQUIRED : 0;
> int length = gdata->error_data_length;
> char error_type[120];
> bool queued = false;
> @@ -910,7 +910,8 @@ static void ghes_log_hwerr(int sev, guid_t *sec_type)
> }
>
> static void ghes_do_proc(struct ghes *ghes,
> - const struct acpi_hest_generic_status *estatus)
> + const struct acpi_hest_generic_status *estatus,
> + enum context context)
> {
> int sev, sec_sev;
> struct acpi_hest_generic_data *gdata;
> @@ -937,11 +938,11 @@ static void ghes_do_proc(struct ghes *ghes,
> atomic_notifier_call_chain(&ghes_report_chain, sev, mem_err);
>
> arch_apei_report_mem_error(sev, mem_err);
> - queued = ghes_handle_memory_failure(gdata, sev, sync);
> + queued = ghes_handle_memory_failure(gdata, sev, sync, context);
> } else if (guid_equal(sec_type, &CPER_SEC_PCIE)) {
> ghes_handle_aer(gdata);
> } else if (guid_equal(sec_type, &CPER_SEC_PROC_ARM)) {
> - queued = ghes_handle_arm_hw_error(gdata, sev, sync);
> + queued = ghes_handle_arm_hw_error(gdata, sev, sync, context);
> } else if (guid_equal(sec_type, &CPER_SEC_CXL_PROT_ERR)) {
> struct cxl_cper_sec_prot_err *prot_err = acpi_hest_get_payload(gdata);
>
> @@ -1190,7 +1191,7 @@ static int ghes_proc(struct ghes *ghes)
> if (ghes_print_estatus(NULL, ghes->generic, estatus))
> ghes_estatus_cache_add(ghes->generic, estatus);
> }
> - ghes_do_proc(ghes, estatus);
> + ghes_do_proc(ghes, estatus, NO_USE);
>
> out:
> ghes_clear_estatus(ghes, estatus, buf_paddr, FIX_APEI_GHES_IRQ);
> @@ -1297,7 +1298,7 @@ static void ghes_proc_in_irq(struct irq_work *irq_work)
> len = cper_estatus_len(estatus);
> node_len = GHES_ESTATUS_NODE_LEN(len);
>
> - ghes_do_proc(estatus_node->ghes, estatus);
> + ghes_do_proc(estatus_node->ghes, estatus, estatus_node->context);
>
> if (!ghes_estatus_cached(estatus)) {
> generic = estatus_node->generic;
> @@ -1335,7 +1336,8 @@ static void ghes_print_queued_estatus(void)
> }
>
> static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
> - enum fixed_addresses fixmap_idx)
> + enum fixed_addresses fixmap_idx,
> + enum context context)
> {
> struct acpi_hest_generic_status *estatus, tmp_header;
> struct ghes_estatus_node *estatus_node;
> @@ -1364,6 +1366,7 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
> if (!estatus_node)
> return -ENOMEM;
>
> + estatus_node->context = context;
> estatus_node->ghes = ghes;
> estatus_node->generic = ghes->generic;
> estatus = GHES_ESTATUS_FROM_NODE(estatus_node);
> @@ -1398,14 +1401,15 @@ static int ghes_in_nmi_queue_one_entry(struct ghes *ghes,
> }
>
> static int ghes_in_nmi_spool_from_list(struct list_head *rcu_list,
> - enum fixed_addresses fixmap_idx)
> + enum fixed_addresses fixmap_idx,
> + enum context context)
> {
> int ret = -ENOENT;
> struct ghes *ghes;
>
> rcu_read_lock();
> list_for_each_entry_rcu(ghes, rcu_list, list) {
> - if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx))
> + if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, context))
> ret = 0;
> }
> rcu_read_unlock();
> @@ -1488,7 +1492,7 @@ static LIST_HEAD(ghes_sea);
> * Return 0 only if one of the SEA error sources successfully reported an error
> * record sent from the firmware.
> */
> -int ghes_notify_sea(void)
> +int ghes_notify_sea(enum context context)
> {
> static DEFINE_RAW_SPINLOCK(ghes_notify_lock_sea);
> int rv;
> @@ -1497,7 +1501,7 @@ int ghes_notify_sea(void)
> return -ENOENT;
>
> raw_spin_lock(&ghes_notify_lock_sea);
> - rv = ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA);
> + rv = ghes_in_nmi_spool_from_list(&ghes_sea, FIX_APEI_GHES_SEA, context);
> raw_spin_unlock(&ghes_notify_lock_sea);
>
> return rv;
> @@ -1552,7 +1556,7 @@ static int ghes_notify_nmi(unsigned int cmd, struct pt_regs *regs)
> return ret;
>
> raw_spin_lock(&ghes_notify_lock_nmi);
> - if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI))
> + if (!ghes_in_nmi_spool_from_list(&ghes_nmi, FIX_APEI_GHES_NMI, NO_USE))
> ret = NMI_HANDLED;
> raw_spin_unlock(&ghes_notify_lock_nmi);
>
> @@ -1606,7 +1610,7 @@ static void ghes_nmi_init_cxt(void)
> static int __ghes_sdei_callback(struct ghes *ghes,
> enum fixed_addresses fixmap_idx)
> {
> - if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx)) {
> + if (!ghes_in_nmi_queue_one_entry(ghes, fixmap_idx, NO_USE)) {
> irq_work_queue(&ghes_proc_irq_work);
>
> return 0;
> diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h
> index 8d7e5caef3f1..646cd5c3c0ca 100644
> --- a/include/acpi/ghes.h
> +++ b/include/acpi/ghes.h
> @@ -33,10 +33,12 @@ struct ghes {
> void __iomem *error_status_vaddr;
> };
>
> +enum context {NO_USE = -1, IN_KERNEL = 0, IN_USER = 1};
This goes into include/acpi/ghes.h, which is included widely. The
type name and all three enumerators are far too generic; "IN_USER"
and "IN_KERNEL" in particular are likely to collide with driver-local
enums in the future, and once this is merged the names are hard to
change.
Please prefix:
enum ghes_exec_ctx {
GHES_CTX_NA = -1,
GHES_CTX_KERNEL = 0,
GHES_CTX_USER = 1,
};
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC
2026-05-18 8:49 ` [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC Ruidong Tian
@ 2026-05-27 11:35 ` Shuai Xue
2026-06-04 8:10 ` Ruidong Tian
0 siblings, 1 reply; 20+ messages in thread
From: Shuai Xue @ 2026-05-27 11:35 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> From: Tong Tiangen <tongtiangen@huawei•com>
>
> For the arm64 kernel, when it processes hardware memory errors for
> synchronize notifications(do_sea()), if the errors is consumed within the
> kernel, the current processing is panic. However, it is not optimal.
>
> Take copy_from/to_user for example, If ld* triggers a memory error, even in
> kernel mode, only the associated process is affected. Killing the user
> process and isolating the corrupt page is a better choice.
>
> Add new fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR to identify insn
> that can recover from memory errors triggered by access to kernel memory,
> and this fixup type is used in __arch_copy_to_user(), This make the regular
> copy_to_user() will handle kernel memory errors.
>
> [Ruidong: handle EX_TYPE_UACCESS_CPY in fixup_exception_me()]
>
> Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
> ---
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/asm-extable.h | 22 +++++++++++++++++++-
> arch/arm64/include/asm/asm-uaccess.h | 4 ++++
> arch/arm64/include/asm/extable.h | 1 +
> arch/arm64/lib/copy_to_user.S | 10 +++++-----
> arch/arm64/mm/extable.c | 21 +++++++++++++++++++
> arch/arm64/mm/fault.c | 30 ++++++++++++++++++++--------
> 7 files changed, 75 insertions(+), 14 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fe60738e5943..831b20d45893 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -21,6 +21,7 @@ config ARM64
> select ARCH_HAS_CACHE_LINE_SIZE
> select ARCH_HAS_CC_PLATFORM
> select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
> + select ARCH_HAS_COPY_MC if ACPI_APEI_GHES
At this commit:
- arch/arm64/lib/memcpy_mc.S does not exist (patch 7)
- arch/arm64/lib/copy_mc_page.S does not exist (patch 5)
- arm64 has no copy_mc_to_kernel() override
- __HAVE_ARCH_COPY_MC_USER_HIGHPAGE is not defined
Build does not break because the generic fallback in
include/linux/uaccess.h and include/linux/highmem.h covers it, but
ARCH_HAS_COPY_MC=y silently means "plain memcpy() with no MC
handling at all" between this commit and patch 7. Anyone bisecting
an MC regression in this window will be very confused.
Please move this select to the last arm64 implementation patch in
the series.
> select ARCH_HAS_CURRENT_STACK_POINTER
> select ARCH_HAS_DEBUG_VIRTUAL
> select ARCH_HAS_DEBUG_VM_PGTABLE
> diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/include/asm/asm-extable.h
> index d67e2fdd1aee..4980023f2fbd 100644
> --- a/arch/arm64/include/asm/asm-extable.h
> +++ b/arch/arm64/include/asm/asm-extable.h
> @@ -11,6 +11,8 @@
> #define EX_TYPE_KACCESS_ERR_ZERO 3
> #define EX_TYPE_UACCESS_CPY 4
> #define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 5
> +/* kernel access memory error safe */
> +#define EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR 6
KACCESS_ERR_ZERO is already "encode err reg + zero reg". Tacking
_MEM_ERR on the end reads like "+ another mem-err reg". Please
rename to e.g. EX_TYPE_KACCESS_ERR_ZERO_MC, which directly tells
the reader "MC-safe variant of KACCESS_ERR_ZERO".
>
> /* Data fields for EX_TYPE_UACCESS_ERR_ZERO */
> #define EX_DATA_REG_ERR_SHIFT 0
> @@ -42,7 +44,7 @@
> (.L__gpr_num_##gpr << EX_DATA_REG_##reg##_SHIFT)
>
> #define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \
> - __ASM_EXTABLE_RAW(insn, fixup, \
> + __ASM_EXTABLE_RAW(insn, fixup, \
> EX_TYPE_UACCESS_ERR_ZERO, \
> ( \
> EX_DATA_REG(ERR, err) | \
> @@ -55,6 +57,17 @@
> #define _ASM_EXTABLE_UACCESS(insn, fixup) \
> _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, wzr, wzr)
>
> +#define _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, err, zero) \
> + __ASM_EXTABLE_RAW(insn, fixup, \
> + EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, \
> + ( \
> + EX_DATA_REG(ERR, err) | \
> + EX_DATA_REG(ZERO, zero) \
> + ))
> +
> +#define _ASM_EXTABLE_KACCESS_MEM_ERR(insn, fixup) \
> + _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, wzr, wzr)
> +
> /*
> * Create an exception table entry for uaccess `insn`, which will branch to `fixup`
> * when an unhandled fault is taken.
> @@ -76,6 +89,13 @@
> .macro _asm_extable_uaccess_cpy, insn, fixup, uaccess_is_write
> __ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_UACCESS_CPY, \uaccess_is_write)
> .endm
> +/*
> + * Create an exception table entry for kaccess `insn`, which will branch to
> + * `fixup` when an unhandled fault is taken.
> + */
> + .macro _asm_extable_kaccess_mem_err, insn, fixup
> + _ASM_EXTABLE_KACCESS_MEM_ERR(\insn, \fixup)
> + .endm
>
> #else /* __ASSEMBLER__ */
>
> diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h
> index 12aa6a283249..c8f0af5fde63 100644
> --- a/arch/arm64/include/asm/asm-uaccess.h
> +++ b/arch/arm64/include/asm/asm-uaccess.h
> @@ -57,6 +57,10 @@ alternative_else_nop_endif
> .endm
> #endif
>
> +#define KERNEL_MEM_ERR(l, x...) \
> +9999: x; \
> + _asm_extable_kaccess_mem_err 9999b, l
> +
> #define USER(l, x...) \
> 9999: x; \
> _asm_extable_uaccess 9999b, l
> diff --git a/arch/arm64/include/asm/extable.h b/arch/arm64/include/asm/extable.h
> index 9dc39612bdf5..47c851d7df4f 100644
> --- a/arch/arm64/include/asm/extable.h
> +++ b/arch/arm64/include/asm/extable.h
> @@ -48,4 +48,5 @@ bool ex_handler_bpf(const struct exception_table_entry *ex,
> #endif /* !CONFIG_BPF_JIT */
>
> bool fixup_exception(struct pt_regs *regs, unsigned long esr);
> +bool fixup_exception_me(struct pt_regs *regs);
> #endif
> diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S
> index 819f2e3fc7a9..991d94ecc1a8 100644
> --- a/arch/arm64/lib/copy_to_user.S
> +++ b/arch/arm64/lib/copy_to_user.S
> @@ -20,7 +20,7 @@
> * x0 - bytes not copied
> */
> .macro ldrb1 reg, ptr, val
> - ldrb \reg, [\ptr], \val
> + KERNEL_MEM_ERR(9998f, ldrb \reg, [\ptr], \val)
> .endm
>
> .macro strb1 reg, ptr, val
> @@ -28,7 +28,7 @@
> .endm
>
> .macro ldrh1 reg, ptr, val
> - ldrh \reg, [\ptr], \val
> + KERNEL_MEM_ERR(9998f, ldrh \reg, [\ptr], \val)
> .endm
>
> .macro strh1 reg, ptr, val
> @@ -36,7 +36,7 @@
> .endm
>
> .macro ldr1 reg, ptr, val
> - ldr \reg, [\ptr], \val
> + KERNEL_MEM_ERR(9998f, ldr \reg, [\ptr], \val)
> .endm
>
> .macro str1 reg, ptr, val
> @@ -44,7 +44,7 @@
> .endm
>
> .macro ldp1 reg1, reg2, ptr, val
> - ldp \reg1, \reg2, [\ptr], \val
> + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr], \val)
> .endm
>
> .macro stp1 reg1, reg2, ptr, val
> @@ -74,7 +74,7 @@ SYM_FUNC_START(__arch_copy_to_user)
> 9997: cmp dst, dstin
> b.ne 9998f
> // Before being absolutely sure we couldn't copy anything, try harder
> - ldrb tmp1w, [srcin]
> +KERNEL_MEM_ERR(9998f, ldrb tmp1w, [srcin])
> USER(9998f, sttrb tmp1w, [dst])
> add dst, dst, #1
> 9998: sub x0, end, dst // bytes not copied
> diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c
> index 6e0528831cd3..f78ac7e92845 100644
> --- a/arch/arm64/mm/extable.c
> +++ b/arch/arm64/mm/extable.c
> @@ -110,7 +110,28 @@ bool fixup_exception(struct pt_regs *regs, unsigned long esr)
> return ex_handler_uaccess_cpy(ex, regs, esr);
> case EX_TYPE_LOAD_UNALIGNED_ZEROPAD:
> return ex_handler_load_unaligned_zeropad(ex, regs);
> + case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR:
> + return false;
> }
>
> BUG();
> }
> +
> +bool fixup_exception_me(struct pt_regs *regs)
> +{
> + const struct exception_table_entry *ex;
> +
> + ex = search_exception_tables(instruction_pointer(regs));
> + if (!ex)
> + return false;
> +
> + switch (ex->type) {
> + case EX_TYPE_UACCESS_CPY:
> + return ex_handler_uaccess_cpy(ex, regs, 0);
Pointed by sashiko:
copy_to_user.S annotates its MOPS prologue/main/epilogue with
USER_CPY(..., 1, cpyf{p,m,e}wt ...), so uaccess_is_write=1 for the
whole MOPS sequence. With esr=0 hard-coded here:
cpy_faulted_on_uaccess(): uaccess_is_write=1, fault_on_write=0
-> returns false
ex_handler_uaccess_cpy() -> returns false
fixup_exception_me() -> returns false
do_apei_claim_sea() -> returns -ENOENT
do_sea() -> panic
So any MC SEA taken inside a MOPS copy_to_user() panics the kernel,
exactly defeating the recovery this patch claims to provide.
> + case EX_TYPE_UACCESS_ERR_ZERO:
> + case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR:
> + return ex_handler_uaccess_err_zero(ex, regs);
> + }
> +
> + return false;
> +}
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 0f3c5c7ca054..efbda54770be 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -858,21 +858,35 @@ static int do_bad(unsigned long far, unsigned long esr, struct pt_regs *regs)
> return 1; /* "fault" */
> }
>
> +/*
> + * APEI claimed this as a firmware-first notification.
> + * Some processing deferred to task_work before ret_to_user().
> + */
> +static int do_apei_claim_sea(struct pt_regs *regs)
> +{
> + int ret;
> +
> + ret = apei_claim_sea(regs);
> + if (ret)
> + return ret;
> +
> + if (!user_mode(regs) && IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC)) {
The IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) test is also dead on arm64:
ARCH_HAS_COPY_MC is selected iff ACPI_APEI_GHES, and without
ACPI_APEI_GHES apei_claim_sea() returns -ENOENT and we never reach
this branch. Please drop it.
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage()
2026-05-18 8:49 ` [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage() Ruidong Tian
@ 2026-05-27 11:44 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-27 11:44 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev,
Jonathan Cameron, Mauro Carvalho Chehab
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> From: Tong Tiangen <tongtiangen@huawei•com>
>
> Currently, copy_mc_[user]_highpage() returns zero on success, or in case
> of failures, the number of bytes that weren't copied.
>
> While tracking the number of not copied works fine for x86 and PPC, There
> are some difficulties in doing the same thing on ARM64 because there is no
> available caller-saved register in copy_page()(lib/copy_page.S) to save
> "bytes not copied", and the following copy_mc_page() will also encounter
> the same problem.
>
> Consider the caller of copy_mc_[user]_highpage() cannot do any processing
> on the remaining data(The page has hardware errors), they only check if
> copy was succeeded or not, make the interface more generic by using an
> error code when copy fails (-EFAULT) or return zero on success.
>
> Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei•com>
> Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel•org>
Reviewed-by: Shuai Xue <xueshuai@linux•alibaba.com>
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage()
2026-05-18 8:49 ` [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage() Ruidong Tian
@ 2026-05-27 12:11 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-27 12:11 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> From: Tong Tiangen <tongtiangen@huawei•com>
>
> Currently, many scenarios that can tolerate memory errors when copying page
> have been supported in the kernel[1~5], all of which are implemented by
> copy_mc_[user]_highpage(). arm64 should also support this mechanism.
>
> Due to mte, arm64 needs to have its own copy_mc_[user]_highpage()
> architecture implementation, macros __HAVE_ARCH_COPY_MC_HIGHPAGE and
> __HAVE_ARCH_COPY_MC_USER_HIGHPAGE have been added to control it.
>
> Add new helper copy_mc_page() which provide a page copy implementation with
> hardware memory error safe. The code logic of copy_mc_page() is the same as
> copy_page(), the main difference is that the ldp insn of copy_mc_page()
> contains the fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, therefore, the
> main logic is extracted to copy_page_template.S. In addition, the fixup of
> MOPS insn is not considered at present.
>
> [Ruidong: add FEAT_MOPS support]
copy_page_template.S now references the cpy1 macro inside the MOPS
alternative:
#ifdef CONFIG_AS_HAS_MOPS
alternative_if_not ARM64_HAS_MOPS
b .Lno_mops
alternative_else_nop_endif
mov x2, #PAGE_SIZE
cpy1 dst, src, x2
b .Lexitfunc
.Lno_mops:
#endif
copy_mc_page.S provides cpy1 (using cpyfp/m/ert + USER_CPY).
copy_page.S, however, only provides ldp1 -- no cpy1 -- so any
toolchain with FEAT_MOPS support fails to assemble copy_page.S
with "unknown mnemonic 'cpy1'".
>
> [1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits poison, take page offline")
> [2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults")
> [3] commit 6b970599e807 ("mm: hwpoison: support recovery from ksm_might_need_to_copy()")
> [4] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned anonymous memory")
> [5] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-backed memory")
>
> Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
> ---
> arch/arm64/include/asm/mte.h | 9 ++++
> arch/arm64/include/asm/page.h | 10 ++++
> arch/arm64/lib/Makefile | 2 +
> arch/arm64/lib/copy_mc_page.S | 44 +++++++++++++++++
> arch/arm64/lib/copy_page.S | 62 ++----------------------
> arch/arm64/lib/copy_page_template.S | 71 +++++++++++++++++++++++++++
> arch/arm64/lib/mte.S | 29 +++++++++++
> arch/arm64/mm/copypage.c | 75 +++++++++++++++++++++++++++++
> include/linux/highmem.h | 8 +++
> 9 files changed, 253 insertions(+), 57 deletions(-)
> create mode 100644 arch/arm64/lib/copy_mc_page.S
> create mode 100644 arch/arm64/lib/copy_page_template.S
>
> diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h
> index 7f7b97e09996..a0b1757f4847 100644
> --- a/arch/arm64/include/asm/mte.h
> +++ b/arch/arm64/include/asm/mte.h
> @@ -98,6 +98,11 @@ static inline bool try_page_mte_tagging(struct page *page)
> void mte_zero_clear_page_tags(void *addr);
> void mte_sync_tags(pte_t pte, unsigned int nr_pages);
> void mte_copy_page_tags(void *kto, const void *kfrom);
> +
> +#ifdef CONFIG_ARCH_HAS_COPY_MC
> +int mte_copy_mc_page_tags(void *kto, const void *kfrom);
> +#endif
> +
> void mte_thread_init_user(void);
> void mte_thread_switch(struct task_struct *next);
> void mte_cpu_setup(void);
> @@ -134,6 +139,10 @@ static inline void mte_sync_tags(pte_t pte, unsigned int nr_pages)
> static inline void mte_copy_page_tags(void *kto, const void *kfrom)
> {
> }
> +static inline int mte_copy_mc_page_tags(void *kto, const void *kfrom)
> +{
> + return 0;
> +}
> static inline void mte_thread_init_user(void)
> {
> }
> diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
> index e25d0d18f6d7..f65818ee614a 100644
> --- a/arch/arm64/include/asm/page.h
> +++ b/arch/arm64/include/asm/page.h
> @@ -29,6 +29,16 @@ void copy_user_highpage(struct page *to, struct page *from,
> void copy_highpage(struct page *to, struct page *from);
> #define __HAVE_ARCH_COPY_HIGHPAGE
>
> +#ifdef CONFIG_ARCH_HAS_COPY_MC
> +int copy_mc_page(void *to, const void *from);
> +int copy_mc_highpage(struct page *to, struct page *from);
> +#define __HAVE_ARCH_COPY_MC_HIGHPAGE
> +
> +int copy_mc_user_highpage(struct page *to, struct page *from,
> + unsigned long vaddr, struct vm_area_struct *vma);
> +#define __HAVE_ARCH_COPY_MC_USER_HIGHPAGE
> +#endif
> +
> struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma,
> unsigned long vaddr);
> #define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio
> diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
> index 448c917494f3..1f4c3f743a20 100644
> --- a/arch/arm64/lib/Makefile
> +++ b/arch/arm64/lib/Makefile
> @@ -7,6 +7,8 @@ lib-y := clear_user.o delay.o copy_from_user.o \
>
> lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
>
> +lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
> +
> obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
>
> obj-$(CONFIG_ARM64_MTE) += mte.o
> diff --git a/arch/arm64/lib/copy_mc_page.S b/arch/arm64/lib/copy_mc_page.S
> new file mode 100644
> index 000000000000..ad1371e9e687
> --- /dev/null
> +++ b/arch/arm64/lib/copy_mc_page.S
> @@ -0,0 +1,44 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +
> +#include <linux/linkage.h>
> +#include <linux/const.h>
> +#include <asm/assembler.h>
> +#include <asm/page.h>
> +#include <asm/cpufeature.h>
> +#include <asm/alternative.h>
> +#include <asm/asm-extable.h>
> +#include <asm/asm-uaccess.h>
> +
> +/*
> + * Copy a page from src to dest (both are page aligned) with memory error safe
> + *
> + * Parameters:
> + * x0 - dest
> + * x1 - src
> + * Returns:
> + * x0 - Return 0 if copy success, or -EFAULT if anything goes wrong
> + * while copying.
> + */
> + .macro ldp1 reg1, reg2, ptr, val
> + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr, \val])
> + .endm
> +
> + .macro cpy1 dst, src, count
> + .arch_extension mops
> + USER_CPY(9998f, 0, cpyfprt [\dst]!, [\src]!, \count!)
> + USER_CPY(9998f, 0, cpyfmrt [\dst]!, [\src]!, \count!)
> + USER_CPY(9998f, 0, cpyfert [\dst]!, [\src]!, \count!)
> + .endm
> +
> +SYM_FUNC_START(__pi_copy_mc_page)
> +#include "copy_page_template.S"
> +
> + mov x0, #0
> + ret
> +
> +9998: mov x0, #-EFAULT
> + ret
> +
> +SYM_FUNC_END(__pi_copy_mc_page)
> +SYM_FUNC_ALIAS(copy_mc_page, __pi_copy_mc_page)
> +EXPORT_SYMBOL(copy_mc_page)
> diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
> index e6374e7e5511..d0186bbf99f1 100644
> --- a/arch/arm64/lib/copy_page.S
> +++ b/arch/arm64/lib/copy_page.S
> @@ -17,65 +17,13 @@
> * x0 - dest
> * x1 - src
> */
> -SYM_FUNC_START(__pi_copy_page)
> -#ifdef CONFIG_AS_HAS_MOPS
> - .arch_extension mops
> -alternative_if_not ARM64_HAS_MOPS
> - b .Lno_mops
> -alternative_else_nop_endif
> -
> - mov x2, #PAGE_SIZE
> - cpypwn [x0]!, [x1]!, x2!
> - cpymwn [x0]!, [x1]!, x2!
> - cpyewn [x0]!, [x1]!, x2!
> - ret
> -.Lno_mops:
> -#endif
> - ldp x2, x3, [x1]
> - ldp x4, x5, [x1, #16]
> - ldp x6, x7, [x1, #32]
> - ldp x8, x9, [x1, #48]
> - ldp x10, x11, [x1, #64]
> - ldp x12, x13, [x1, #80]
> - ldp x14, x15, [x1, #96]
> - ldp x16, x17, [x1, #112]
> -
> - add x0, x0, #256
> - add x1, x1, #128
> -1:
> - tst x0, #(PAGE_SIZE - 1)
>
> - stnp x2, x3, [x0, #-256]
> - ldp x2, x3, [x1]
> - stnp x4, x5, [x0, #16 - 256]
> - ldp x4, x5, [x1, #16]
> - stnp x6, x7, [x0, #32 - 256]
> - ldp x6, x7, [x1, #32]
> - stnp x8, x9, [x0, #48 - 256]
> - ldp x8, x9, [x1, #48]
> - stnp x10, x11, [x0, #64 - 256]
> - ldp x10, x11, [x1, #64]
> - stnp x12, x13, [x0, #80 - 256]
> - ldp x12, x13, [x1, #80]
> - stnp x14, x15, [x0, #96 - 256]
> - ldp x14, x15, [x1, #96]
> - stnp x16, x17, [x0, #112 - 256]
> - ldp x16, x17, [x1, #112]
> -
> - add x0, x0, #128
> - add x1, x1, #128
> -
> - b.ne 1b
> -
> - stnp x2, x3, [x0, #-256]
> - stnp x4, x5, [x0, #16 - 256]
> - stnp x6, x7, [x0, #32 - 256]
> - stnp x8, x9, [x0, #48 - 256]
> - stnp x10, x11, [x0, #64 - 256]
> - stnp x12, x13, [x0, #80 - 256]
> - stnp x14, x15, [x0, #96 - 256]
> - stnp x16, x17, [x0, #112 - 256]
> + .macro ldp1 reg1, reg2, ptr, val
> + ldp \reg1, \reg2, [\ptr, \val]
> + .endm
>
> +SYM_FUNC_START(__pi_copy_page)
> +#include "copy_page_template.S"
> ret
> SYM_FUNC_END(__pi_copy_page)
> SYM_FUNC_ALIAS(copy_page, __pi_copy_page)
> diff --git a/arch/arm64/lib/copy_page_template.S b/arch/arm64/lib/copy_page_template.S
> new file mode 100644
> index 000000000000..d466b51c8ed9
> --- /dev/null
> +++ b/arch/arm64/lib/copy_page_template.S
> @@ -0,0 +1,71 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2012 ARM Ltd.
> + */
> +
> +/*
> + * Copy a page from src to dest (both are page aligned)
> + *
> + * Parameters:
> + * x0 - dest
> + * x1 - src
> + */
> +dstin .req x0
> +src .req x1
> +
> +#ifdef CONFIG_AS_HAS_MOPS
> + .arch_extension mops
> +alternative_if_not ARM64_HAS_MOPS
> + b .Lno_mops
> +alternative_else_nop_endif
> + mov x2, #PAGE_SIZE
> + cpy1 dst, src, x2
copy_page_template.S now references the cpy1 macro inside the MOPS
alternative:
copy_page.S, however, only provides ldp1 -- no cpy1 -- so any
toolchain with FEAT_MOPS support fails to assemble copy_page.S
with "unknown mnemonic 'cpy1'".
Trivial fix:
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
@@
.macro ldp1 reg1, reg2, ptr, val
ldp \reg1, \reg2, [\ptr, \val]
.endm
+ .macro cpy1 dst, src, count
+ .arch_extension mops
+ cpyp [\dst]!, [\src]!, \count!
+ cpym [\dst]!, [\src]!, \count!
+ cpye [\dst]!, [\src]!, \count!
+ .endm
+
SYM_FUNC_START(__pi_copy_page)
#include "copy_page_template.S"
> + b .Lexitfunc
> +.Lno_mops:
> +#endif
> +
> + ldp1 x2, x3, x1, #0
> + ldp1 x4, x5, x1, #16
> + ldp1 x6, x7, x1, #32
> + ldp1 x8, x9, x1, #48
> + ldp1 x10, x11, x1, #64
> + ldp1 x12, x13, x1, #80
> + ldp1 x14, x15, x1, #96
> + ldp1 x16, x17, x1, #112
> +
> + add x0, x0, #256
> + add x1, x1, #128
> +1:
> + tst x0, #(PAGE_SIZE - 1)
> +
> + stnp x2, x3, [x0, #-256]
> + ldp1 x2, x3, x1, #0
> + stnp x4, x5, [x0, #16 - 256]
> + ldp1 x4, x5, x1, #16
> + stnp x6, x7, [x0, #32 - 256]
> + ldp1 x6, x7, x1, #32
> + stnp x8, x9, [x0, #48 - 256]
> + ldp1 x8, x9, x1, #48
> + stnp x10, x11, [x0, #64 - 256]
> + ldp1 x10, x11, x1, #64
> + stnp x12, x13, [x0, #80 - 256]
> + ldp1 x12, x13, x1, #80
> + stnp x14, x15, [x0, #96 - 256]
> + ldp1 x14, x15, x1, #96
> + stnp x16, x17, [x0, #112 - 256]
> + ldp1 x16, x17, x1, #112
> +
> + add x0, x0, #128
> + add x1, x1, #128
> +
> + b.ne 1b
> +
> + stnp x2, x3, [x0, #-256]
> + stnp x4, x5, [x0, #16 - 256]
> + stnp x6, x7, [x0, #32 - 256]
> + stnp x8, x9, [x0, #48 - 256]
> + stnp x10, x11, [x0, #64 - 256]
> + stnp x12, x13, [x0, #80 - 256]
> + stnp x14, x15, [x0, #96 - 256]
> + stnp x16, x17, [x0, #112 - 256]
> +.Lexitfunc:
> diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
> index 5018ac03b6bf..9d4eeb76a838 100644
> --- a/arch/arm64/lib/mte.S
> +++ b/arch/arm64/lib/mte.S
> @@ -80,6 +80,35 @@ SYM_FUNC_START(mte_copy_page_tags)
> ret
> SYM_FUNC_END(mte_copy_page_tags)
>
> +#ifdef CONFIG_ARCH_HAS_COPY_MC
> +/*
> + * Copy the tags from the source page to the destination one with memory error safe
> + * x0 - address of the destination page
> + * x1 - address of the source page
> + * Returns:
> + * x0 - Return 0 if copy success, or
> + * -EFAULT if anything goes wrong while copying.
> + */
> +SYM_FUNC_START(mte_copy_mc_page_tags)
> + mov x2, x0
> + mov x3, x1
> + multitag_transfer_size x5, x6
> +1:
> +KERNEL_MEM_ERR(2f, ldgm x4, [x3])
> + stgm x4, [x2]
> + add x2, x2, x5
> + add x3, x3, x5
> + tst x2, #(PAGE_SIZE - 1)
> + b.ne 1b
> +
> + mov x0, #0
> + ret
> +
> +2: mov x0, #-EFAULT
> + ret
> +SYM_FUNC_END(mte_copy_mc_page_tags)
> +#endif
> +
> /*
> * Read tags from a user buffer (one tag per byte) and set the corresponding
> * tags at the given kernel address. Used by PTRACE_POKEMTETAGS.
> diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c
> index cd5912ba617b..9fd773baf17b 100644
> --- a/arch/arm64/mm/copypage.c
> +++ b/arch/arm64/mm/copypage.c
> @@ -72,3 +72,78 @@ void copy_user_highpage(struct page *to, struct page *from,
> flush_dcache_page(to);
> }
> EXPORT_SYMBOL_GPL(copy_user_highpage);
> +
> +#ifdef CONFIG_ARCH_HAS_COPY_MC
> +/*
> + * Return -EFAULT if anything goes wrong while copying page or mte.
> + */
> +int copy_mc_highpage(struct page *to, struct page *from)
> +{
> + void *kto = page_address(to);
> + void *kfrom = page_address(from);
> + struct folio *src = page_folio(from);
> + struct folio *dst = page_folio(to);
> + unsigned int i, nr_pages;
> + int ret;
> +
> + ret = copy_mc_page(kto, kfrom);
> + if (ret)
> + return -EFAULT;
The generic fallback in include/linux/highmem.h does:
ret = copy_mc_to_kernel(vto, vfrom, PAGE_SIZE);
...
if (ret) memory_failure_queue(page_to_pfn(from), 0);
The arm64 implementation in arch/arm64/mm/copypage.c does not call
emory_failure_queue() on failure. Please document it explicitly.
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
2026-05-18 8:49 ` [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests Ruidong Tian
@ 2026-05-27 13:43 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-27 13:43 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> Add KUnit tests for copy_page() and copy_mc_page(), modeled after
> the existing memcpy_test() style: a static page-aligned src and a
> two-page dst, filled with random bytes plus non-zero edges, then
> verify byte-for-byte equality and that the adjacent page is
> untouched. The copy_mc_page() case additionally checks the return
> value is 0 on clean memory and is gated on CONFIG_ARCH_HAS_COPY_MC.
>
> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
> ---
> lib/tests/memcpy_kunit.c | 67 +++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 66 insertions(+), 1 deletion(-)
>
> diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
> index d36933554e46..85df53ccfb0c 100644
> --- a/lib/tests/memcpy_kunit.c
> +++ b/lib/tests/memcpy_kunit.c
> @@ -493,6 +493,67 @@ static void memmove_overlap_test(struct kunit *test)
> }
> }
>
> +/* --- Page-sized copy tests --- */
> +
> +static u8 page_src[PAGE_SIZE] __aligned(PAGE_SIZE);
> +static u8 page_dst[PAGE_SIZE * 2] __aligned(PAGE_SIZE);
> +static const u8 page_zero[PAGE_SIZE] __aligned(PAGE_SIZE);
> +
> +static void init_page(struct kunit *test)
> +{
> + /* Get many bit patterns. */
> + get_random_bytes(page_src, PAGE_SIZE);
> +
> + /* Make sure we have non-zero edges. */
> + set_random_nonzero(test, &page_src[0]);
> + set_random_nonzero(test, &page_src[PAGE_SIZE - 1]);
> +
> + /* Explicitly zero the entire destination. */
> + memset(page_dst, 0, ARRAY_SIZE(page_dst));
init_page() memsets the entire two-page page_dst to zero, and the
test then asserts the second page is still zero. A buggy copy_page()
that overwrites the second page with bytes that happen to include a
zero byte would still pass.
Please fill page_dst with a non-zero sentinel (e.g. 0xA5) instead
of zero, and check the second half against that sentinel using
memchr_inv():
memset(page_dst, 0xA5, ARRAY_SIZE(page_dst));
...
KUNIT_ASSERT_TRUE_MSG(test,
!memchr_inv(page_dst + PAGE_SIZE, 0xA5, PAGE_SIZE),
"copy_page overflow into adjacent page");
This also lets you delete page_zero[] entirely, which on
PAGE_SIZE=64K configs (arm64) saves a non-trivial amount of BSS.
> +}
> +
> +static void copy_page_test(struct kunit *test)
> +{
> + init_page(test);
> +
> + /* Copy. */
> + copy_page(page_dst, page_src);
> +
> + /* Verify byte-for-byte exact. */
> + KUNIT_ASSERT_EQ_MSG(test,
> + memcmp(page_dst, page_src, PAGE_SIZE), 0,
> + "copy_page content mismatch with random data");
> +
> + /* Verify no overflow into second page. */
> + KUNIT_ASSERT_EQ_MSG(test,
> + memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0,
> + "copy_page overflow into adjacent page");
> +}
> +
> +#ifdef CONFIG_ARCH_HAS_COPY_MC
> +static void copy_mc_page_test(struct kunit *test)
> +{
> + int ret;
> +
> + init_page(test);
> +
> + /* Copy and check return value. */
> + ret = copy_mc_page(page_dst, page_src);
From sashiko:
Does this introduce a build regression on architectures like x86_64 or
powerpc?
The CONFIG_ARCH_HAS_COPY_MC flag is selected by multiple architectures,
but copy_mc_page() is an arm64-specific API. Other architectures use the
generic copy_mc_to_kernel() API instead.
If the kernel is built with KUnit enabled on non-arm64 architectures, it
appears copy_mc_page() would be an undeclared identifier.
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation
2026-05-18 8:49 ` [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation Ruidong Tian
@ 2026-05-28 3:10 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-28 3:10 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> From: Tong Tiangen <tongtiangen@huawei•com>
>
> The copy_mc_to_kernel() helper is memory copy implementation that handles
> source exceptions. It can be used in memory copy scenarios that tolerate
> hardware memory errors(e.g: pmem_read/dax_copy_to_iter).
>
> Currently, only x86 and ppc support this helper, Add this for ARM64 as
> well, if ARCH_HAS_COPY_MC is defined, by implementing copy_mc_to_kernel()
> and memcpy_mc() functions.
>
> Because there is no caller-saved GPR is available for saving "bytes not
> copied" in memcpy(), the memcpy_mc() is referenced to the implementation
> of copy_from_user(). In addition, the fixup of MOPS insn is not considered
> at present.
>
> [Ruidong: refactor memcpy_mc on top of the new memcpy implementation.]
>
> Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
> ---
> arch/arm64/include/asm/string.h | 5 +
> arch/arm64/include/asm/uaccess.h | 17 +++
> arch/arm64/lib/Makefile | 2 +-
> arch/arm64/lib/memcpy.S | 253 +++----------------------------
> arch/arm64/lib/memcpy_mc.S | 56 +++++++
> arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++++++
> mm/kasan/shadow.c | 12 ++
> 7 files changed, 359 insertions(+), 235 deletions(-)
> create mode 100644 arch/arm64/lib/memcpy_mc.S
> create mode 100644 arch/arm64/lib/memcpy_template.S
>
> diff --git a/arch/arm64/include/asm/string.h b/arch/arm64/include/asm/string.h
> index 3a3264ff47b9..23eca4fb24fa 100644
> --- a/arch/arm64/include/asm/string.h
> +++ b/arch/arm64/include/asm/string.h
> @@ -35,6 +35,10 @@ extern void *memchr(const void *, int, __kernel_size_t);
> extern void *memcpy(void *, const void *, __kernel_size_t);
> extern void *__memcpy(void *, const void *, __kernel_size_t);
>
> +#define __HAVE_ARCH_MEMCPY_MC
> +extern int memcpy_mc(void *, const void *, __kernel_size_t);
> +extern int __memcpy_mc(void *, const void *, __kernel_size_t);
> +
> #define __HAVE_ARCH_MEMMOVE
> extern void *memmove(void *, const void *, __kernel_size_t);
> extern void *__memmove(void *, const void *, __kernel_size_t);
> @@ -57,6 +61,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t cnt);
> */
>
> #define memcpy(dst, src, len) __memcpy(dst, src, len)
> +#define memcpy_mc(dst, src, len) __memcpy_mc(dst, src, len)
> #define memmove(dst, src, len) __memmove(dst, src, len)
> #define memset(s, c, n) __memset(s, c, n)
>
> diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
> index b0c83a08dda9..93277eca2268 100644
> --- a/arch/arm64/include/asm/uaccess.h
> +++ b/arch/arm64/include/asm/uaccess.h
> @@ -499,5 +499,22 @@ static inline size_t probe_subpage_writeable(const char __user *uaddr,
> }
>
> #endif /* CONFIG_ARCH_HAS_SUBPAGE_FAULTS */
> +#ifdef CONFIG_ARCH_HAS_COPY_MC
> +/**
> + * copy_mc_to_kernel - memory copy that handles source exceptions
> + *
> + * @to: destination address
> + * @from: source address
> + * @size: number of bytes to copy
> + *
> + * Return 0 for success, or bytes not copied.
> + */
> +static inline unsigned long __must_check
> +copy_mc_to_kernel(void *to, const void *from, unsigned long size)
> +{
> + return memcpy_mc(to, from, size);
> +}
> +#define copy_mc_to_kernel copy_mc_to_kernel
> +#endif
>
> #endif /* __ASM_UACCESS_H */
> diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
> index 1f4c3f743a20..a5820e6c33d4 100644
> --- a/arch/arm64/lib/Makefile
> +++ b/arch/arm64/lib/Makefile
> @@ -7,7 +7,7 @@ lib-y := clear_user.o delay.o copy_from_user.o \
>
> lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
>
> -lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o
> +lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc_page.o memcpy_mc.o
>
> obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
>
> diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S
> index 9b99106fb95f..ef6aea2de9b4 100644
> --- a/arch/arm64/lib/memcpy.S
> +++ b/arch/arm64/lib/memcpy.S
> @@ -15,247 +15,32 @@
> *
> */
>
> -#define L(label) .L ## label
> + .macro ldrb1 reg, addr:vararg
> + ldrb \reg, \addr
> + .endm
>
> -#define dstin x0
> -#define src x1
> -#define count x2
> -#define dst x3
> -#define srcend x4
> -#define dstend x5
> -#define A_l x6
> -#define A_lw w6
> -#define A_h x7
> -#define B_l x8
> -#define B_lw w8
> -#define B_h x9
> -#define C_l x10
> -#define C_lw w10
> -#define C_h x11
> -#define D_l x12
> -#define D_h x13
> -#define E_l x14
> -#define E_h x15
> -#define F_l x16
> -#define F_h x17
> -#define G_l count
> -#define G_h dst
> -#define H_l src
> -#define H_h srcend
> -#define tmp1 x14
> + .macro ldr1 reg, addr:vararg
> + ldr \reg, \addr
> + .endm
>
> -/* This implementation handles overlaps and supports both memcpy and memmove
> - from a single entry point. It uses unaligned accesses and branchless
> - sequences to keep the code small, simple and improve performance.
> + .macro ldp1 reg1, reg2, addr:vararg
> + ldp \reg1, \reg2, \addr
> + .endm
>
> - Copies are split into 3 main cases: small copies of up to 32 bytes, medium
> - copies of up to 128 bytes, and large copies. The overhead of the overlap
> - check is negligible since it is only required for large copies.
> + .macro ret1
> + ret
> + .endm
>
> - Large copies use a software pipelined loop processing 64 bytes per iteration.
> - The destination pointer is 16-byte aligned to minimize unaligned accesses.
> - The loop tail is handled by always copying 64 bytes from the end.
> -*/
> + .macro cpy1 dst, src, count
> + .arch_extension mops
> + cpyp [\dst]!, [\src]!, \count!
> + cpym [\dst]!, [\src]!, \count!
> + cpye [\dst]!, [\src]!, \count!
> + .endm
>
> -SYM_FUNC_START_LOCAL(__pi_memcpy_generic)
> - add srcend, src, count
> - add dstend, dstin, count
> - cmp count, 128
> - b.hi L(copy_long)
> - cmp count, 32
> - b.hi L(copy32_128)
> -
> - /* Small copies: 0..32 bytes. */
> - cmp count, 16
> - b.lo L(copy16)
> - ldp A_l, A_h, [src]
> - ldp D_l, D_h, [srcend, -16]
> - stp A_l, A_h, [dstin]
> - stp D_l, D_h, [dstend, -16]
> - ret
> -
> - /* Copy 8-15 bytes. */
> -L(copy16):
> - tbz count, 3, L(copy8)
> - ldr A_l, [src]
> - ldr A_h, [srcend, -8]
> - str A_l, [dstin]
> - str A_h, [dstend, -8]
> - ret
> -
> - .p2align 3
> - /* Copy 4-7 bytes. */
> -L(copy8):
> - tbz count, 2, L(copy4)
> - ldr A_lw, [src]
> - ldr B_lw, [srcend, -4]
> - str A_lw, [dstin]
> - str B_lw, [dstend, -4]
> - ret
> -
> - /* Copy 0..3 bytes using a branchless sequence. */
> -L(copy4):
> - cbz count, L(copy0)
> - lsr tmp1, count, 1
> - ldrb A_lw, [src]
> - ldrb C_lw, [srcend, -1]
> - ldrb B_lw, [src, tmp1]
> - strb A_lw, [dstin]
> - strb B_lw, [dstin, tmp1]
> - strb C_lw, [dstend, -1]
> -L(copy0):
> - ret
> -
> - .p2align 4
> - /* Medium copies: 33..128 bytes. */
> -L(copy32_128):
> - ldp A_l, A_h, [src]
> - ldp B_l, B_h, [src, 16]
> - ldp C_l, C_h, [srcend, -32]
> - ldp D_l, D_h, [srcend, -16]
> - cmp count, 64
> - b.hi L(copy128)
> - stp A_l, A_h, [dstin]
> - stp B_l, B_h, [dstin, 16]
> - stp C_l, C_h, [dstend, -32]
> - stp D_l, D_h, [dstend, -16]
> - ret
> -
> - .p2align 4
> - /* Copy 65..128 bytes. */
> -L(copy128):
> - ldp E_l, E_h, [src, 32]
> - ldp F_l, F_h, [src, 48]
> - cmp count, 96
> - b.ls L(copy96)
> - ldp G_l, G_h, [srcend, -64]
> - ldp H_l, H_h, [srcend, -48]
> - stp G_l, G_h, [dstend, -64]
> - stp H_l, H_h, [dstend, -48]
> -L(copy96):
> - stp A_l, A_h, [dstin]
> - stp B_l, B_h, [dstin, 16]
> - stp E_l, E_h, [dstin, 32]
> - stp F_l, F_h, [dstin, 48]
> - stp C_l, C_h, [dstend, -32]
> - stp D_l, D_h, [dstend, -16]
> - ret
> -
> - .p2align 4
> - /* Copy more than 128 bytes. */
> -L(copy_long):
> - /* Use backwards copy if there is an overlap. */
> - sub tmp1, dstin, src
> - cbz tmp1, L(copy0)
> - cmp tmp1, count
> - b.lo L(copy_long_backwards)
> -
> - /* Copy 16 bytes and then align dst to 16-byte alignment. */
> -
> - ldp D_l, D_h, [src]
> - and tmp1, dstin, 15
> - bic dst, dstin, 15
> - sub src, src, tmp1
> - add count, count, tmp1 /* Count is now 16 too large. */
> - ldp A_l, A_h, [src, 16]
> - stp D_l, D_h, [dstin]
> - ldp B_l, B_h, [src, 32]
> - ldp C_l, C_h, [src, 48]
> - ldp D_l, D_h, [src, 64]!
> - subs count, count, 128 + 16 /* Test and readjust count. */
> - b.ls L(copy64_from_end)
> -
> -L(loop64):
> - stp A_l, A_h, [dst, 16]
> - ldp A_l, A_h, [src, 16]
> - stp B_l, B_h, [dst, 32]
> - ldp B_l, B_h, [src, 32]
> - stp C_l, C_h, [dst, 48]
> - ldp C_l, C_h, [src, 48]
> - stp D_l, D_h, [dst, 64]!
> - ldp D_l, D_h, [src, 64]!
> - subs count, count, 64
> - b.hi L(loop64)
> -
> - /* Write the last iteration and copy 64 bytes from the end. */
> -L(copy64_from_end):
> - ldp E_l, E_h, [srcend, -64]
> - stp A_l, A_h, [dst, 16]
> - ldp A_l, A_h, [srcend, -48]
> - stp B_l, B_h, [dst, 32]
> - ldp B_l, B_h, [srcend, -32]
> - stp C_l, C_h, [dst, 48]
> - ldp C_l, C_h, [srcend, -16]
> - stp D_l, D_h, [dst, 64]
> - stp E_l, E_h, [dstend, -64]
> - stp A_l, A_h, [dstend, -48]
> - stp B_l, B_h, [dstend, -32]
> - stp C_l, C_h, [dstend, -16]
> - ret
> -
> - .p2align 4
> -
> - /* Large backwards copy for overlapping copies.
> - Copy 16 bytes and then align dst to 16-byte alignment. */
> -L(copy_long_backwards):
> - ldp D_l, D_h, [srcend, -16]
> - and tmp1, dstend, 15
> - sub srcend, srcend, tmp1
> - sub count, count, tmp1
> - ldp A_l, A_h, [srcend, -16]
> - stp D_l, D_h, [dstend, -16]
> - ldp B_l, B_h, [srcend, -32]
> - ldp C_l, C_h, [srcend, -48]
> - ldp D_l, D_h, [srcend, -64]!
> - sub dstend, dstend, tmp1
> - subs count, count, 128
> - b.ls L(copy64_from_start)
> -
> -L(loop64_backwards):
> - stp A_l, A_h, [dstend, -16]
> - ldp A_l, A_h, [srcend, -16]
> - stp B_l, B_h, [dstend, -32]
> - ldp B_l, B_h, [srcend, -32]
> - stp C_l, C_h, [dstend, -48]
> - ldp C_l, C_h, [srcend, -48]
> - stp D_l, D_h, [dstend, -64]!
> - ldp D_l, D_h, [srcend, -64]!
> - subs count, count, 64
> - b.hi L(loop64_backwards)
> -
> - /* Write the last iteration and copy 64 bytes from the start. */
> -L(copy64_from_start):
> - ldp G_l, G_h, [src, 48]
> - stp A_l, A_h, [dstend, -16]
> - ldp A_l, A_h, [src, 32]
> - stp B_l, B_h, [dstend, -32]
> - ldp B_l, B_h, [src, 16]
> - stp C_l, C_h, [dstend, -48]
> - ldp C_l, C_h, [src]
> - stp D_l, D_h, [dstend, -64]
> - stp G_l, G_h, [dstin, 48]
> - stp A_l, A_h, [dstin, 32]
> - stp B_l, B_h, [dstin, 16]
> - stp C_l, C_h, [dstin]
> - ret
> -SYM_FUNC_END(__pi_memcpy_generic)
> -
> -#ifdef CONFIG_AS_HAS_MOPS
> - .arch_extension mops
> SYM_FUNC_START(__pi_memcpy)
> -alternative_if_not ARM64_HAS_MOPS
> - b __pi_memcpy_generic
> -alternative_else_nop_endif
> -
> - mov dst, dstin
> - cpyp [dst]!, [src]!, count!
> - cpym [dst]!, [src]!, count!
> - cpye [dst]!, [src]!, count!
> - ret
> +#include "memcpy_template.S"
> SYM_FUNC_END(__pi_memcpy)
> -#else
> -SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic)
> -#endif
>
> SYM_FUNC_ALIAS(__memcpy, __pi_memcpy)
> EXPORT_SYMBOL(__memcpy)
> diff --git a/arch/arm64/lib/memcpy_mc.S b/arch/arm64/lib/memcpy_mc.S
> new file mode 100644
> index 000000000000..90624d35af4b
> --- /dev/null
> +++ b/arch/arm64/lib/memcpy_mc.S
> @@ -0,0 +1,56 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2012-2021, Arm Limited.
> + *
> + * Adapted from the original at:
> + * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
> + */
> +
> +#include <linux/linkage.h>
> +#include <asm/assembler.h>
> +#include <asm/asm-uaccess.h>
> +
> +/* Assumptions:
> + *
> + * ARMv8-a, AArch64, unaligned accesses.
> + *
> + */
> +
> + .macro ldrb1 reg, addr:vararg
> + KERNEL_MEM_ERR(9998f, ldrb \reg, \addr)
> + .endm
> +
> + .macro ldr1 reg, addr:vararg
> + KERNEL_MEM_ERR(9998f, ldr \reg, \addr)
> + .endm
> +
> + .macro ldp1 reg1, reg2, addr:vararg
> + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, \addr)
> + .endm
> +
> + .macro ret1
> + mov x0, #0
> + ret
> + .endm
> +
> + .macro cpy1 dst, src, count
> + .arch_extension mops
> + USER_CPY(9998f, 0, cpyp [\dst]!, [\src]!, \count!)
> + USER_CPY(9996f, 0, cpym [\dst]!, [\src]!, \count!)
> + USER_CPY(9996f, 0, cpye [\dst]!, [\src]!, \count!)
memcpy_mc.S annotates kernel-to-kernel MOPS with USER_CPY, registering
EX_TYPE_UACCESS_CPY entries. fixup_exception_me() then routes those
through ex_handler_uaccess_cpy(...., esr=0), whose
cpy_faulted_on_uaccess() applies page-fault read/write match
semantics. It only happens to work today because uaccess_is_write=0
matches a hard-coded fault_on_write=0; any tightening (e.g. real esr
threading) immediately breaks recovery.
Please fix this in conjunction with patch 3:
(a) introduce KERNEL_CPY / EX_TYPE_KACCESS_CPY_MC, or
(b) give EX_TYPE_UACCESS_CPY a separate MC handler in
fixup_exception_me() that just redirects PC.
> + .endm
> +
> +SYM_FUNC_START(__memcpy_mc)
> +#include "memcpy_template.S"
> +
> + // Exception fixups
> +9996: b.cs 9998f
> + // Registers are in Option A format
> + add dst, dst, count
> +9998: sub x0, dstend, dstin // bytes not copied
The MOPS branch in memcpy_template.S executes cpy1 *before* the
no_mops block runs `add dstend, dstin, count`. So if the cpy1 takes
an SEA, control jumps to the 9998 fixup in memcpy_mc.S:
9998: sub x0, dstend, dstin // bytes not copied
ret
with dstend uninitialised. The "bytes not copied" return value is
arbitrary garbage in this case, and copy_mc_to_kernel() will pass
that to its caller (dax/pmem), which will then re-touch dst[0..ret]
based on a bogus boundary. Please initialise srcend/dstend before
the cpy1, or save the original count to a callee-saved register and
return it directly
> + ret
> +SYM_FUNC_END(__memcpy_mc)
> +
> +EXPORT_SYMBOL(__memcpy_mc)
> +SYM_FUNC_ALIAS_WEAK(memcpy_mc, __memcpy_mc)
> +EXPORT_SYMBOL(memcpy_mc)
> diff --git a/arch/arm64/lib/memcpy_template.S b/arch/arm64/lib/memcpy_template.S
> new file mode 100644
> index 000000000000..205516c6e076
> --- /dev/null
> +++ b/arch/arm64/lib/memcpy_template.S
> @@ -0,0 +1,249 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (c) 2012-2021, Arm Limited.
> + *
> + * Adapted from the original at:
> + * https://github.com/ARM-software/optimized-routines/blob/afd6244a1f8d9229/string/aarch64/memcpy.S
> + */
> +
> +#include <linux/linkage.h>
> +#include <asm/assembler.h>
> +
> +/* Assumptions:
> + *
> + * ARMv8-a, AArch64, unaligned accesses.
> + *
> + */
> +
> +#define L(label) .L ## label
> +
> +#define dstin x0
> +#define src x1
> +#define count x2
> +#define dst x3
> +#define srcend x4
> +#define dstend x5
> +#define A_l x6
> +#define A_lw w6
> +#define A_h x7
> +#define B_l x8
> +#define B_lw w8
> +#define B_h x9
> +#define C_l x10
> +#define C_lw w10
> +#define C_h x11
> +#define D_l x12
> +#define D_h x13
> +#define E_l x14
> +#define E_h x15
> +#define F_l x16
> +#define F_h x17
> +#define G_l count
> +#define G_h dst
> +#define H_l src
> +#define H_h srcend
> +#define tmp1 x14
> +
> +/* This implementation handles overlaps and supports both memcpy and memmove
> + from a single entry point. It uses unaligned accesses and branchless
> + sequences to keep the code small, simple and improve performance.
> +
> + Copies are split into 3 main cases: small copies of up to 32 bytes, medium
> + copies of up to 128 bytes, and large copies. The overhead of the overlap
> + check is negligible since it is only required for large copies.
> +
> + Large copies use a software pipelined loop processing 64 bytes per iteration.
> + The destination pointer is 16-byte aligned to minimize unaligned accesses.
> + The loop tail is handled by always copying 64 bytes from the end.
> +*/
> +
> +#ifdef CONFIG_AS_HAS_MOPS
> +alternative_if_not ARM64_HAS_MOPS
> + b L(no_mops):
Trailing colon. Should be:
b L(no_mops)
GNU as either rejects this outright or accepts it as nonsense. Either
way, FEAT_MOPS-capable toolchain builds (binutils >= 2.39, recent LLVM)
will fail or produce broken code. Please verify v2 with both
CONFIG_AS_HAS_MOPS=y and =n.
> +alternative_else_nop_endif
> +
> + cpy1 dst, src, count
Does this execute the hardware copy instruction using an uninitialized
dst (x3) register?
Before the refactoring, memcpy.S had:
mov dst, dstin ; x3 = x0 (preserve x0 as return value)
cpyp [dst]!, [src]!, count!
cpym [dst]!, [src]!, count!
cpye [dst]!, [src]!, count!
ret ; x0 still holds original dest
After: the template jumps straight to `cpy1 dst, src, count` which
expands to `cpyp [x3]!, [x1]!, x2!` with x3 uninitialised. On any
CPU with FEAT_MOPS (Cortex-X3/X4, A720, etc.) this writes src data to
whatever garbage address x3 held at function entry -- silent memory
corruption or an immediate abort, depending on luck.
This affects both plain memcpy() (through memcpy.S -> memcpy_template.S)
and memcpy_mc() (through memcpy_mc.S -> memcpy_template.S).
> + ret1
> +#endif
> +
> +L(no_mops):
> + add srcend, src, count
> + add dstend, dstin, count
> + cmp count, 128
> + b.hi L(copy_long)
> + cmp count, 32
> + b.hi L(copy32_128)
> +
> + /* Small copies: 0..32 bytes. */
> + cmp count, 16
> + b.lo L(copy16)
> + ldp1 A_l, A_h, [src]
> + ldp1 D_l, D_h, [srcend, -16]
> + stp A_l, A_h, [dstin]
> + stp D_l, D_h, [dstend, -16]
> + ret1
> +
> + /* Copy 8-15 bytes. */
> +L(copy16):
> + tbz count, 3, L(copy8)
> + ldr1 A_l, [src]
> + ldr1 A_h, [srcend, -8]
> + str A_l, [dstin]
> + str A_h, [dstend, -8]
> + ret1
> +
> + .p2align 3
> + /* Copy 4-7 bytes. */
> +L(copy8):
> + tbz count, 2, L(copy4)
> + ldr1 A_lw, [src]
> + ldr1 B_lw, [srcend, -4]
> + str A_lw, [dstin]
> + str B_lw, [dstend, -4]
> + ret1
> +
> + /* Copy 0..3 bytes using a branchless sequence. */
> +L(copy4):
> + cbz count, L(copy0)
> + lsr tmp1, count, 1
> + ldrb1 A_lw, [src]
> + ldrb1 C_lw, [srcend, -1]
> + ldrb1 B_lw, [src, tmp1]
> + strb A_lw, [dstin]
> + strb B_lw, [dstin, tmp1]
> + strb C_lw, [dstend, -1]
> +L(copy0):
> + ret1
> +
> + .p2align 4
> + /* Medium copies: 33..128 bytes. */
> +L(copy32_128):
> + ldp1 A_l, A_h, [src]
> + ldp1 B_l, B_h, [src, 16]
> + ldp1 C_l, C_h, [srcend, -32]
> + ldp1 D_l, D_h, [srcend, -16]
> + cmp count, 64
> + b.hi L(copy128)
> + stp A_l, A_h, [dstin]
> + stp B_l, B_h, [dstin, 16]
> + stp C_l, C_h, [dstend, -32]
> + stp D_l, D_h, [dstend, -16]
> + ret1
> +
> + .p2align 4
> + /* Copy 65..128 bytes. */
> +L(copy128):
> + ldp1 E_l, E_h, [src, 32]
> + ldp1 F_l, F_h, [src, 48]
> + cmp count, 96
> + b.ls L(copy96)
> + ldp1 G_l, G_h, [srcend, -64]
> + ldp1 H_l, H_h, [srcend, -48]
> + stp G_l, G_h, [dstend, -64]
> + stp H_l, H_h, [dstend, -48]
> +L(copy96):
> + stp A_l, A_h, [dstin]
> + stp B_l, B_h, [dstin, 16]
> + stp E_l, E_h, [dstin, 32]
> + stp F_l, F_h, [dstin, 48]
> + stp C_l, C_h, [dstend, -32]
> + stp D_l, D_h, [dstend, -16]
> + ret1
> +
> + .p2align 4
> + /* Copy more than 128 bytes. */
> +L(copy_long):
> + /* Use backwards copy if there is an overlap. */
> + sub tmp1, dstin, src
> + cbz tmp1, L(copy0)
> + cmp tmp1, count
> + b.lo L(copy_long_backwards)
> +
> + /* Copy 16 bytes and then align dst to 16-byte alignment. */
> +
> + ldp1 D_l, D_h, [src]
> + and tmp1, dstin, 15
> + bic dst, dstin, 15
> + sub src, src, tmp1
> + add count, count, tmp1 /* Count is now 16 too large. */
> + ldp1 A_l, A_h, [src, 16]
> + stp D_l, D_h, [dstin]
> + ldp1 B_l, B_h, [src, 32]
> + ldp1 C_l, C_h, [src, 48]
> + ldp1 D_l, D_h, [src, 64]!
> + subs count, count, 128 + 16 /* Test and readjust count. */
> + b.ls L(copy64_from_end)
> +
> +L(loop64):
> + stp A_l, A_h, [dst, 16]
> + ldp1 A_l, A_h, [src, 16]
> + stp B_l, B_h, [dst, 32]
> + ldp1 B_l, B_h, [src, 32]
> + stp C_l, C_h, [dst, 48]
> + ldp1 C_l, C_h, [src, 48]
> + stp D_l, D_h, [dst, 64]!
> + ldp1 D_l, D_h, [src, 64]!
> + subs count, count, 64
> + b.hi L(loop64)
> +
> + /* Write the last iteration and copy 64 bytes from the end. */
> +L(copy64_from_end):
> + ldp1 E_l, E_h, [srcend, -64]
> + stp A_l, A_h, [dst, 16]
> + ldp1 A_l, A_h, [srcend, -48]
> + stp B_l, B_h, [dst, 32]
> + ldp1 B_l, B_h, [srcend, -32]
> + stp C_l, C_h, [dst, 48]
> + ldp1 C_l, C_h, [srcend, -16]
> + stp D_l, D_h, [dst, 64]
> + stp E_l, E_h, [dstend, -64]
> + stp A_l, A_h, [dstend, -48]
> + stp B_l, B_h, [dstend, -32]
> + stp C_l, C_h, [dstend, -16]
> + ret1
> +
> + .p2align 4
> +
> + /* Large backwards copy for overlapping copies.
> + Copy 16 bytes and then align dst to 16-byte alignment. */
> +L(copy_long_backwards):
> + ldp1 D_l, D_h, [srcend, -16]
> + and tmp1, dstend, 15
> + sub srcend, srcend, tmp1
> + sub count, count, tmp1
> + ldp1 A_l, A_h, [srcend, -16]
> + stp D_l, D_h, [dstend, -16]
> + ldp1 B_l, B_h, [srcend, -32]
> + ldp1 C_l, C_h, [srcend, -48]
> + ldp1 D_l, D_h, [srcend, -64]!
> + sub dstend, dstend, tmp1
> + subs count, count, 128
> + b.ls L(copy64_from_start)
> +
> +L(loop64_backwards):
> + stp A_l, A_h, [dstend, -16]
> + ldp1 A_l, A_h, [srcend, -16]
> + stp B_l, B_h, [dstend, -32]
> + ldp1 B_l, B_h, [srcend, -32]
> + stp C_l, C_h, [dstend, -48]
> + ldp1 C_l, C_h, [srcend, -48]
> + stp D_l, D_h, [dstend, -64]!
> + ldp1 D_l, D_h, [srcend, -64]!
> + subs count, count, 64
> + b.hi L(loop64_backwards)
> +
> + /* Write the last iteration and copy 64 bytes from the start. */
> +L(copy64_from_start):
> + ldp1 G_l, G_h, [src, 48]
> + stp A_l, A_h, [dstend, -16]
> + ldp1 A_l, A_h, [src, 32]
> + stp B_l, B_h, [dstend, -32]
> + ldp1 B_l, B_h, [src, 16]
> + stp C_l, C_h, [dstend, -48]
> + ldp1 C_l, C_h, [src]
> + stp D_l, D_h, [dstend, -64]
> + stp G_l, G_h, [dstin, 48]
> + stp A_l, A_h, [dstin, 32]
> + stp B_l, B_h, [dstin, 16]
> + stp C_l, C_h, [dstin]
> + ret1
> diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
> index d286e0a04543..3128f0d9cc46 100644
> --- a/mm/kasan/shadow.c
> +++ b/mm/kasan/shadow.c
> @@ -79,6 +79,18 @@ void *memcpy(void *dest, const void *src, size_t len)
> }
> #endif
>
> +#ifdef __HAVE_ARCH_MEMCPY_MC
> +#undef memcpy_mc
> +int memcpy_mc(void *dest, const void *src, size_t len)
> +{
> + if (!kasan_check_range(src, len, false, _RET_IP_) ||
> + !kasan_check_range(dest, len, true, _RET_IP_))
> + return (int)len;
> +
> + return __memcpy_mc(dest, src, len);
> +}
memcpy_mc() is declared `int` while copy_mc_to_kernel() returns
unsigned long, and the asm side returns the full 64-bit (dstend - dstin).
For len >= 2GiB (real on dax_copy_to_iter over large NVDIMM regions)
the int truncation produces garbage "bytes not copied". Please use
unsigned long throughout:
extern unsigned long memcpy_mc(void *, const void *,
__kernel_size_t);
extern unsigned long __memcpy_mc(void *, const void *,
__kernel_size_t);
and in mm/kasan/shadow.c drop the `(int)len` cast accordingly:
unsigned long memcpy_mc(...)
{
...
return len; /* not (int)len */
return __memcpy_mc(...);
}
Thanks.
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
2026-05-18 8:49 ` [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test Ruidong Tian
@ 2026-05-28 3:17 ` Shuai Xue
0 siblings, 0 replies; 20+ messages in thread
From: Shuai Xue @ 2026-05-28 3:17 UTC (permalink / raw)
To: Ruidong Tian, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
On 5/18/26 4:49 PM, Ruidong Tian wrote:
> memcpy_mc() is the Machine-Check safe memcpy variant that returns the
> number of bytes NOT copied on a hardware memory error, or 0 on success.
>
> Add two test cases modeled after the existing memcpy_test() and
> memcpy_large_test() implementations:
Same build issue as with copy_mc_page_test: memcpy_mc() is an arm64-
only symbol.
1. Cross-architecture build break (BLOCKER, same as patch 6)
These tests are gated on CONFIG_ARCH_HAS_COPY_MC, which is also
selected by x86_64 and ppc64. Neither architecture provides a
memcpy_mc() symbol -- they use copy_mc_to_kernel() directly.
On x86_64:
lib/tests/memcpy_kunit.c: error: implicit declaration of
function 'memcpy_mc' [-Werror=implicit-function-declaration]
Fix: guard with __HAVE_ARCH_MEMCPY_MC instead:
#ifdef __HAVE_ARCH_MEMCPY_MC
static void memcpy_mc_test(...) { ... }
static void memcpy_mc_large_test(...) { ... }
#endif
>
> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
> ---
> lib/tests/memcpy_kunit.c | 113 ++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 112 insertions(+), 1 deletion(-)
>
> diff --git a/lib/tests/memcpy_kunit.c b/lib/tests/memcpy_kunit.c
> index 85df53ccfb0c..b4b2dafb50f1 100644
> --- a/lib/tests/memcpy_kunit.c
> +++ b/lib/tests/memcpy_kunit.c
> @@ -552,6 +552,115 @@ static void copy_mc_page_test(struct kunit *test)
> memcmp(page_dst + PAGE_SIZE, page_zero, PAGE_SIZE), 0,
> "copy_mc_page overflow into adjacent page");
> }
> +/*
> + * memcpy_mc() is a Machine-Check safe memcpy variant.
> + * Signature: int memcpy_mc(void *dst, const void *src, size_t len)
> + * Returns: 0 on success, or number of bytes NOT copied on MC error.
> + *
> + * In the normal (no-poison) path it must behave identically to memcpy()
> + * and always return 0.
> + */
> +static void memcpy_mc_test(struct kunit *test)
> +{
> +#define TEST_OP "memcpy_mc"
> + struct some_bytes control = {
> + .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + },
> + };
> + struct some_bytes zero = { };
> + struct some_bytes middle = {
> + .data = { 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x00, 0x00, 0x00, 0x00,
> + 0x00, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + },
> + };
> + struct some_bytes three = {
> + .data = { 0x00, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
> + },
> + };
> + struct some_bytes dest = { };
> + int ret, count;
> + u8 *ptr;
> +
> + /* Verify static initializers. */
> + check(control, 0x20);
> + check(zero, 0);
> + compare("static initializers", dest, zero);
> +
> + /* Verify assignment. */
> + dest = control;
> + compare("direct assignment", dest, control);
> +
> + /* Verify complete overwrite. */
> + ret = memcpy_mc(dest.data, control.data, sizeof(dest.data));
> + KUNIT_ASSERT_EQ(test, ret, 0);
> + compare("complete overwrite", dest, control);
> +
> + /* Verify middle overwrite: 7 bytes at offset 12. */
> + dest = control;
> + ret = memcpy_mc(dest.data + 12, zero.data, 7);
> + KUNIT_ASSERT_EQ(test, ret, 0);
> + compare("middle overwrite", dest, middle);
> +
> + /* Verify zero-length copy is a no-op. */
> + dest = control;
> + ret = memcpy_mc(dest.data, zero.data, 0);
> + KUNIT_ASSERT_EQ(test, ret, 0);
> + compare("zero length", dest, control);
> +
> + /* Verify argument side-effects aren't repeated. */
> + dest = control;
> + ptr = dest.data;
> + count = 1;
> + memcpy(ptr++, zero.data, count++);
> + ptr += 8;
> + memcpy(ptr++, zero.data, count++);
This is a verbatim paste from memcpy_test(). The intent is to
verify that the memcpy_mc macro doesn't double-evaluate arguments,
but the test doesn't actually call memcpy_mc(). Please change the
two memcpy() calls to memcpy_mc() and assert their return values.
Thanks
Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC
2026-05-27 11:35 ` Shuai Xue
@ 2026-06-04 8:10 ` Ruidong Tian
0 siblings, 0 replies; 20+ messages in thread
From: Ruidong Tian @ 2026-06-04 8:10 UTC (permalink / raw)
To: Shuai Xue, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, tongtiangen, james.morse, robin.murphy, andreyknvl,
dvyukov, vincenzo.frascino, mpe, npiggin, ryabinin.a.a, glider,
christophe.leroy, aneesh.kumar, naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
在 2026/5/27 19:35, Shuai Xue 写道:
>
>
> On 5/18/26 4:49 PM, Ruidong Tian wrote:
>> From: Tong Tiangen <tongtiangen@huawei•com>
>>
>> For the arm64 kernel, when it processes hardware memory errors for
>> synchronize notifications(do_sea()), if the errors is consumed within the
>> kernel, the current processing is panic. However, it is not optimal.
>>
>> Take copy_from/to_user for example, If ld* triggers a memory error,
>> even in
>> kernel mode, only the associated process is affected. Killing the user
>> process and isolating the corrupt page is a better choice.
>>
>> Add new fixup type EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR to identify insn
>> that can recover from memory errors triggered by access to kernel memory,
>> and this fixup type is used in __arch_copy_to_user(), This make the
>> regular
>> copy_to_user() will handle kernel memory errors.
>>
>> [Ruidong: handle EX_TYPE_UACCESS_CPY in fixup_exception_me()]
>>
>> Signed-off-by: Tong Tiangen <tongtiangen@huawei•com>
>> Signed-off-by: Ruidong Tian <tianruidong@linux•alibaba.com>
>> ---
>> arch/arm64/Kconfig | 1 +
>> arch/arm64/include/asm/asm-extable.h | 22 +++++++++++++++++++-
>> arch/arm64/include/asm/asm-uaccess.h | 4 ++++
>> arch/arm64/include/asm/extable.h | 1 +
>> arch/arm64/lib/copy_to_user.S | 10 +++++-----
>> arch/arm64/mm/extable.c | 21 +++++++++++++++++++
>> arch/arm64/mm/fault.c | 30 ++++++++++++++++++++--------
>> 7 files changed, 75 insertions(+), 14 deletions(-)
>>
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index fe60738e5943..831b20d45893 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -21,6 +21,7 @@ config ARM64
>> select ARCH_HAS_CACHE_LINE_SIZE
>> select ARCH_HAS_CC_PLATFORM
>> select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
>> + select ARCH_HAS_COPY_MC if ACPI_APEI_GHES
>
>
> At this commit:
>
> - arch/arm64/lib/memcpy_mc.S does not exist (patch 7)
> - arch/arm64/lib/copy_mc_page.S does not exist (patch 5)
> - arm64 has no copy_mc_to_kernel() override
> - __HAVE_ARCH_COPY_MC_USER_HIGHPAGE is not defined
>
> Build does not break because the generic fallback in
> include/linux/uaccess.h and include/linux/highmem.h covers it, but
> ARCH_HAS_COPY_MC=y silently means "plain memcpy() with no MC
> handling at all" between this commit and patch 7. Anyone bisecting
> an MC regression in this window will be very confused.
>
> Please move this select to the last arm64 implementation patch in
> the series.
>
>
>> select ARCH_HAS_CURRENT_STACK_POINTER
>> select ARCH_HAS_DEBUG_VIRTUAL
>> select ARCH_HAS_DEBUG_VM_PGTABLE
>> diff --git a/arch/arm64/include/asm/asm-extable.h b/arch/arm64/
>> include/asm/asm-extable.h
>> index d67e2fdd1aee..4980023f2fbd 100644
>> --- a/arch/arm64/include/asm/asm-extable.h
>> +++ b/arch/arm64/include/asm/asm-extable.h
>> @@ -11,6 +11,8 @@
>> #define EX_TYPE_KACCESS_ERR_ZERO 3
>> #define EX_TYPE_UACCESS_CPY 4
>> #define EX_TYPE_LOAD_UNALIGNED_ZEROPAD 5
>> +/* kernel access memory error safe */
>> +#define EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR 6
>
> KACCESS_ERR_ZERO is already "encode err reg + zero reg". Tacking
> _MEM_ERR on the end reads like "+ another mem-err reg". Please
> rename to e.g. EX_TYPE_KACCESS_ERR_ZERO_MC, which directly tells
> the reader "MC-safe variant of KACCESS_ERR_ZERO".
MC is an x86-specific term and may not be appropriate here. I plan to
rename this to EX_TYPE_KACCESS_SEA (and drop ERR_ZERO, since this case
does not need it) so it is clear that the current fixup is caused by SEA
rather than a translation fault.
>
>> /* Data fields for EX_TYPE_UACCESS_ERR_ZERO */
>> #define EX_DATA_REG_ERR_SHIFT 0
>> @@ -42,7 +44,7 @@
>> (.L__gpr_num_##gpr << EX_DATA_REG_##reg##_SHIFT)
>> #define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \
>> - __ASM_EXTABLE_RAW(insn, fixup, \
>> + __ASM_EXTABLE_RAW(insn, fixup, \
>> EX_TYPE_UACCESS_ERR_ZERO, \
>> ( \
>> EX_DATA_REG(ERR, err) | \
>> @@ -55,6 +57,17 @@
>> #define _ASM_EXTABLE_UACCESS(insn, fixup) \
>> _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, wzr, wzr)
>> +#define _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, err,
>> zero) \
>> + __ASM_EXTABLE_RAW(insn, fixup, \
>> + EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR, \
>> + ( \
>> + EX_DATA_REG(ERR, err) | \
>> + EX_DATA_REG(ZERO, zero) \
>> + ))
>> +
>> +#define _ASM_EXTABLE_KACCESS_MEM_ERR(insn, fixup) \
>> + _ASM_EXTABLE_KACCESS_ERR_ZERO_MEM_ERR(insn, fixup, wzr, wzr)
>> +
>> /*
>> * Create an exception table entry for uaccess `insn`, which will
>> branch to `fixup`
>> * when an unhandled fault is taken.
>> @@ -76,6 +89,13 @@
>> .macro _asm_extable_uaccess_cpy, insn, fixup,
>> uaccess_is_write
>> __ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_UACCESS_CPY,
>> \uaccess_is_write)
>> .endm
>> +/*
>> + * Create an exception table entry for kaccess `insn`, which will
>> branch to
>> + * `fixup` when an unhandled fault is taken.
>> + */
>> + .macro _asm_extable_kaccess_mem_err, insn, fixup
>> + _ASM_EXTABLE_KACCESS_MEM_ERR(\insn, \fixup)
>> + .endm
>> #else /* __ASSEMBLER__ */
>> diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/
>> include/asm/asm-uaccess.h
>> index 12aa6a283249..c8f0af5fde63 100644
>> --- a/arch/arm64/include/asm/asm-uaccess.h
>> +++ b/arch/arm64/include/asm/asm-uaccess.h
>> @@ -57,6 +57,10 @@ alternative_else_nop_endif
>> .endm
>> #endif
>> +#define KERNEL_MEM_ERR(l, x...) \
>> +9999: x; \
>> + _asm_extable_kaccess_mem_err 9999b, l
>> +
>> #define USER(l, x...) \
>> 9999: x; \
>> _asm_extable_uaccess 9999b, l
>> diff --git a/arch/arm64/include/asm/extable.h b/arch/arm64/include/
>> asm/extable.h
>> index 9dc39612bdf5..47c851d7df4f 100644
>> --- a/arch/arm64/include/asm/extable.h
>> +++ b/arch/arm64/include/asm/extable.h
>> @@ -48,4 +48,5 @@ bool ex_handler_bpf(const struct
>> exception_table_entry *ex,
>> #endif /* !CONFIG_BPF_JIT */
>> bool fixup_exception(struct pt_regs *regs, unsigned long esr);
>> +bool fixup_exception_me(struct pt_regs *regs);
>> #endif
>> diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/
>> copy_to_user.S
>> index 819f2e3fc7a9..991d94ecc1a8 100644
>> --- a/arch/arm64/lib/copy_to_user.S
>> +++ b/arch/arm64/lib/copy_to_user.S
>> @@ -20,7 +20,7 @@
>> * x0 - bytes not copied
>> */
>> .macro ldrb1 reg, ptr, val
>> - ldrb \reg, [\ptr], \val
>> + KERNEL_MEM_ERR(9998f, ldrb \reg, [\ptr], \val)
>> .endm
>> .macro strb1 reg, ptr, val
>> @@ -28,7 +28,7 @@
>> .endm
>> .macro ldrh1 reg, ptr, val
>> - ldrh \reg, [\ptr], \val
>> + KERNEL_MEM_ERR(9998f, ldrh \reg, [\ptr], \val)
>> .endm
>> .macro strh1 reg, ptr, val
>> @@ -36,7 +36,7 @@
>> .endm
>> .macro ldr1 reg, ptr, val
>> - ldr \reg, [\ptr], \val
>> + KERNEL_MEM_ERR(9998f, ldr \reg, [\ptr], \val)
>> .endm
>> .macro str1 reg, ptr, val
>> @@ -44,7 +44,7 @@
>> .endm
>> .macro ldp1 reg1, reg2, ptr, val
>> - ldp \reg1, \reg2, [\ptr], \val
>> + KERNEL_MEM_ERR(9998f, ldp \reg1, \reg2, [\ptr], \val)
>> .endm
>> .macro stp1 reg1, reg2, ptr, val
>> @@ -74,7 +74,7 @@ SYM_FUNC_START(__arch_copy_to_user)
>> 9997: cmp dst, dstin
>> b.ne 9998f
>> // Before being absolutely sure we couldn't copy anything, try
>> harder
>> - ldrb tmp1w, [srcin]
>> +KERNEL_MEM_ERR(9998f, ldrb tmp1w, [srcin])
>> USER(9998f, sttrb tmp1w, [dst])
>> add dst, dst, #1
>> 9998: sub x0, end, dst // bytes not copied
>> diff --git a/arch/arm64/mm/extable.c b/arch/arm64/mm/extable.c
>> index 6e0528831cd3..f78ac7e92845 100644
>> --- a/arch/arm64/mm/extable.c
>> +++ b/arch/arm64/mm/extable.c
>> @@ -110,7 +110,28 @@ bool fixup_exception(struct pt_regs *regs,
>> unsigned long esr)
>> return ex_handler_uaccess_cpy(ex, regs, esr);
>> case EX_TYPE_LOAD_UNALIGNED_ZEROPAD:
>> return ex_handler_load_unaligned_zeropad(ex, regs);
>> + case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR:
>> + return false;
>> }
>> BUG();
>> }
>> +
>> +bool fixup_exception_me(struct pt_regs *regs)
>> +{
>> + const struct exception_table_entry *ex;
>> +
>> + ex = search_exception_tables(instruction_pointer(regs));
>> + if (!ex)
>> + return false;
>> +
>> + switch (ex->type) {
>> + case EX_TYPE_UACCESS_CPY:
>> + return ex_handler_uaccess_cpy(ex, regs, 0);
>
> Pointed by sashiko:
>
> copy_to_user.S annotates its MOPS prologue/main/epilogue with
> USER_CPY(..., 1, cpyf{p,m,e}wt ...), so uaccess_is_write=1 for the
> whole MOPS sequence. With esr=0 hard-coded here:
>
> cpy_faulted_on_uaccess(): uaccess_is_write=1, fault_on_write=0
> -> returns false
> ex_handler_uaccess_cpy() -> returns false
> fixup_exception_me() -> returns false
> do_apei_claim_sea() -> returns -ENOENT
> do_sea() -> panic
>
> So any MC SEA taken inside a MOPS copy_to_user() panics the kernel,
> exactly defeating the recovery this patch claims to provide.
>
>
>> + case EX_TYPE_UACCESS_ERR_ZERO:
>> + case EX_TYPE_KACCESS_ERR_ZERO_MEM_ERR:
>> + return ex_handler_uaccess_err_zero(ex, regs);
>> + }
>> +
>> + return false;
>> +}
>> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
>> index 0f3c5c7ca054..efbda54770be 100644
>> --- a/arch/arm64/mm/fault.c
>> +++ b/arch/arm64/mm/fault.c
>> @@ -858,21 +858,35 @@ static int do_bad(unsigned long far, unsigned
>> long esr, struct pt_regs *regs)
>> return 1; /* "fault" */
>> }
>> +/*
>> + * APEI claimed this as a firmware-first notification.
>> + * Some processing deferred to task_work before ret_to_user().
>> + */
>> +static int do_apei_claim_sea(struct pt_regs *regs)
>> +{
>> + int ret;
>> +
>> + ret = apei_claim_sea(regs);
>> + if (ret)
>> + return ret;
>> +
>> + if (!user_mode(regs) && IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC)) {
>
> The IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) test is also dead on arm64:
> ARCH_HAS_COPY_MC is selected iff ACPI_APEI_GHES, and without
> ACPI_APEI_GHES apei_claim_sea() returns -ENOENT and we never reach
> this branch. Please drop it.
>
>
> Thanks.
> Shuai
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support
2026-05-18 15:05 ` [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Kefeng Wang
@ 2026-06-05 7:33 ` Ruidong Tian
0 siblings, 0 replies; 20+ messages in thread
From: Ruidong Tian @ 2026-06-05 7:33 UTC (permalink / raw)
To: Kefeng Wang, catalin.marinas, will, rafael, tony.luck, guohanjun,
mchehab, xueshuai, tongtiangen, james.morse, robin.murphy,
andreyknvl, dvyukov, vincenzo.frascino, mpe, npiggin,
ryabinin.a.a, glider, christophe.leroy, aneesh.kumar,
naveen.n.rao, tglx, mingo
Cc: linux-arm-kernel, linux-mm, linuxppc-dev, linux-kernel, kasan-dev
在 2026/5/18 23:05, Kefeng Wang 写道:
>
>
> On 5/18/2026 4:49 PM, Ruidong Tian wrote:
>> This series continues Tong Tiangen's work on arm64 ARCH_HAS_COPY_MC
>> support. We encounter the same problem, and from a forward-looking
>> perspective, large-memory ARM machines such as Grace and Vera will suffer
>> more from this class of issues, which motivates us to push this feature
>> upstream.
>>
>> Problem
>> =========
>> With the increase of memory capacity and density, the probability of
>> memory
>> error also increases. The increasing size and density of server RAM in
>> data
>> centers and clouds have shown increased uncorrectable memory errors.
>>
>> Currently, more and more scenarios that can tolerate memory errors,
>> such as
>> COW[1,2], KSM copy[3], coredump copy[4], khugepaged[5,6], uaccess
>> copy[7],
>> etc.
>
> We have encountered more scenarios and have made more enhancements, eg,
>
> 658be46520ce mm: support poison recovery from copy_present_page()
> aa549f923f5e mm: support poison recovery from do_cow_fault()
> f00b295b9b61 fs: hugetlbfs: support poisoned recover from
> hugetlbfs_migrate_folio()
> 060913999d7a mm: migrate: support poisoned recover from migrate folio
>
> Hope that the architecture-related sections can receive relevant reviews
> and responses.
>
> Thanks.
Thanks for the additional examples and for bringing up these scenarios.
Ruidong.> > Solution
>> =========
>>
>> This patchset introduces a new processing framework on ARM64, which
>> enables
>> ARM64 to support error recovery in the above scenarios, and more
>> scenarios
>> can be expanded based on this in the future.
>>
>> In arm64, memory error handling in do_sea(), which is divided into two
>> cases:
>> 1. If the user state consumed the memory errors, the solution is to
>> kill
>> the user process and isolate the error page.
>> 2. If the kernel state consumed the memory errors, the solution is to
>> panic.
>>
>> For case 2, Undifferentiated panic may not be the optimal choice, as
>> it can
>> be handled better. In some scenarios, we can avoid panic, such as
>> uaccess,
>> if the uaccess fails due to memory error, only the user process will be
>> affected, returning an error to the caller and isolating the user page
>> with
>> hardware memory errors is a better choice.
>>
>> [1] commit d302c2398ba2 ("mm, hwpoison: when copy-on-write hits
>> poison, take page offline")
>> [2] commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage
>> copy-on-write faults")
>> [3] commit 6b970599e807 ("mm: hwpoison: support recovery from
>> ksm_might_need_to_copy()")
>> [4] commit 245f09226893 ("mm: hwpoison: coredump: support recovery
>> from dump_user_range()")
>> [5] commit 98c76c9f1ef7 ("mm/khugepaged: recover from poisoned
>> anonymous memory")
>> [6] commit 12904d953364 ("mm/khugepaged: recover from poisoned file-
>> backed memory")
>> [7] commit 278b917f8cb9 ("x86/mce: Add _ASM_EXTABLE_CPY for copy user
>> access")
>>
>> ------------------
>> Test result:
>>
>> Tested on Kunpeng 920.
>>
>> 1. copy_page(), copy_mc_page() basic function test pass, and the
>> disassembly
>> contents remains the same before and after refactor.
>>
>> 2. copy_to/from_user() access kernel NULL pointer raise translation fault
>> and dump error message then die(), test pass.
>>
>> 3. Test following scenarios: copy_from_user(), get_user(), COW.
>>
>> Before patched: trigger a hardware memory error then panic.
>> After patched: trigger a hardware memory error without panic.
>>
>> Testing step:
>> step1. start an user-process.
>> step2. poison(einj) the user-process's page.
>> step3: user-process access the poison page in kernel mode, then
>> trigger SEA.
>> step4: the kernel will not panic, only the user process is killed,
>> the poison
>> page is isolated. (before patched, the kernel will panic in
>> do_sea())
>>
>> The above tests can also be reproduced using ras-tools, which
>> provides
>> einj-based injection and validation for uaccess and COW scenarios.
>> Example usage:
>>
>> einj_mem_uc futex # get_user
>> einj_mem_uc copyin # copy_to_user
>> einj_mem_uc copy-on-write # COW
>>
>> Link: https://git.kernel.org/pub/scm/linux/kernel/git/aegl/ras-
>> tools.git
>>
>> ------------------
>>
>> Benefits
>> =========
>> According to Huawei's statistics from their storage products, memory
>> errors
>> triggered in kernel-mode by COW and page cache read (uaccess) scenarios
>> account for more than 50%. With this patchset deployed, all kernel panics
>> caused by COW and page cache memory errors are eliminated.
>> Alibaba Cloud has also observed memory errors occurring in uaccess
>> contexts.
>>
>> Since V13:
>> 1. Changed MC-safe functions to return an error rather than kill the user
>> process. When a user program invokes a syscall and the kernel
>> encounters
>> a memory error during uaccess, killing the process is unexpected; the
>> syscall should return an error.
>> 2. Added FEAT_MOPS support for the copy_page_mc paths.
>> 3. Refactored copy_page() and memcpy() on top of the shared
>> memcpy_template,
>> reducing duplicated assembly code.
>>
>> Since v12:
>> Thanks to the suggestions of Jonathan, Mark, and Mauro, the following
>> modifications
>> are made:
>> 1. Rebase to latest kernel version.
>> 2. Patch1, add Jonathan's and Mauro's review-by.
>> 3. Patch2, modified do_apei_claim_sea() according to Mark's and
>> Jonathan's suggestions,
>> and optimized the commit message according to Mark's
>> suggestions(Added description of
>> the impact on regular copy_to_user()).
>> 4. Patch3, optimized the commit message according to Mauro's
>> suggestions and add Jonathan's
>> review-by.
>> 5. Patch4, modified copy_mc_user_highpage() and Optimized the commit
>> message according to
>> Jonathan's suggestions(no functional changes).
>> 6. Patch5, optimized the commit message according to Mauro's suggestions.
>> 7. Patch4/5, FEAT_MOPS is added to the code logic. Currently, the
>> fixup is not performed
>> on the MOPS instruction.
>> 8. Remove patch6 in v12 according to Jonathan's suggestions.
>>
>> Since v11:
>> 1. Rebase to latest kernel version 6.9-rc1.
>> 2. Add patch 5, Since the problem described in "Since V10 Besides 3" has
>> been solved in a50026bdb867 ('iov_iter: get rid of 'copy_mc' flag').
>> 3. Add the benefit of applying the patch set to our company to the
>> description of patch0.
>>
>> Since V10:
>> Accroding Mark's suggestion:
>> 1. Merge V10's patch2 and patch3 to V11's patch2.
>> 2. Patch2(V11): use new fixup_type for ld* in copy_to_user(), fix fatal
>> issues (NULL kernel pointeraccess) been fixup incorrectly.
>> 3. Patch2(V11): refactoring the logic of do_sea().
>> 4. Patch4(V11): Remove duplicate assembly logic and remove do_mte().
>>
>> Besides:
>> 1. Patch2(V11): remove st* insn's fixup, st* generally not trigger
>> memory error.
>> 2. Split a part of the logic of patch2(V11) to patch5(V11), for detail,
>> see patch5(V11)'s commit msg.
>> 3. Remove patch6(v10) “arm64: introduce copy_mc_to_kernel()
>> implementation”.
>> During modification, some problems that cannot be solved in a short
>> period are found. The patch will be released after the problems are
>> solved.
>> 4. Add test result in this patch.
>> 5. Modify patchset title, do not use machine check and remove "-next".
>>
>> Since V9:
>> 1. Rebase to latest kernel version 6.8-rc2.
>> 2. Add patch 6/6 to support copy_mc_to_kernel().
>>
>> Since V8:
>> 1. Rebase to latest kernel version and fix topo in some of the patches.
>> 2. According to the suggestion of Catalin, I attempted to modify the
>> return value of function copy_mc_[user]_highpage() to bytes not
>> copied.
>> During the modification process, I found that it would be more
>> reasonable to return -EFAULT when copy error occurs (referring to
>> the
>> newly added patch 4).
>>
>> For ARM64, the implementation of copy_mc_[user]_highpage() needs to
>> consider MTE. Considering the scenario where data copying is
>> successful
>> but the MTE tag copying fails, it is also not reasonable to return
>> bytes not copied.
>> 3. Considering the recent addition of machine check safe support for
>> multiple scenarios, modify commit message for patch 5 (patch 4
>> for V8).
>>
>> Since V7:
>> Currently, there are patches supporting recover from poison
>> consumption for the cow scenario[1]. Therefore, Supporting cow
>> scenario under the arm64 architecture only needs to modify the relevant
>> code under the arch/.
>> [1]https://lore.kernel.org/lkml/20221031201029.102123-1-
>> tony.luck@intel•com/
>>
>> Since V6:
>> Resend patches that are not merged into the mainline in V6.
>>
>> Since V5:
>> 1. Add patch2/3 to add uaccess assembly helpers.
>> 2. Optimize the implementation logic of arm64_do_kernel_sea() in
>> patch8.
>> 3. Remove kernel access fixup in patch9.
>> All suggestion are from Mark.
>>
>> Since V4:
>> 1. According Michael's suggestion, add patch5.
>> 2. According Mark's suggestiog, do some restructuring to arm64
>> extable, then a new adaptation of machine check safe support is made
>> based
>> on this.
>> 3. According Mark's suggestion, support machine check safe in
>> do_mte() in
>> cow scene.
>> 4. In V4, two patches have been merged into -next, so V5 not send these
>> two patches.
>>
>> Since V3:
>> 1. According to Robin's suggestion, direct modify user_ldst and
>> user_ldp in asm-uaccess.h and modify mte.S.
>> 2. Add new macro USER_MC in asm-uaccess.h, used in copy_from_user.S
>> and copy_to_user.S.
>> 3. According to Robin's suggestion, using micro in copy_page_mc.S to
>> simplify code.
>> 4. According to KeFeng's suggestion, modify powerpc code in patch1.
>> 5. According to KeFeng's suggestion, modify mm/extable.c and some code
>> optimization.
>>
>> Since V2:
>> 1. According to Mark's suggestion, all uaccess can be recovered due to
>> memory error.
>> 2. Scenario pagecache reading is also supported as part of uaccess
>> (copy_to_user()) and duplication code problem is also solved.
>> Thanks for Robin's suggestion.
>> 3. According Mark's suggestion, update commit message of patch 2/5.
>> 4. According Borisllav's suggestion, update commit message of patch
>> 1/5.
>>
>> Since V1:
>> 1.Consistent with PPC/x86, Using CONFIG_ARCH_HAS_COPY_MC instead of
>> ARM64_UCE_KERNEL_RECOVERY.
>> 2.Add two new scenes, cow and pagecache reading.
>> 3.Fix two small bug(the first two patch).
>>
>> V1 in here:
>> https://lore.kernel.org/lkml/20220323033705.3966643-1-
>> tongtiangen@huawei•com/
>>
>> Ruidong Tian (3):
>> ACPI: APEI: GHES: use exception context to gate SIGBUS on poison
>> consumption
>> lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests
>> lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test
>>
>> Tong Tiangen (5):
>> uaccess: add generic fallback version of copy_mc_to_user()
>> arm64: add support for ARCH_HAS_COPY_MC
>> mm/hwpoison: return -EFAULT when copy fail in
>> copy_mc_[user]_highpage()
>> arm64: support copy_mc_[user]_highpage()
>> arm64: introduce copy_mc_to_kernel() implementation
>>
>> arch/arm64/Kconfig | 1 +
>> arch/arm64/include/asm/asm-extable.h | 22 ++-
>> arch/arm64/include/asm/asm-uaccess.h | 4 +
>> arch/arm64/include/asm/extable.h | 1 +
>> arch/arm64/include/asm/mte.h | 9 +
>> arch/arm64/include/asm/page.h | 10 ++
>> arch/arm64/include/asm/string.h | 5 +
>> arch/arm64/include/asm/uaccess.h | 17 ++
>> arch/arm64/kernel/acpi.c | 2 +-
>> arch/arm64/lib/Makefile | 2 +
>> arch/arm64/lib/copy_mc_page.S | 44 +++++
>> arch/arm64/lib/copy_page.S | 62 +------
>> arch/arm64/lib/copy_page_template.S | 71 ++++++++
>> arch/arm64/lib/copy_to_user.S | 10 +-
>> arch/arm64/lib/memcpy.S | 253 ++-------------------------
>> arch/arm64/lib/memcpy_mc.S | 56 ++++++
>> arch/arm64/lib/memcpy_template.S | 249 ++++++++++++++++++++++++++
>> arch/arm64/lib/mte.S | 29 +++
>> arch/arm64/mm/copypage.c | 75 ++++++++
>> arch/arm64/mm/extable.c | 21 +++
>> arch/arm64/mm/fault.c | 30 +++-
>> arch/powerpc/include/asm/uaccess.h | 1 +
>> arch/x86/include/asm/uaccess.h | 1 +
>> drivers/acpi/apei/ghes.c | 36 ++--
>> include/acpi/ghes.h | 6 +-
>> include/linux/highmem.h | 16 +-
>> include/linux/uaccess.h | 8 +
>> lib/tests/memcpy_kunit.c | 178 ++++++++++++++++++-
>> mm/kasan/shadow.c | 12 ++
>> mm/khugepaged.c | 4 +-
>> 30 files changed, 904 insertions(+), 331 deletions(-)
>> create mode 100644 arch/arm64/lib/copy_mc_page.S
>> create mode 100644 arch/arm64/lib/copy_page_template.S
>> create mode 100644 arch/arm64/lib/memcpy_mc.S
>> create mode 100644 arch/arm64/lib/memcpy_template.S
>>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2026-06-05 7:33 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-18 8:49 [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Ruidong Tian
2026-05-18 8:49 ` [PATCH v14 1/8] uaccess: add generic fallback version of copy_mc_to_user() Ruidong Tian
2026-05-27 9:16 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 2/8] ACPI: APEI: GHES: use exception context to gate SIGBUS on poison consumption Ruidong Tian
2026-05-27 9:34 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 3/8] arm64: add support for ARCH_HAS_COPY_MC Ruidong Tian
2026-05-27 11:35 ` Shuai Xue
2026-06-04 8:10 ` Ruidong Tian
2026-05-18 8:49 ` [PATCH v14 4/8] mm/hwpoison: return -EFAULT when copy fail in copy_mc_[user]_highpage() Ruidong Tian
2026-05-27 11:44 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 5/8] arm64: support copy_mc_[user]_highpage() Ruidong Tian
2026-05-27 12:11 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 6/8] lib/test: memcpy_kunit: add copy_page() and copy_mc_page() tests Ruidong Tian
2026-05-27 13:43 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 7/8] arm64: introduce copy_mc_to_kernel() implementation Ruidong Tian
2026-05-28 3:10 ` Shuai Xue
2026-05-18 8:49 ` [PATCH v14 8/8] lib/tests: memcpy_kunit: add memcpy_mc() and memcpy_mc_large() test Ruidong Tian
2026-05-28 3:17 ` Shuai Xue
2026-05-18 15:05 ` [PATCH v14 0/8] arm64: add ARCH_HAS_COPY_MC support Kefeng Wang
2026-06-05 7:33 ` Ruidong Tian
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox