From: Kevin Brodsky <kevin.brodsky@arm•com>
To: linux-hardening@vger•kernel.org
Cc: Kevin Brodsky <kevin.brodsky@arm•com>,
Andrew Morton <akpm@linux-foundation•org>,
Andy Lutomirski <luto@kernel•org>,
Catalin Marinas <catalin.marinas@arm•com>,
Dave Hansen <dave.hansen@linux•intel.com>,
"David Hildenbrand (Arm)" <david@kernel•org>,
Ira Weiny <ira.weiny@intel•com>, Jann Horn <jannh@google•com>,
Jeff Xu <jeffxu@chromium•org>, Joey Gouly <joey.gouly@arm•com>,
Kees Cook <kees@kernel•org>, Linus Walleij <linusw@kernel•org>,
Marc Zyngier <maz@kernel•org>, Mark Brown <broonie@kernel•org>,
Matthew Wilcox <willy@infradead•org>,
Maxwell Bland <mbland@motorola•com>,
"Mike Rapoport (IBM)" <rppt@kernel•org>,
Peter Zijlstra <peterz@infradead•org>,
Pierre Langlois <pierre.langlois@arm•com>,
Quentin Perret <qperret@google•com>,
Rick Edgecombe <rick.p.edgecombe@intel•com>,
Ryan Roberts <ryan.roberts@arm•com>,
Vlastimil Babka <vbabka@kernel•org>,
Will Deacon <will@kernel•org>,
Yang Shi <yang@os•amperecomputing.com>,
Yeoreum Yun <yeoreum.yun@arm•com>,
linux-arm-kernel@lists•infradead.org, linux-mm@kvack•org,
x86@kernel•org, Lorenzo Stoakes <ljs@kernel•org>,
Thomas Gleixner <tglx@kernel•org>
Subject: [PATCH RFC v8 22/24] arm64: kpkeys: Batch KPKEYS_CTX_PGTABLES switches
Date: Tue, 26 May 2026 12:16:11 +0100 [thread overview]
Message-ID: <20260526-kpkeys-v8-22-eaaacdacc67c@arm.com> (raw)
In-Reply-To: <20260526-kpkeys-v8-0-eaaacdacc67c@arm.com>
The kpkeys_hardened_pgtables feature currently switches kpkeys
context in every helper that writes to page tables, such as
set_pte(). With kpkeys implemented using POE, this entails a pair of
ISBs whenever such helper is called.
A simple way to reduce this overhead is to make use of the lazy MMU
mode. We amend the kpkeys_hardened_pgtables guard so that no kpkeys
context switch (i.e. POR_EL1 update) is issued while the lazy MMU
mode is active. Instead, we switch to KPKEYS_CTX_PGTABLES when
entering the lazy MMU mode, and restore the previous context when
exiting it.
Restoring the previous kpkeys context requires storing the original
value of POR_EL1 somewhere. This is a full 64-bit value so we cannot
simply use a TIF flag. There is no straightforward way to reuse
current->thread.por_el1 for that purpose - this is where the current
value of POR_EL1 is stored on a context switch, i.e. the value
corresponding to KPKEYS_CTX_PGTABLES inside a lazy_mmu section.
Instead, we add a new member to thread_struct to hold that value
temporarily. This isn't optimal as that member is unused outside of
lazy MMU sections, but it is the simplest option. Nesting of
sections is not a concern as arch_{enter,leave}_lazy_mmu_mode() are
not called in inner sections (nor do we need to do anything there).
A further optimisation this patch makes is to merge the ISBs when
exiting lazy_mmu mode. That is, if an ISB is going to be issued by
emit_pte_barriers() because kernel pgtables were modified in the
lazy MMU section, we skip the ISB after restoring POR_EL1. This is
done by checking TIF_LAZY_MMU_PENDING and ensuring that POR_EL1 is
restored before emit_pte_barriers() is called.
Signed-off-by: Kevin Brodsky <kevin.brodsky@arm•com>
---
arch/arm64/include/asm/pgtable.h | 50 +++++++++++++++++++++++++++++++++++---
arch/arm64/include/asm/processor.h | 1 +
2 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 20072f32677d..1c0dcfd14678 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -43,10 +43,44 @@
#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES,
- kpkeys_hardened_pgtables_enabled())
-#else
+ kpkeys_hardened_pgtables_enabled() &&
+ !is_lazy_mmu_mode_active())
+
+static void kpkeys_lazy_mmu_enter(void)
+{
+ if (!kpkeys_hardened_pgtables_enabled())
+ return;
+
+ current->thread.por_el1_lazy_mmu = kpkeys_set_context(KPKEYS_CTX_PGTABLES);
+}
+
+static void kpkeys_lazy_mmu_exit(void)
+{
+ u64 saved_por_el1;
+
+ if (!kpkeys_hardened_pgtables_enabled())
+ return;
+
+ saved_por_el1 = current->thread.por_el1_lazy_mmu;
+
+ /*
+ * We skip any barrier if TIF_LAZY_MMU_PENDING is set:
+ * emit_pte_barriers() will issue an ISB just after this function
+ * returns.
+ */
+ if (test_thread_flag(TIF_LAZY_MMU_PENDING))
+ __kpkeys_set_pkey_reg_nosync(saved_por_el1);
+ else
+ arch_kpkeys_restore_pkey_reg(saved_por_el1);
+}
+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */
KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables)
-#endif
+
+static void kpkeys_lazy_mmu_enter(void) {}
+static void kpkeys_lazy_mmu_exit(void) {}
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+
static inline void emit_pte_barriers(void)
{
@@ -79,7 +113,10 @@ static inline void queue_pte_barriers(void)
}
}
-static inline void arch_enter_lazy_mmu_mode(void) {}
+static inline void arch_enter_lazy_mmu_mode(void)
+{
+ kpkeys_lazy_mmu_enter();
+}
static inline void arch_flush_lazy_mmu_mode(void)
{
@@ -89,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void)
static inline void arch_leave_lazy_mmu_mode(void)
{
+ /*
+ * The ordering should be preserved to allow kpkeys_lazy_mmu_exit()
+ * to skip any barrier when TIF_LAZY_MMU_PENDING is set.
+ */
+ kpkeys_lazy_mmu_exit();
arch_flush_lazy_mmu_mode();
}
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index 6095322343fc..c3a86ddce637 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -193,6 +193,7 @@ struct thread_struct {
u64 tpidr2_el0;
u64 por_el0;
u64 por_el1;
+ u64 por_el1_lazy_mmu;
#ifdef CONFIG_ARM64_GCS
unsigned int gcs_el0_mode;
unsigned int gcs_el0_locked;
--
2.51.2
next prev parent reply other threads:[~2026-05-26 11:19 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-26 11:15 [PATCH RFC v8 00/24] pkeys-based page table hardening Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 01/24] mm: Introduce kpkeys Kevin Brodsky
2026-05-26 13:17 ` Linus Walleij
2026-05-27 8:24 ` Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 02/24] set_memory: Introduce set_memory_pkey() stub Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 03/24] arm64: mm: Enable overlays for all EL1 indirect permissions Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 04/24] arm64: Introduce por_elx_set_pkey_perms() helper Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 05/24] arm64: Implement asm/kpkeys.h using POE Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 06/24] arm64: set_memory: Implement set_memory_pkey() Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 07/24] arm64: Context-switch POR_EL1 Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 08/24] arm64: Initialize POR_EL1 register on cpu_resume() Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 09/24] arm64: Enable kpkeys Kevin Brodsky
2026-05-26 11:15 ` [PATCH RFC v8 10/24] memblock: Move INIT_MEMBLOCK_* macros to header Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 11/24] mm: kpkeys: Introduce kpkeys_hardened_pgtables feature Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 12/24] mm: kpkeys: Protect regular page tables Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 13/24] mm: kpkeys: Introduce early page table allocator Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 14/24] mm: kpkeys: Protect vmemmap page tables Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 15/24] mm: kpkeys: Introduce hook for protecting static " Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 16/24] arm64: kpkeys: Implement arch_supports_kpkeys_early() Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 17/24] arm64: kpkeys: Support KPKEYS_CTX_PGTABLES Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 18/24] arm64: kpkeys: Ensure the linear map can be modified Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 19/24] arm64: kpkeys: Protect early page tables Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 20/24] arm64: kpkeys: Protect init_pg_dir Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 21/24] arm64: kpkeys: Guard page table writes Kevin Brodsky
2026-05-26 11:16 ` Kevin Brodsky [this message]
2026-05-26 11:16 ` [PATCH RFC v8 23/24] arm64: kpkeys: Enable kpkeys_hardened_pgtables support Kevin Brodsky
2026-05-26 11:16 ` [PATCH RFC v8 24/24] mm: Add basic tests for kpkeys_hardened_pgtables Kevin Brodsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260526-kpkeys-v8-22-eaaacdacc67c@arm.com \
--to=kevin.brodsky@arm$(echo .)com \
--cc=akpm@linux-foundation$(echo .)org \
--cc=broonie@kernel$(echo .)org \
--cc=catalin.marinas@arm$(echo .)com \
--cc=dave.hansen@linux$(echo .)intel.com \
--cc=david@kernel$(echo .)org \
--cc=ira.weiny@intel$(echo .)com \
--cc=jannh@google$(echo .)com \
--cc=jeffxu@chromium$(echo .)org \
--cc=joey.gouly@arm$(echo .)com \
--cc=kees@kernel$(echo .)org \
--cc=linusw@kernel$(echo .)org \
--cc=linux-arm-kernel@lists$(echo .)infradead.org \
--cc=linux-hardening@vger$(echo .)kernel.org \
--cc=linux-mm@kvack$(echo .)org \
--cc=ljs@kernel$(echo .)org \
--cc=luto@kernel$(echo .)org \
--cc=maz@kernel$(echo .)org \
--cc=mbland@motorola$(echo .)com \
--cc=peterz@infradead$(echo .)org \
--cc=pierre.langlois@arm$(echo .)com \
--cc=qperret@google$(echo .)com \
--cc=rick.p.edgecombe@intel$(echo .)com \
--cc=rppt@kernel$(echo .)org \
--cc=ryan.roberts@arm$(echo .)com \
--cc=tglx@kernel$(echo .)org \
--cc=vbabka@kernel$(echo .)org \
--cc=will@kernel$(echo .)org \
--cc=willy@infradead$(echo .)org \
--cc=x86@kernel$(echo .)org \
--cc=yang@os$(echo .)amperecomputing.com \
--cc=yeoreum.yun@arm$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox