From: Pranjal Shrivastava <praan@google•com>
To: iommu@lists•linux.dev
Cc: Will Deacon <will@kernel•org>, Joerg Roedel <joro@8bytes•org>,
Robin Murphy <robin.murphy@arm•com>,
Jason Gunthorpe <jgg@ziepe•ca>,
Mostafa Saleh <smostafa@google•com>,
Nicolin Chen <nicolinc@nvidia•com>,
Daniel Mentz <danielmentz@google•com>,
Ashish Mhetre <amhetre@nvidia•com>,
linux-arm-kernel@lists•infradead.org,
Pranjal Shrivastava <praan@google•com>
Subject: [PATCH v7 08/11] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops
Date: Wed, 27 May 2026 22:14:04 +0000 [thread overview]
Message-ID: <20260527221407.1756491-9-praan@google.com> (raw)
In-Reply-To: <20260527221407.1756491-1-praan@google.com>
Implement pm_runtime and system sleep ops for arm-smmu-v3.
The suspend callback configures the SMMU to abort new transactions,
disables the main translation unit and then drains the command queue
to ensure completion of any in-flight commands. A software gate
(STOP_FLAG) and synchronization barriers are used to quiesce the command
submission pipeline and ensure state consistency before power-off.
To prevent software metadata flags from leaking into physical registers
or polluting the tracking pointer, a newly introduced bitmask
(CMDQ_PROD_IDX_MASK) is applied to all fast-path register writes and
tracking updates, avoiding post-resume deadlocks.
The resume callback restores the MSI configuration and performs a full
device reset via `arm_smmu_device_reset` to bring the SMMU back to an
operational state. The MSIs are cached during the msi_write and are
restored during the resume operation by using the helper. The STOP_FLAG
is cleared only after the CMDQ is enabled in hardware.
Suggested-by: Daniel Mentz <danielmentz@google•com>
Signed-off-by: Pranjal Shrivastava <praan@google•com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 165 +++++++++++++++++++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 18 +++
2 files changed, 181 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a28417108a8a..6cbb1724b377 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -28,6 +28,7 @@
#include <linux/platform_device.h>
#include <linux/sort.h>
#include <linux/string_choices.h>
+#include <linux/pm_runtime.h>
#include <kunit/visibility.h>
#include <uapi/linux/iommufd.h>
@@ -110,6 +111,38 @@ static const char * const event_class_str[] = {
static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
static bool arm_smmu_ats_supported(struct arm_smmu_master *master);
+/* Runtime PM helpers */
+__maybe_unused static int arm_smmu_rpm_get(struct arm_smmu_device *smmu)
+{
+ int ret;
+
+ if (pm_runtime_enabled(smmu->dev)) {
+ ret = pm_runtime_resume_and_get(smmu->dev);
+ if (ret < 0) {
+ dev_err(smmu->dev, "failed to resume device: %d\n", ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+__maybe_unused static void arm_smmu_rpm_put(struct arm_smmu_device *smmu)
+{
+ int ret;
+
+ if (pm_runtime_enabled(smmu->dev)) {
+ ret = pm_runtime_put_autosuspend(smmu->dev);
+ if (ret < 0)
+ dev_err(smmu->dev, "failed to suspend device: %d\n", ret);
+ }
+}
+
+static inline u32 arm_smmu_cmdq_owner_prod_idx(struct arm_smmu_cmdq *cmdq)
+{
+ return atomic_read(&cmdq->owner_prod) & CMDQ_PROD_IDX_MASK;
+}
+
static void parse_driver_options(struct arm_smmu_device *smmu)
{
int i = 0;
@@ -789,7 +822,8 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
/* b. Stop gathering work by clearing the owned flag */
prod = atomic_fetch_andnot_relaxed(CMDQ_PROD_OWNED_FLAG,
&cmdq->q.llq.atomic.prod);
- prod &= ~CMDQ_PROD_OWNED_FLAG;
+ /* Strip all metadata flags */
+ prod &= CMDQ_PROD_IDX_MASK;
/*
* c. Wait for any gathered work to be written to the queue.
@@ -4827,7 +4861,8 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu)
/* Command queue */
writeq_relaxed(smmu->cmdq.q.q_base, smmu->base + ARM_SMMU_CMDQ_BASE);
- writel_relaxed(smmu->cmdq.q.llq.prod, smmu->base + ARM_SMMU_CMDQ_PROD);
+ writel_relaxed(smmu->cmdq.q.llq.prod & CMDQ_PROD_IDX_MASK,
+ smmu->base + ARM_SMMU_CMDQ_PROD);
writel_relaxed(smmu->cmdq.q.llq.cons, smmu->base + ARM_SMMU_CMDQ_CONS);
enables = CR0_CMDQEN;
@@ -4838,6 +4873,10 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu)
return ret;
}
+ /* Clear any flags from the previous life */
+ atomic_andnot(CMDQ_PROD_STOP_FLAG, &smmu->cmdq.owner_prod);
+ atomic_andnot(CMDQ_PROD_STOP_FLAG, &smmu->cmdq.q.llq.atomic.prod);
+
/* Invalidate any cached configuration */
arm_smmu_cmdq_issue_cmd_with_sync(smmu, arm_smmu_make_cmd_cfgi_all());
@@ -4897,6 +4936,21 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu)
if (is_kdump_kernel())
enables &= ~(CR0_EVTQEN | CR0_PRIQEN);
+ /*
+ * While the SMMU was suspended, concurrent CPU threads may have
+ * updated in-memory structures (such as STEs, CDs, and PTEs).
+ * Any invalidations corresponding to those updates were safely
+ * elided because the command queue was stopped (STOP_FLAG == 1).
+ *
+ * Since the reset invalidate-all commands above have fully cleared
+ * the HW TLBs and config caches, the SMMU will fetch these descriptors
+ * directly from RAM as soon as translation is enabled.
+ *
+ * Add a memory barrier to collect all prior RAM writes to ensure the
+ * SMMU sees a consistent view of memory before translation is enabled.
+ */
+ smp_mb();
+
/* Enable the SMMU interface */
enables |= CR0_SMMUEN;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
@@ -5579,6 +5633,112 @@ static void arm_smmu_device_shutdown(struct platform_device *pdev)
arm_smmu_device_disable(smmu);
}
+static int __maybe_unused arm_smmu_runtime_suspend(struct device *dev)
+{
+ struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+ struct arm_smmu_cmdq *cmdq = &smmu->cmdq;
+ int timeout = ARM_SMMU_SUSPEND_TIMEOUT_US;
+ u32 enables, target;
+ int ret;
+
+ /* Abort all transactions before disable to avoid spurious bypass */
+ arm_smmu_update_gbpa(smmu, GBPA_ABORT, 0);
+
+ /* Disable the SMMU via CR0.EN and all queues except CMDQ */
+ enables = CR0_CMDQEN;
+ ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0, ARM_SMMU_CR0ACK);
+ if (ret) {
+ dev_err(smmu->dev, "failed to disable SMMU\n");
+ return ret;
+ }
+
+ /*
+ * At this point the SMMU is completely disabled and won't access
+ * any translation/config structures, even speculative accesses
+ * aren't performed as per the IHI0070 spec (section 6.3.9.6).
+ */
+
+ /* Mark the CMDQ to stop and get the target index before the stop */
+ target = atomic_fetch_or_relaxed(CMDQ_PROD_STOP_FLAG, &cmdq->q.llq.atomic.prod);
+ target &= CMDQ_PROD_IDX_MASK;
+
+
+ /* Wait for the last committed owner to reach the hardware */
+ while ((arm_smmu_cmdq_owner_prod_idx(cmdq) != target) && --timeout)
+ udelay(1);
+
+ /*
+ * Entering suspend implies no active clients. A timeout here
+ * indicates a fatal CMDQ lockup or hardware stall. We proceed
+ * anyway to prioritize memory safety (avoiding stale TLBs)
+ */
+ if (!timeout)
+ dev_err(smmu->dev, "cmdq owner wait timeout, (check runtime PM + devlinks)\n");
+
+ /* Drain the CMDQs */
+ ret = arm_smmu_drain_queues(smmu);
+ if (ret)
+ dev_warn(smmu->dev, "failed to drain queues, forcing suspend\n");
+
+ /* Wait for cmdq->lock == 0 to ensure last CMDQ_CONS_REG is written */
+ timeout = ARM_SMMU_SUSPEND_TIMEOUT_US;
+ while (atomic_read(&cmdq->lock) != 0 && --timeout)
+ udelay(1);
+
+ /* Timing out here implies misconfigured Runtime PM or broken devlinks */
+ if (!timeout)
+ dev_err(smmu->dev, "cmdq lock != 0, forcing suspend. Polling CPUs may fault.\n");
+
+ /* Disable everything */
+ arm_smmu_device_disable(smmu);
+
+ /* Handle any pending gerrors before powering down */
+ arm_smmu_handle_gerror(smmu);
+
+ dev_dbg(dev, "suspended smmu\n");
+
+ return 0;
+}
+
+static int __maybe_unused arm_smmu_runtime_resume(struct device *dev)
+{
+ struct arm_smmu_device *smmu = dev_get_drvdata(dev);
+ int ret;
+
+ /* Re-configure MSIs */
+ arm_smmu_resume_msis(smmu);
+
+ ret = arm_smmu_device_reset(smmu);
+ if (ret)
+ dev_err(dev, "failed to reset during resume operation: %d\n", ret);
+
+ dev_dbg(dev, "resumed smmu\n");
+
+ return ret;
+}
+
+static int __maybe_unused arm_smmu_pm_suspend(struct device *dev)
+{
+ if (pm_runtime_suspended(dev))
+ return 0;
+
+ return arm_smmu_runtime_suspend(dev);
+}
+
+static int __maybe_unused arm_smmu_pm_resume(struct device *dev)
+{
+ if (pm_runtime_suspended(dev))
+ return 0;
+
+ return arm_smmu_runtime_resume(dev);
+}
+
+static const struct dev_pm_ops arm_smmu_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(arm_smmu_pm_suspend, arm_smmu_pm_resume)
+ SET_RUNTIME_PM_OPS(arm_smmu_runtime_suspend,
+ arm_smmu_runtime_resume, NULL)
+};
+
static const struct of_device_id arm_smmu_of_match[] = {
{ .compatible = "arm,smmu-v3", },
{ },
@@ -5595,6 +5755,7 @@ static struct platform_driver arm_smmu_driver = {
.driver = {
.name = "arm-smmu-v3",
.of_match_table = arm_smmu_of_match,
+ .pm = &arm_smmu_pm_ops,
.suppress_bind_attrs = true,
},
.probe = arm_smmu_device_probe,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index a3c8417c87d8..97ec9d9caffe 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -392,6 +392,9 @@ static inline unsigned int arm_smmu_cdtab_l2_idx(unsigned int ssid)
#define CMDQ_PROD_STOP_FLAG (1U << 30)
#define Q_STOP(p) ((p) & CMDQ_PROD_STOP_FLAG)
+/* Mask out software-only metadata flags to get the pure queue index/wrap bits */
+#define CMDQ_PROD_IDX_MASK ~(CMDQ_PROD_STOP_FLAG | CMDQ_PROD_OWNED_FLAG)
+
struct arm_smmu_cmd {
u64 data[CMDQ_ENT_DWORDS];
};
@@ -655,11 +658,14 @@ arm_smmu_make_cmd_tlbi(enum arm_smmu_cmdq_opcode op, u16 asid, u16 vmid)
/* High-level queue structures */
#define ARM_SMMU_POLL_TIMEOUT_US 1000000 /* 1s! */
+#define ARM_SMMU_SUSPEND_TIMEOUT_US 1000000 /* 1s! */
#define ARM_SMMU_POLL_SPIN_COUNT 10
#define MSI_IOVA_BASE 0x8000000
#define MSI_IOVA_LENGTH 0x100000
+#define RPM_AUTOSUSPEND_DELAY_MS 15
+
struct arm_smmu_ll_queue {
union {
u64 val;
@@ -1217,6 +1223,18 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
struct arm_smmu_cmd *cmds, int n,
bool sync);
+/*
+ * Lockless pre-check to elide invalidations if SMMU is suspended.
+ * Races with concurrent suspend are benign: the cmpxchg loop in
+ * arm_smmu_cmdq_issue_cmdlist() acts as the true commit point.
+ * If we lose the race, that loop observes Q_STOP == 1 and safely
+ * drops the command. If we win, the suspend thread waits for us.
+ */
+static inline bool arm_smmu_can_elide(struct arm_smmu_device *smmu)
+{
+ return !!Q_STOP(READ_ONCE(smmu->cmdq.q.llq.prod));
+}
+
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
void arm_smmu_sva_notifier_synchronize(void);
--
2.54.0.794.g4f17f83d09-goog
next prev parent reply other threads:[~2026-05-27 22:15 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-27 22:13 [PATCH v7 00/11] iommu/arm-smmu-v3: Implement Runtime/System Sleep ops Pranjal Shrivastava
2026-05-27 22:13 ` [PATCH v7 01/11] iommu/arm-smmu-v3: Refactor arm_smmu_setup_irqs Pranjal Shrivastava
2026-05-27 22:13 ` [PATCH v7 02/11] iommu/arm-smmu-v3: Add a helper to drain cmd queues Pranjal Shrivastava
2026-05-28 1:35 ` Nicolin Chen
2026-05-28 10:34 ` Pranjal Shrivastava
2026-05-28 22:09 ` Nicolin Chen
2026-05-29 14:32 ` Pranjal Shrivastava
2026-05-27 22:13 ` [PATCH v7 03/11] iommu/tegra241-cmdqv: Add a helper to drain VCMDQs Pranjal Shrivastava
2026-05-27 22:14 ` [PATCH v7 04/11] iommu/tegra241-cmdqv: Restore PROD and CONS after resume Pranjal Shrivastava
2026-05-28 18:14 ` Nicolin Chen
2026-05-27 22:14 ` [PATCH v7 05/11] iommu/arm-smmu-v3: Cache and restore MSI config Pranjal Shrivastava
2026-05-28 18:36 ` Nicolin Chen
2026-05-28 21:57 ` Pranjal Shrivastava
2026-05-28 22:03 ` Nicolin Chen
2026-05-27 22:14 ` [PATCH v7 06/11] iommu/arm-smmu-v3: Handle gerror during suspend Pranjal Shrivastava
2026-05-28 18:53 ` Nicolin Chen
2026-05-28 21:59 ` Pranjal Shrivastava
2026-05-27 22:14 ` [PATCH v7 07/11] iommu/arm-smmu-v3: Add CMDQ_PROD_STOP_FLAG to gate CMDQ submissions Pranjal Shrivastava
2026-05-28 19:41 ` Nicolin Chen
2026-05-28 21:57 ` Pranjal Shrivastava
2026-05-27 22:14 ` Pranjal Shrivastava [this message]
2026-05-28 19:39 ` [PATCH v7 08/11] iommu/arm-smmu-v3: Implement pm_runtime & system sleep ops Nicolin Chen
2026-05-28 21:21 ` Pranjal Shrivastava
2026-05-28 22:13 ` Nicolin Chen
2026-05-28 23:30 ` Pranjal Shrivastava
2026-05-27 22:14 ` [PATCH v7 09/11] iommu/arm-smmu-v3: Enable pm_runtime and setup devlinks Pranjal Shrivastava
2026-05-28 20:13 ` Nicolin Chen
2026-05-28 21:36 ` Pranjal Shrivastava
2026-05-27 22:14 ` [PATCH v7 10/11] iommu/arm-smmu-v3: Invoke pm_runtime before hw access Pranjal Shrivastava
2026-05-28 20:28 ` Nicolin Chen
2026-05-28 21:46 ` Pranjal Shrivastava
2026-05-28 22:01 ` Nicolin Chen
2026-05-28 22:25 ` Pranjal Shrivastava
2026-05-28 23:18 ` Nicolin Chen
2026-05-29 14:48 ` Pranjal Shrivastava
2026-06-03 22:18 ` Daniel Mentz
2026-06-04 7:18 ` Pranjal Shrivastava
2026-05-27 22:14 ` [PATCH v7 11/11] iommu/arm-smmu-v3: Add KUnit unit tests for Runtime PM Pranjal Shrivastava
2026-05-28 21:43 ` Nicolin Chen
2026-05-28 23:10 ` Pranjal Shrivastava
2026-05-28 23:21 ` Nicolin Chen
2026-05-28 23:33 ` Pranjal Shrivastava
2026-05-28 18:05 ` [PATCH v7 00/11] iommu/arm-smmu-v3: Implement Runtime/System Sleep ops Nicolin Chen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260527221407.1756491-9-praan@google.com \
--to=praan@google$(echo .)com \
--cc=amhetre@nvidia$(echo .)com \
--cc=danielmentz@google$(echo .)com \
--cc=iommu@lists$(echo .)linux.dev \
--cc=jgg@ziepe$(echo .)ca \
--cc=joro@8bytes$(echo .)org \
--cc=linux-arm-kernel@lists$(echo .)infradead.org \
--cc=nicolinc@nvidia$(echo .)com \
--cc=robin.murphy@arm$(echo .)com \
--cc=smostafa@google$(echo .)com \
--cc=will@kernel$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox