From: Ashish Mhetre <amhetre@nvidia•com>
To: <will@kernel•org>, <robin.murphy@arm•com>, <joro@8bytes•org>,
<jgg@ziepe•ca>, <nicolinc@nvidia•com>
Cc: <linux-arm-kernel@lists•infradead.org>, <iommu@lists•linux.dev>,
<linux-kernel@vger•kernel.org>, <linux-tegra@vger•kernel.org>,
Ashish Mhetre <amhetre@nvidia•com>
Subject: [PATCH 2/2] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264
Date: Thu, 28 May 2026 10:16:17 +0000 [thread overview]
Message-ID: <20260528101617.4068249-3-amhetre@nvidia.com> (raw)
In-Reply-To: <20260528101617.4068249-1-amhetre@nvidia.com>
Apply the workaround for Tegra264 erratum by issuing every CFGI/TLBI
command twice on affected SMMU instances, with CMD_SYNC after each.
The erratum requires this exact sequencing:
TLBI/CFGI ... CMD_SYNC TLBI/CFGI ... CMD_SYNC
To get this sequence with minimal surgery, hook the workaround into
arm_smmu_cmdq_issue_cmdlist(). Rename the original function to
__arm_smmu_cmdq_issue_cmdlist() and add a thin wrapper that, on
affected SMMUs and when @sync is true, re-issues the same cmdlist a
second time.
A new arm_smmu_cmd_needs_tlbi_twice() helper classifies which opcodes
need the doubling: CFGI_* and TLBI_*.
For batches that exceed CMDQ_BATCH_ENTRIES commands,
arm_smmu_cmdq_batch_add_cmd_p() normally flushes the full buffer with
sync=false, deferring the SYNC to the eventual batch_submit(). On
affected SMMUs this would leave the first chunk's commands issued
only once, since the WAR hook in arm_smmu_cmdq_issue_cmdlist() only
fires on synced submissions. Force a SYNC on the capacity rollover
when the buffer carries CFGI/TLBI commands so every flushed chunk is
correctly doubled.
Signed-off-by: Ashish Mhetre <amhetre@nvidia•com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 66 +++++++++++++++++++--
1 file changed, 61 insertions(+), 5 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 88296c0a5337..38d45f175a2c 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -698,10 +698,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq,
* insert their own list of commands then all of the commands from one
* CPU will appear before any of the commands from the other CPU.
*/
-int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
- struct arm_smmu_cmdq *cmdq,
- struct arm_smmu_cmd *cmds, int n,
- bool sync)
+static int __arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+ struct arm_smmu_cmdq *cmdq,
+ struct arm_smmu_cmd *cmds, int n,
+ bool sync)
{
struct arm_smmu_cmd cmd_sync;
u32 prod;
@@ -820,6 +820,52 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
return ret;
}
+/*
+ * Returns true if @opcode is a CFGI_* or TLBI_* command, i.e. one of the
+ * invalidations covered by Tegra264 erratum (see ARM_SMMU_OPT_TLBI_TWICE).
+ */
+static bool arm_smmu_cmd_needs_tlbi_twice(u8 opcode)
+{
+ switch (opcode) {
+ case CMDQ_OP_CFGI_STE:
+ case CMDQ_OP_CFGI_ALL:
+ case CMDQ_OP_CFGI_CD:
+ case CMDQ_OP_CFGI_CD_ALL:
+ case CMDQ_OP_TLBI_NH_ALL:
+ case CMDQ_OP_TLBI_NH_ASID:
+ case CMDQ_OP_TLBI_NH_VA:
+ case CMDQ_OP_TLBI_NH_VAA:
+ case CMDQ_OP_TLBI_EL2_ALL:
+ case CMDQ_OP_TLBI_EL2_ASID:
+ case CMDQ_OP_TLBI_EL2_VA:
+ case CMDQ_OP_TLBI_S12_VMALL:
+ case CMDQ_OP_TLBI_S2_IPA:
+ case CMDQ_OP_TLBI_NSNH_ALL:
+ return true;
+ default:
+ return false;
+ }
+}
+
+int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
+ struct arm_smmu_cmdq *cmdq,
+ struct arm_smmu_cmd *cmds, int n,
+ bool sync)
+{
+ int ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+ /*
+ * The driver's batch invariants keep a single submission's
+ * opcode class uniform, so checking the first command is enough.
+ */
+ if (!ret && sync && (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+ arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+ cmds[0].data[0])))
+ ret = __arm_smmu_cmdq_issue_cmdlist(smmu, cmdq, cmds, n, sync);
+
+ return ret;
+}
+
static int arm_smmu_cmdq_issue_cmd_p(struct arm_smmu_device *smmu,
struct arm_smmu_cmd *cmd, bool sync)
{
@@ -863,8 +909,18 @@ static void arm_smmu_cmdq_batch_add_cmd_p(struct arm_smmu_device *smmu,
}
if (cmds->num == CMDQ_BATCH_ENTRIES) {
+ /*
+ * Force a SYNC only when the batch carries commands that
+ * have to be doubled (see ARM_SMMU_OPT_TLBI_TWICE).
+ * The batch holds a uniform opcode class, so checking
+ * the first command is sufficient.
+ */
+ bool need_sync = (smmu->options & ARM_SMMU_OPT_TLBI_TWICE) &&
+ arm_smmu_cmd_needs_tlbi_twice(FIELD_GET(CMDQ_0_OP,
+ cmds->cmds[0].data[0]));
+
arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmdq, cmds->cmds,
- cmds->num, false);
+ cmds->num, need_sync);
arm_smmu_cmdq_batch_init_cmd(smmu, cmds, cmd);
}
--
2.50.1
next prev parent reply other threads:[~2026-05-28 10:17 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-28 10:16 [PATCH 0/2] iommu/arm-smmu-v3: Tegra264 invalidation workaround Ashish Mhetre
2026-05-28 10:16 ` [PATCH 1/2] iommu/arm-smmu-v3: Detect Tegra264 erratum Ashish Mhetre
2026-05-28 10:34 ` Robin Murphy
2026-05-28 16:06 ` Ashish Mhetre
2026-05-28 18:38 ` Nicolin Chen
2026-05-29 6:11 ` Ashish Mhetre
2026-05-28 10:16 ` Ashish Mhetre [this message]
2026-05-29 0:59 ` [PATCH 2/2] iommu/arm-smmu-v3: Issue CFGI/TLBI twice on Tegra264 Jason Gunthorpe
2026-05-28 18:41 ` [PATCH 0/2] iommu/arm-smmu-v3: Tegra264 invalidation workaround Nicolin Chen
2026-05-29 6:18 ` Ashish Mhetre
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260528101617.4068249-3-amhetre@nvidia.com \
--to=amhetre@nvidia$(echo .)com \
--cc=iommu@lists$(echo .)linux.dev \
--cc=jgg@ziepe$(echo .)ca \
--cc=joro@8bytes$(echo .)org \
--cc=linux-arm-kernel@lists$(echo .)infradead.org \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linux-tegra@vger$(echo .)kernel.org \
--cc=nicolinc@nvidia$(echo .)com \
--cc=robin.murphy@arm$(echo .)com \
--cc=will@kernel$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox