From: Saeed Mahameed <saeed@kernel•org>
To: "David S. Miller" <davem@davemloft•net>,
Jakub Kicinski <kuba@kernel•org>, Paolo Abeni <pabeni@redhat•com>,
Eric Dumazet <edumazet@google•com>
Cc: Saeed Mahameed <saeedm@nvidia•com>,
netdev@vger•kernel.org, Tariq Toukan <tariqt@nvidia•com>,
Gal Pressman <gal@nvidia•com>,
Leon Romanovsky <leonro@nvidia•com>, Jiri Pirko <jiri@nvidia•com>,
Vlad Dumitrescu <vdumitrescu@nvidia•com>
Subject: [PATCH net-next 05/14] net/mlx5: Implement devlink total_vfs parameter
Date: Thu, 27 Feb 2025 18:12:18 -0800 [thread overview]
Message-ID: <20250228021227.871993-6-saeed@kernel.org> (raw)
In-Reply-To: <20250228021227.871993-1-saeed@kernel.org>
From: Vlad Dumitrescu <vdumitrescu@nvidia•com>
Some devices support both symmetric (same value for all PFs) and
asymmetric, while others only support symmetric configuration. This
implementation prefers asymmetric, since it is closer to the devlink
model (per function settings), but falls back to symmetric when needed.
Example usage:
devlink dev param set pci/0000:01:00.0 name total_vfs value <u16> cmode permanent
devlink dev reload pci/0000:01:00.0 action fw_activate
echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
echo 1 >/sys/bus/pci/rescan
cat /sys/bus/pci/devices/0000:01:00.0/sriov_totalvfs
Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia•com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia•com>
Reviewed-by: Jiri Pirko <jiri@nvidia•com>
---
Documentation/networking/devlink/mlx5.rst | 22 +++
.../mellanox/mlx5/core/lib/nv_param.c | 125 ++++++++++++++++++
2 files changed, 147 insertions(+)
diff --git a/Documentation/networking/devlink/mlx5.rst b/Documentation/networking/devlink/mlx5.rst
index 587e0200c1cd..00a43324dec2 100644
--- a/Documentation/networking/devlink/mlx5.rst
+++ b/Documentation/networking/devlink/mlx5.rst
@@ -40,6 +40,28 @@ Parameters
- Boolean
- Applies to each physical function (PF) independently, if the device
supports it. Otherwise, it applies symmetrically to all PFs.
+ * - ``total_vfs``
+ - permanent
+ - The range is between 1 and a device-specific max.
+ - Applies to each physical function (PF) independently, if the device
+ supports it. Otherwise, it applies symmetrically to all PFs.
+
+Note: permanent parameters such as ``enable_sriov`` and ``total_vfs`` require FW reset to take effect
+
+.. code-block:: bash
+
+ # setup parameters
+ devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent
+ devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent
+
+ # Fw reset
+ devlink dev reload pci/0000:01:00.0 action fw_activate
+
+ # for PCI related config such as sriov PCI reset/rescan is required:
+ echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
+ echo 1 >/sys/bus/pci/rescan
+ grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
+
The ``mlx5`` driver also implements the following driver-specific
parameters.
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
index 6b63fc110e2d..97d74d890582 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/nv_param.c
@@ -387,10 +387,135 @@ static int mlx5_devlink_enable_sriov_set(struct devlink *devlink, u32 id,
return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
}
+static int mlx5_devlink_total_vfs_get(struct devlink *devlink, u32 id,
+ struct devlink_param_gset_ctx *ctx)
+{
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)] = {};
+ void *data;
+ int err;
+
+ data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+
+ err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
+ if (err)
+ return err;
+
+ if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
+ ctx->val.vu32 = 0;
+ return 0;
+ }
+
+ memset(mnvda, 0, sizeof(mnvda));
+ err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
+ if (err)
+ return err;
+
+ if (!MLX5_GET(nv_global_pci_conf, data, per_pf_total_vf)) {
+ ctx->val.vu32 = MLX5_GET(nv_global_pci_conf, data, total_vfs);
+ return 0;
+ }
+
+ /* SRIOV is per PF */
+ memset(mnvda, 0, sizeof(mnvda));
+ err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
+ if (err)
+ return err;
+
+ ctx->val.vu32 = MLX5_GET(nv_pf_pci_conf, data, total_vf);
+
+ return 0;
+}
+
+static int mlx5_devlink_total_vfs_set(struct devlink *devlink, u32 id,
+ struct devlink_param_gset_ctx *ctx,
+ struct netlink_ext_ack *extack)
+{
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ u32 mnvda[MLX5_ST_SZ_DW(mnvda_reg)];
+ bool per_pf_support;
+ void *data;
+ int err;
+
+ err = mlx5_nv_param_read_global_pci_cap(dev, mnvda, sizeof(mnvda));
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Failed to read global pci cap");
+ return err;
+ }
+
+ data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+ if (!MLX5_GET(nv_global_pci_cap, data, sriov_support)) {
+ NL_SET_ERR_MSG_MOD(extack, "Not configurable on this device");
+ return -EOPNOTSUPP;
+ }
+
+ per_pf_support = MLX5_GET(nv_global_pci_cap, data, per_pf_total_vf_supported);
+ memset(mnvda, 0, sizeof(mnvda));
+
+ err = mlx5_nv_param_read_global_pci_conf(dev, mnvda, sizeof(mnvda));
+ if (err)
+ return err;
+
+ MLX5_SET(nv_global_pci_conf, data, sriov_valid, 1);
+ MLX5_SET(nv_global_pci_conf, data, per_pf_total_vf, per_pf_support);
+
+ if (!per_pf_support) {
+ MLX5_SET(nv_global_pci_conf, data, total_vfs, ctx->val.vu32);
+ return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+ }
+
+ /* SRIOV is per PF */
+ err = mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+ if (err)
+ return err;
+
+ memset(mnvda, 0, sizeof(mnvda));
+ err = mlx5_nv_param_read_per_host_pf_conf(dev, mnvda, sizeof(mnvda));
+ if (err)
+ return err;
+
+ data = MLX5_ADDR_OF(mnvda_reg, mnvda, configuration_item_data);
+ MLX5_SET(nv_pf_pci_conf, data, pf_total_vf_en, 1);
+ MLX5_SET(nv_pf_pci_conf, data, total_vf, ctx->val.vu32);
+ return mlx5_nv_param_write(dev, mnvda, sizeof(mnvda));
+}
+
+static int mlx5_devlink_total_vfs_validate(struct devlink *devlink, u32 id,
+ union devlink_param_value val,
+ struct netlink_ext_ack *extack)
+{
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ u32 cap[MLX5_ST_SZ_DW(mnvda_reg)];
+ void *data;
+ u16 max;
+ int err;
+
+ data = MLX5_ADDR_OF(mnvda_reg, cap, configuration_item_data);
+
+ err = mlx5_nv_param_read_global_pci_cap(dev, cap, sizeof(cap));
+ if (err)
+ return err;
+
+ if (!MLX5_GET(nv_global_pci_cap, data, max_vfs_per_pf_valid))
+ return 0; /* optimistic, but set might fail later */
+
+ max = MLX5_GET(nv_global_pci_cap, data, max_vfs_per_pf);
+ if (val.vu16 > max) {
+ NL_SET_ERR_MSG_FMT_MOD(extack,
+ "Max allowed by device is %u", max);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static const struct devlink_param mlx5_nv_param_devlink_params[] = {
DEVLINK_PARAM_GENERIC(ENABLE_SRIOV, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
mlx5_devlink_enable_sriov_get,
mlx5_devlink_enable_sriov_set, NULL),
+ DEVLINK_PARAM_GENERIC(TOTAL_VFS, BIT(DEVLINK_PARAM_CMODE_PERMANENT),
+ mlx5_devlink_total_vfs_get, mlx5_devlink_total_vfs_set,
+ mlx5_devlink_total_vfs_validate),
DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_CQE_COMPRESSION_TYPE,
"cqe_compress_type", DEVLINK_PARAM_TYPE_STRING,
BIT(DEVLINK_PARAM_CMODE_PERMANENT),
--
2.48.1
next prev parent reply other threads:[~2025-02-28 2:13 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-02-28 2:12 [PATCH net-next 00/14] devlink, mlx5: Add new parameters for link management and SRIOV/eSwitch configurations Saeed Mahameed
2025-02-28 2:12 ` [PATCH net-next 01/14] devlink: define enum for attr types of dynamic attributes Saeed Mahameed
2025-03-06 12:05 ` Simon Horman
2025-03-19 22:45 ` Saeed Mahameed
2025-02-28 2:12 ` [PATCH net-next 02/14] devlink: Add 'total_vfs' generic device param Saeed Mahameed
2025-02-28 12:39 ` Jiri Pirko
2025-03-04 16:42 ` Kamal Heib
2025-02-28 2:12 ` [PATCH net-next 03/14] net/mlx5: Implement cqe_compress_type via devlink params Saeed Mahameed
2025-02-28 2:12 ` [PATCH net-next 04/14] net/mlx5: Implement devlink enable_sriov parameter Saeed Mahameed
2025-02-28 12:46 ` Jiri Pirko
2025-02-28 18:19 ` Saeed Mahameed
2025-03-03 11:35 ` Jiri Pirko
2025-03-03 2:27 ` kernel test robot
2025-03-04 16:43 ` Kamal Heib
2025-02-28 2:12 ` Saeed Mahameed [this message]
2025-03-04 16:45 ` [PATCH net-next 05/14] net/mlx5: Implement devlink total_vfs parameter Kamal Heib
2025-02-28 2:12 ` [PATCH net-next 06/14] devlink: pass struct devlink_port * as arg to devlink_nl_param_fill() Saeed Mahameed
2025-02-28 2:12 ` [PATCH net-next 07/14] devlink: Implement port params registration Saeed Mahameed
2025-02-28 11:58 ` Przemek Kitszel
2025-02-28 12:28 ` Jiri Pirko
2025-02-28 13:23 ` Przemek Kitszel
2025-02-28 15:21 ` Jiri Pirko
2025-03-20 8:16 ` Przemek Kitszel
2025-02-28 2:12 ` [PATCH net-next 08/14] devlink: Implement get/dump netlink commands for port params Saeed Mahameed
2025-02-28 2:12 ` [PATCH net-next 09/14] devlink: Implement set netlink command " Saeed Mahameed
2025-02-28 12:49 ` Jiri Pirko
2025-02-28 2:12 ` [PATCH net-next 10/14] devlink: Add 'keep_link_up' generic devlink device param Saeed Mahameed
2025-02-28 12:51 ` Jiri Pirko
2025-02-28 2:12 ` [PATCH net-next 11/14] net/mlx5: Implement devlink keep_link_up port parameter Saeed Mahameed
2025-02-28 12:51 ` Jiri Pirko
2025-02-28 2:12 ` [PATCH net-next 12/14] devlink: Throw extack messages on param value validation error Saeed Mahameed
2025-02-28 12:53 ` Jiri Pirko
2025-03-03 7:06 ` Dan Carpenter
2025-02-28 2:12 ` [PATCH net-next 13/14] devlink: Implement devlink param multi attribute nested data values Saeed Mahameed
2025-02-28 2:12 ` [PATCH net-next 14/14] net/mlx5: Implement eSwitch hairpin per prio buffers devlink params Saeed Mahameed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250228021227.871993-6-saeed@kernel.org \
--to=saeed@kernel$(echo .)org \
--cc=davem@davemloft$(echo .)net \
--cc=edumazet@google$(echo .)com \
--cc=gal@nvidia$(echo .)com \
--cc=jiri@nvidia$(echo .)com \
--cc=kuba@kernel$(echo .)org \
--cc=leonro@nvidia$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=pabeni@redhat$(echo .)com \
--cc=saeedm@nvidia$(echo .)com \
--cc=tariqt@nvidia$(echo .)com \
--cc=vdumitrescu@nvidia$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox