Subject: Re: [PATCH] net :mana : Add per-cpu stats for MANA device

From: Rahul Rameshbabu <[email-masked]>
Date: Fri, 08 Mar 2024 11:52:42 -0800

On Fri, 08 Mar, 2024 19:43:57 +0000 Haiyang Zhang <[email-masked]> wrote:
>> -----Original Message-----
>> From: Jakub Kicinski <[email-masked]>
>> Sent: Friday, March 8, 2024 2:23 PM
>> To: Haiyang Zhang <[email-masked]>
>> Cc: Shradha Gupta <[email-masked]>; Shradha Gupta
>> <[email-masked]>; [email-masked]; linux-
>> [email-masked]; [email-masked];
>> [email-masked]; Eric Dumazet <[email-masked]>; Paolo Abeni
>> <[email-masked]>; Ajay Sharma <[email-masked]>; Leon
>> Romanovsky <[email-masked]>; Thomas Gleixner <[email-masked]>;
>> Sebastian Andrzej Siewior <[email-masked]>; KY Srinivasan
>> <[email-masked]>; Wei Liu <[email-masked]>; Dexuan Cui
>> <[email-masked]>; Long Li <[email-masked]>; Michael Kelley
>> <[email-masked]>
>> Subject: Re: [PATCH] net :mana : Add per-cpu stats for MANA device
>> 
>> On Fri, 8 Mar 2024 18:51:58 +0000 Haiyang Zhang wrote:
>> > > Dynamic is a bit of an exaggeration, right? On a well-configured
>> system
>> > > each CPU should use a single queue assigned thru XPS. And for manual
>> > > debug bpftrace should serve the purpose quite well.
>> >
>> > Some programs, like irqbalancer can dynamically change the CPU
>> affinity,
>> > so we want to add the per-CPU counters for better understanding of the
>> CPU
>> > usage.
>> 
>> Do you have experimental data showing this making a difference
>> in production?
> Shradha, could you please add some data before / after enabling irqbalancer 
> which changes cpu affinity?
>
>> 
>> Seems unlikely, but if it does work we should enable it for all
>> devices, no driver by driver.
> There are some existing drivers, like mlx, rmnet, netvsc, etc. using percpu 
> counters. Are you suggesting we add a common API for all drivers?

Wanted to chime in with regards to mlx. You might be conflating per-cpu
with per-queue. When we run ethtool -S, we present counters per netdev
queue rather than per-cpu. The number of queues we instantiate is
related to CPUs but it not always 1-1.

Jakub just recently supported a proper interface for per-queue stats
counters that we are interested in supporting.

  https://lore.kernel.org/netdev/[email-masked]/

--
Thanks,

Rahul Rameshbabu