* RFH, is it possible to set ndo_start_xmit() cpu affinity in ethernet driver?
@ 2022-09-12 15:43 Василий Умрихин
2022-09-19 1:59 ` Sergey Ryazanov
0 siblings, 1 reply; 3+ messages in thread
From: Василий Умрихин @ 2022-09-12 15:43 UTC (permalink / raw)
To: netdev
Hi netdev,
On the receiving side we have the opportunity to choose the CPU that will process the receive queue (RPS).
On the sender side XPS selects the send queue for the given CPU, but there is no way to select the CPU on which ndo_start_xmit() will be launched.
Taskset is able to bind user task, but in ndo_start_xmit() binding differs.
In my case CPU0 reserved for polling kthread, because our NIC have no interrupts, therefore it is necessary. I need nothing else to run on this CPU.
For example, setting CPU1 for RPS on both nodes:
host1: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
host2: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
Then run iperf on two nodes:
host1: taskset -c 1 iperf -s
host2: taskset -c 1 iperf -c host1
After adding pr_info("cpu%d\n", smp_processor_id()); in my ndo_start_xmit() method, see in dmesg:
host1: dmesg | grep cpu0 | wc -l
0
host2: dmesg | grep cpu0 | wc -l
6512
Is it possible to choose the CPU on which ndo_start_xmit() will be launched on the sender side?
Kind regards, Vasiliy
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: RFH, is it possible to set ndo_start_xmit() cpu affinity in ethernet driver?
2022-09-12 15:43 RFH, is it possible to set ndo_start_xmit() cpu affinity in ethernet driver? Василий Умрихин
@ 2022-09-19 1:59 ` Sergey Ryazanov
0 siblings, 0 replies; 3+ messages in thread
From: Sergey Ryazanov @ 2022-09-19 1:59 UTC (permalink / raw)
To: Василий Умрихин
Cc: netdev
Hello Vasiliy,
On Mon, Sep 12, 2022 at 6:45 PM Василий Умрихин <umrihin@nicevt•ru> wrote:
> On the receiving side we have the opportunity to choose the CPU that will process the receive queue (RPS).
> On the sender side XPS selects the send queue for the given CPU, but there is no way to select the CPU on which ndo_start_xmit() will be launched.
> Taskset is able to bind user task, but in ndo_start_xmit() binding differs.
> In my case CPU0 reserved for polling kthread, because our NIC have no interrupts, therefore it is necessary. I need nothing else to run on this CPU.
>
> For example, setting CPU1 for RPS on both nodes:
>
> host1: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
> host2: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
>
> Then run iperf on two nodes:
>
> host1: taskset -c 1 iperf -s
> host2: taskset -c 1 iperf -c host1
>
> After adding pr_info("cpu%d\n", smp_processor_id()); in my ndo_start_xmit() method, see in dmesg:
>
> host1: dmesg | grep cpu0 | wc -l
> 0
> host2: dmesg | grep cpu0 | wc -l
> 6512
>
> Is it possible to choose the CPU on which ndo_start_xmit() will be launched on the sender side?
AFAIK, there is no mechanism to force the ndo_start_xmit() invocation
on a specific core.
If I realize your goal correctly, then you can implement such
functionality in the driver. Just use an intermediate queue that is
filled by the ndo_start_xmit() callback and processed by a thread
bound to a specific core. E.g.
netdev_tx_t foodriver_start_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct foodev_priv *p = netdev_priv(dev);
skb_queue_tail(&p->xmit_queue, skb);
return NETDEV_TX_OK;
}
int foodriver_xmit_thread(void *arg)
{
struct foodev_priv *p = arg;
while (!kthread_should_stop()) {
struct sk_buff *skb = skb_dequeue(&p->xmit_queue);
/* Do main xmit actions here */
}
}
int foodriver_open(struct net_device *dev)
{
struct foodev_priv *p = netdev_priv(dev);
...
t = kthread_create(foodriver_xmit_thread, p, "xmit_thread");
kthread_bind(t, prefered_cpu);
....
}
--
Sergey
^ permalink raw reply [flat|nested] 3+ messages in thread
* RFH, is it possible to set ndo_start_xmit() cpu affinity in ethernet driver?
@ 2022-09-12 15:48 Василий Умрихин
0 siblings, 0 replies; 3+ messages in thread
From: Василий Умрихин @ 2022-09-12 15:48 UTC (permalink / raw)
To: netdev
Hi netdev,
On the receiving side we have the opportunity to choose the CPU that will process the receive queue (RPS).
On the sender side XPS selects the send queue for the given CPU, but there is no way to select the CPU on which ndo_start_xmit() will be launched.
Taskset is able to bind user task, but in ndo_start_xmit() binding differs.
In my case CPU0 reserved for polling kthread, because our NIC have no interrupts, therefore it is necessary. I need nothing else to run on this CPU.
For example, setting CPU1 for RPS on both nodes:
host1: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
host2: echo 0x2 > /sys/class/net//queues/rx-0/rps_cpus
Then run iperf on two nodes:
host1: taskset -c 1 iperf -s
host2: taskset -c 1 iperf -c host1
After adding pr_info("cpu%d\n", smp_processor_id()); in my ndo_start_xmit() method, see in dmesg:
host1: dmesg | grep cpu0 | wc -l
0
host2: dmesg | grep cpu0 | wc -l
6512
Is it possible to choose the CPU on which ndo_start_xmit() will be launched on the sender side?
Kind regards, Vasiliy
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-09-19 1:59 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-09-12 15:43 RFH, is it possible to set ndo_start_xmit() cpu affinity in ethernet driver? Василий Умрихин
2022-09-19 1:59 ` Sergey Ryazanov
-- strict thread matches above, loose matches on Subject: below --
2022-09-12 15:48 Василий Умрихин
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox