From: Amir Vadai <amirv@mellanox•com>
To: Eric Dumazet <eric.dumazet@gmail•com>
Cc: "David S. Miller" <davem@davemloft•net>, <netdev@vger•kernel.org>,
"Yevgeny Petrilin" <yevgenyp@mellanox•com>,
Or Gerlitz <ogerlitz@mellanox•com>,
"Yuval Atias" <yuvala@mellanox•com>
Subject: Re: [PATCH net-next 1/9] net/mlx4_core: Enforce irq affinity changes immediatly
Date: Tue, 13 May 2014 13:17:30 +0300 [thread overview]
Message-ID: <5371F13A.2040803@mellanox.com> (raw)
In-Reply-To: <1399903028.7973.23.camel@edumazet-glaptop2.roam.corp.google.com>
On 5/12/2014 4:57 PM, Eric Dumazet wrote:
> On Mon, 2014-05-12 at 10:43 +0300, Amir Vadai wrote:
>> From: Yuval Atias <yuvala@mellanox•com>
>>
>> During heavy traffic, napi is constatntly polling the complition queue
>> and no interrupt is fired. Because of that, changes to irq affinity are
>> ignored until traffic is stopped and resumed.
>>
>> By registering to the irq notifier mechanism, and forcing interrupt when
>> affinity is changed, irq affinity changes will be immediatly enforced.
>>
>> Signed-off-by: Yuval Atias <yuvala@mellanox•com>
>> Signed-off-by: Amir Vadai <amirv@mellanox•com>
>> ---
>
> Interesting. Its a bit sad it has to be in fast path...
Right.
The other option we thought of - was to add some sort of 'restart' to
the NAPI, to enable the notifier callback to stop any current running
NAPI loop, and make it start again on the right CPU - this looks very
complicated and adding one if doesn't seem to be so bad. So we chose
that option.
>
>> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
>> index 89585c6..81b7571 100644
>> --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
>> +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
>> @@ -474,9 +474,15 @@ int mlx4_en_poll_tx_cq(struct napi_struct *napi, int budget)
>> /* If we used up all the quota - we're probably not done yet... */
>> if (done < budget) {
>> /* Done for now */
>> + cq->mcq.irq_affinity_change = 0;
>
>
>> napi_complete(napi);
>> mlx4_en_arm_cq(priv, cq);
>> return done;
>> + } else if (unlikely(cq->mcq.irq_affinity_change)) {
>> + cq->mcq.irq_affinity_change = 0;
>> + napi_complete(napi);
>> + mlx4_en_arm_cq(priv, cq);
>> + return 0;
>> }
>> return budget;
>
>
>> priv->msix_ctl.pool_bm &= ~(1ULL << i);
>> diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
>> index ba87bd2..74f5aa8 100644
>> --- a/include/linux/mlx4/device.h
>> +++ b/include/linux/mlx4/device.h
>> @@ -586,6 +586,8 @@ struct mlx4_cq {
>>
>> atomic_t refcount;
>> struct completion free;
>> + u16 irq;
>> + bool irq_affinity_change;
>> };
>>
>> struct mlx4_qp {
>
> But could you place irq_affinity_change in another cache line ?
>
> It seems there is a 32bit hole between cons_index and set_ci_db
>
> (Both fields can be moved)
Done in V1
Thanks,
Amir
>
> Thanks
>
>
next prev parent reply other threads:[~2014-05-13 10:18 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-12 7:43 [PATCH net-next 0/9] Mellanox driver update 2014-05-12 Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 1/9] net/mlx4_core: Enforce irq affinity changes immediatly Amir Vadai
2014-05-12 13:57 ` Eric Dumazet
2014-05-13 10:17 ` Amir Vadai [this message]
2014-05-17 20:33 ` Ben Hutchings
2014-05-19 11:33 ` Amir Vadai
2014-05-26 17:20 ` Ben Hutchings
2014-05-12 7:43 ` [PATCH net-next 2/9] net/mlx4_en: User prio mapping gets corrupted when changing number of channels Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 3/9] net/mlx4_en: Fix errors in MAC address changing when port is down Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 4/9] net/mlx4_core: Fix smatch error - possible access to a null variable Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 5/9] net/mlx4_core: Removed unnecessary bit operation condition Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 6/9] net/mlx4_en: Protect MAC address modification with the state_lock mutex Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 7/9] net/mlx4_en: Fix mac_hash database inconsistency Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 8/9] net/mlx4_en: Using positive error value for unsigned Amir Vadai
2014-05-12 7:43 ` [PATCH net-next 9/9] net/mlx4_core: Fix inaccurate return value of mlx4_flow_attach() Amir Vadai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5371F13A.2040803@mellanox.com \
--to=amirv@mellanox$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=ogerlitz@mellanox$(echo .)com \
--cc=yevgenyp@mellanox$(echo .)com \
--cc=yuvala@mellanox$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox