From: Matan Barak <matanb@mellanox•com>
To: Eric Dumazet <eric.dumazet@gmail•com>,
Or Gerlitz <ogerlitz@mellanox•com>
Cc: "David S. Miller" <davem@davemloft•net>, <netdev@vger•kernel.org>,
"Amir Vadai" <amirv@mellanox•com>, Tal Alon <talal@mellanox•com>,
Jack Morgenstein <jackm@dev•mellanox.co.il>
Subject: Re: [PATCH V1 net-next 03/10] net/mlx4_core: Use tasklet for user-space CQ completion events
Date: Wed, 10 Dec 2014 17:47:31 +0200 [thread overview]
Message-ID: <54886B13.5040409@mellanox.com> (raw)
In-Reply-To: <1418225599.27198.18.camel@edumazet-glaptop2.roam.corp.google.com>
On 12/10/2014 5:33 PM, Eric Dumazet wrote:
> On Wed, 2014-12-10 at 15:09 +0200, Or Gerlitz wrote:
>> From: Matan Barak <matanb@mellanox•com>
>>
>> Previously, we've fired all our completion callbacks straight from our ISR.
>>
>> Some of those callbacks were lightweight (for example, mlx4_en's and
>> IPoIB napi callbacks), but some of them did more work (for example,
>> the user-space RDMA stack uverbs' completion handler). Besides that,
>> doing more than the minimal work in ISR is generally considered wrong,
>> it could even lead to a hard lockup of the system. Since when a lot
>> of completion events are generated by the hardware, the loop over those
>> events could be so long, that we'll get into a hard lockup by the system
>> watchdog.
>
> ...
>
>> +#define TASKLET_THRESHOLD 1000
>> +
>> +void mlx4_cq_tasklet_cb(unsigned long data)
>> +{
>> + unsigned long flags;
>> + unsigned int i = 0;
>> + struct mlx4_eq_tasklet *ctx = (struct mlx4_eq_tasklet *)data;
>> + struct mlx4_cq *mcq, *temp;
>> +
>> + spin_lock_irqsave(&ctx->lock, flags);
>> + list_splice_tail_init(&ctx->list, &ctx->process_list);
>> + spin_unlock_irqrestore(&ctx->lock, flags);
>> +
>> + list_for_each_entry_safe(mcq, temp, &ctx->process_list, tasklet_ctx.list) {
>> + list_del_init(&mcq->tasklet_ctx.list);
>> + mcq->tasklet_ctx.comp(mcq);
>> + if (atomic_dec_and_test(&mcq->refcount))
>> + complete(&mcq->free);
>> + if (++i == TASKLET_THRESHOLD)
>> + break;
>> + }
>> +
>> + if (i == TASKLET_THRESHOLD)
>> + tasklet_schedule(&ctx->task);
>> +}
>> +
>
> What is the max duration of doing this loop up to 1000 times ?
>
> I suspect it might be too long, but not necessarily detected by
> conventional watchdog.
>
> __do_softirq() uses both a counter and a test against jiffies, with a 2
> ms limit.
You're right - we'll measure it accurately, but I think it took over 2ms
(on a system with 400 CQs opened), including the spin_lock on the
list_splice.
We'll add the jiffies test to V2.
Thanks.
>
> Thanks.
>
>
next prev parent reply other threads:[~2014-12-10 16:23 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-10 13:09 [PATCH V1 net-next 00/10] mlx4 driver update Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 01/10] net/mlx4_en: Set csum level for encapsulated packets Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 02/10] net/mlx4_core: Mask out host side virtualization features for guests Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 03/10] net/mlx4_core: Use tasklet for user-space CQ completion events Or Gerlitz
2014-12-10 15:33 ` Eric Dumazet
2014-12-10 15:47 ` Matan Barak [this message]
2014-12-10 13:09 ` [PATCH V1 net-next 04/10] net/mlx4: Change QP allocation scheme Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 05/10] net/mlx4: Add a check if there are too many reserved QPs Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 06/10] net/mlx4: Add mlx4_bitmap zone allocator Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 07/10] net/mlx4: Add A0 hybrid steering Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 08/10] net/mlx4_core: Add explicit error message when rule doesn't meet configuration Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 09/10] net/mlx4: Refactor QUERY_PORT Or Gerlitz
2014-12-10 13:09 ` [PATCH V1 net-next 10/10] net/mlx4: Add support for A0 steering Or Gerlitz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=54886B13.5040409@mellanox.com \
--to=matanb@mellanox$(echo .)com \
--cc=amirv@mellanox$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=jackm@dev$(echo .)mellanox.co.il \
--cc=netdev@vger$(echo .)kernel.org \
--cc=ogerlitz@mellanox$(echo .)com \
--cc=talal@mellanox$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox