From: Heng Qi <hengqi@linux•alibaba.com>
To: Jason Wang <jasowang@redhat•com>
Cc: Paolo Abeni <pabeni@redhat•com>,
netdev@vger•kernel.org,
virtualization@lists•linux-foundation.org, mst@redhat•com,
kuba@kernel•org, yinjun.zhang@corigine•com, edumazet@google•com,
davem@davemloft•net, hawk@kernel•org, john.fastabend@gmail•com,
ast@kernel•org, horms@kernel•org, xuanzhuo@linux•alibaba.com
Subject: Re: [PATCH net-next v6 4/5] virtio-net: add spin lock for ctrl cmd access
Date: Thu, 7 Dec 2023 12:47:09 +0800 [thread overview]
Message-ID: <d2cc8026-eaa8-46d2-ac61-9d228bc409cf@linux.alibaba.com> (raw)
In-Reply-To: <CACGkMEuU18fn8oC=DPNP3Dk=uE0Rutwib7jkoXEZXV+H4H6VcA@mail.gmail.com>
在 2023/12/7 下午12:19, Jason Wang 写道:
> On Wed, Dec 6, 2023 at 9:03 PM Heng Qi <hengqi@linux•alibaba.com> wrote:
>>
>>
>> 在 2023/12/6 下午8:27, Paolo Abeni 写道:
>>> On Tue, 2023-12-05 at 19:05 +0800, Heng Qi wrote:
>>>> 在 2023/12/5 下午4:35, Jason Wang 写道:
>>>>> On Tue, Dec 5, 2023 at 4:02 PM Heng Qi <hengqi@linux•alibaba.com> wrote:
>>>>>> Currently access to ctrl cmd is globally protected via rtnl_lock and works
>>>>>> fine. But if dim work's access to ctrl cmd also holds rtnl_lock, deadlock
>>>>>> may occur due to cancel_work_sync for dim work.
>>>>> Can you explain why?
>>>> For example, during the bus unbind operation, the following call stack
>>>> occurs:
>>>> virtnet_remove -> unregister_netdev -> rtnl_lock[1] -> virtnet_close ->
>>>> cancel_work_sync -> virtnet_rx_dim_work -> rtnl_lock[2] (deadlock occurs).
>>>>
>>>>>> Therefore, treating
>>>>>> ctrl cmd as a separate protection object of the lock is the solution and
>>>>>> the basis for the next patch.
>>>>> Let's don't do that. Reasons are:
>>>>>
>>>>> 1) virtnet_send_command() may wait for cvq commands for an indefinite time
>>>> Yes, I took that into consideration. But ndo_set_rx_mode's need for an
>>>> atomic
>>>> environment rules out the mutex lock.
>>>>
>>>>> 2) hold locks may complicate the future hardening works around cvq
>>>> Agree, but I don't seem to have thought of a better way besides passing
>>>> the lock.
>>>> Do you have any other better ideas or suggestions?
>>> What about:
>>>
>>> - using the rtnl lock only
>>> - virtionet_close() invokes cancel_work(), without flushing the work
>>> - virtnet_remove() calls flush_work() after unregister_netdev(),
>>> outside the rtnl lock
>>>
>>> Should prevent both the deadlock and the UaF.
>>
>> Hi, Paolo and Jason!
>>
>> Thank you very much for your effective suggestions, but I found another
>> solution[1],
>> based on the ideas of rtnl_trylock and refill_work, which works very well:
>>
>> [1]
>> +static void virtnet_rx_dim_work(struct work_struct *work)
>> +{
>> + struct dim *dim = container_of(work, struct dim, work);
>> + struct receive_queue *rq = container_of(dim,
>> + struct receive_queue, dim);
>> + struct virtnet_info *vi = rq->vq->vdev->priv;
>> + struct net_device *dev = vi->dev;
>> + struct dim_cq_moder update_moder;
>> + int i, qnum, err;
>> +
>> + if (!rtnl_trylock())
>> + return;
> Don't we need to reschedule here?
>
> like
>
> if (rq->dim_enabled)
> sechedule_work()
>
> ?
I think no, we don't need this.
The work of each queue will be called by "net_dim()->schedule_work()"
when napi traffic changes (before schedule_work(), the dim->profile_ix
of the corresponding rxq has been updated).
So we only need to traverse and update the profiles of all rxqs in the
work which is obtaining the rtnl_lock.
Thanks!
>
> Thanks
>
>> +
>> + for (i = 0; i < vi->curr_queue_pairs; i++) {
>> + rq = &vi->rq[i];
>> + dim = &rq->dim;
>> + qnum = rq - vi->rq;
>> +
>> + if (!rq->dim_enabled)
>> + continue;
>> +
>> + update_moder = net_dim_get_rx_moderation(dim->mode,
>> dim->profile_ix);
>> + if (update_moder.usec != rq->intr_coal.max_usecs ||
>> + update_moder.pkts != rq->intr_coal.max_packets) {
>> + err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, qnum,
>> + update_moder.usec,
>> + update_moder.pkts);
>> + if (err)
>> + pr_debug("%s: Failed to send dim parameters on rxq%d\n",
>> + dev->name, qnum);
>> + dim->state = DIM_START_MEASURE;
>> + }
>> + }
>> +
>> + rtnl_unlock();
>> +}
>>
>>
>> In addition, other optimizations[2] have been tried, but it may be due
>> to the sparsely
>> scheduled work that the retry condition is always satisfied, affecting
>> performance,
>> so [1] is the final solution:
>>
>> [2]
>>
>> +static void virtnet_rx_dim_work(struct work_struct *work)
>> +{
>> + struct dim *dim = container_of(work, struct dim, work);
>> + struct receive_queue *rq = container_of(dim,
>> + struct receive_queue, dim);
>> + struct virtnet_info *vi = rq->vq->vdev->priv;
>> + struct net_device *dev = vi->dev;
>> + struct dim_cq_moder update_moder;
>> + int i, qnum, err, count;
>> +
>> + if (!rtnl_trylock())
>> + return;
>> +retry:
>> + count = vi->curr_queue_pairs;
>> + for (i = 0; i < vi->curr_queue_pairs; i++) {
>> + rq = &vi->rq[i];
>> + dim = &rq->dim;
>> + qnum = rq - vi->rq;
>> + update_moder = net_dim_get_rx_moderation(dim->mode,
>> dim->profile_ix);
>> + if (update_moder.usec != rq->intr_coal.max_usecs ||
>> + update_moder.pkts != rq->intr_coal.max_packets) {
>> + --count;
>> + err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, qnum,
>> + update_moder.usec,
>> + update_moder.pkts);
>> + if (err)
>> + pr_debug("%s: Failed to send dim parameters on rxq%d\n",
>> + dev->name, qnum);
>> + dim->state = DIM_START_MEASURE;
>> + }
>> + }
>> +
>> + if (need_resched()) {
>> + rtnl_unlock();
>> + schedule();
>> + }
>> +
>> + if (count)
>> + goto retry;
>> +
>> + rtnl_unlock();
>> +}
>>
>> Thanks a lot!
>>
>>> Side note: for this specific case any functional test with a
>>> CONFIG_LOCKDEP enabled build should suffice to catch the deadlock
>>> scenario above.
>>>
>>> Cheers,
>>>
>>> Paolo
next prev parent reply other threads:[~2023-12-07 4:47 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-05 8:02 [PATCH net-next v6 0/5] virtio-net: support dynamic coalescing moderation Heng Qi
2023-12-05 8:02 ` [PATCH net-next v6 1/5] virtio-net: returns whether napi is complete Heng Qi
2023-12-05 8:02 ` [PATCH net-next v6 2/5] virtio-net: separate rx/tx coalescing moderation cmds Heng Qi
2023-12-05 8:02 ` [PATCH net-next v6 3/5] virtio-net: extract virtqueue coalescig cmd for reuse Heng Qi
2023-12-05 8:02 ` [PATCH net-next v6 4/5] virtio-net: add spin lock for ctrl cmd access Heng Qi
2023-12-05 8:35 ` Jason Wang
2023-12-05 11:05 ` Heng Qi
2023-12-06 9:11 ` Jason Wang
2023-12-06 12:27 ` Paolo Abeni
2023-12-06 13:03 ` Heng Qi
2023-12-07 4:19 ` Jason Wang
2023-12-07 4:47 ` Heng Qi [this message]
2023-12-07 5:31 ` Jason Wang
2023-12-05 8:02 ` [PATCH net-next v6 5/5] virtio-net: support rx netdim Heng Qi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d2cc8026-eaa8-46d2-ac61-9d228bc409cf@linux.alibaba.com \
--to=hengqi@linux$(echo .)alibaba.com \
--cc=ast@kernel$(echo .)org \
--cc=davem@davemloft$(echo .)net \
--cc=edumazet@google$(echo .)com \
--cc=hawk@kernel$(echo .)org \
--cc=horms@kernel$(echo .)org \
--cc=jasowang@redhat$(echo .)com \
--cc=john.fastabend@gmail$(echo .)com \
--cc=kuba@kernel$(echo .)org \
--cc=mst@redhat$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=pabeni@redhat$(echo .)com \
--cc=virtualization@lists$(echo .)linux-foundation.org \
--cc=xuanzhuo@linux$(echo .)alibaba.com \
--cc=yinjun.zhang@corigine$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox