From: "Toke Høiland-Jørgensen" <toke@kernel•org>
To: Hsin-Wei Hung <hsinweih@uci•edu>,
Alexei Starovoitov <ast@kernel•org>,
Daniel Borkmann <daniel@iogearbox•net>,
Andrii Nakryiko <andrii@kernel•org>,
Martin KaFai Lau <kafai@fb•com>, Song Liu <songliubraving@fb•com>,
Yonghong Song <yhs@fb•com>,
John Fastabend <john.fastabend@gmail•com>,
KP Singh <kpsingh@kernel•org>,
netdev@vger•kernel.org, bpf@vger•kernel.org,
Arnaldo Carvalho de Melo <acme@kernel•org>
Subject: Re: Possible deadlock in bpf queue map
Date: Thu, 07 Sep 2023 12:26:27 +0200 [thread overview]
Message-ID: <87o7ienuss.fsf@toke.dk> (raw)
In-Reply-To: <CABcoxUbYwuZUL-xm1+5juO42nJMgpQX7cNyQELYz+g2XkZi9TQ@mail.gmail.com>
+Arnaldo
> Hi,
>
> Our bpf fuzzer, a customized Syzkaller, triggered a lockdep warning in
> the bpf queue map in v5.15. Since queue_stack_maps.c has no major changes
> since v5.15, we think this should still exist in the latest kernel.
> The bug can be occasionally triggered, and we suspect one of the
> eBPF programs involved to be the following one. We also attached the lockdep
> warning at the end.
>
> #define DEFINE_BPF_MAP_NO_KEY(the_map, TypeOfMap, MapFlags,
> TypeOfValue, MaxEntries) \
> struct { \
> __uint(type, TypeOfMap); \
> __uint(map_flags, (MapFlags)); \
> __uint(max_entries, (MaxEntries)); \
> __type(value, TypeOfValue); \
> } the_map SEC(".maps");
>
> DEFINE_BPF_MAP_NO_KEY(map_0, BPF_MAP_TYPE_QUEUE, 0 | BPF_F_WRONLY,
> struct_0, 162);
> SEC("perf_event")
> int func(struct bpf_perf_event_data *ctx) {
> char v0[96] = {};
> uint64_t v1 = 0;
> v1 = bpf_map_pop_elem(&map_0, v0);
> return 163819661;
> }
>
>
> The program is attached to the following perf event.
>
> struct perf_event_attr attr_type_hw = {
> .type = PERF_TYPE_HARDWARE,
> .config = PERF_COUNT_HW_CPU_CYCLES,
> .sample_freq = 50,
> .inherit = 1,
> .freq = 1,
> };
>
> ================================WARNING: inconsistent lock state
> 5.15.26+ #2 Not tainted
> --------------------------------
> inconsistent {INITIAL USE} -> {IN-NMI} usage.
> syz-executor.5/19749 [HC1[1]:SC0[0]:HE0:SE1] takes:
> ffff88804c9fc198 (&qs->lock){..-.}-{2:2}, at: __queue_map_get+0x31/0x250
> {INITIAL USE} state was registered at:
> lock_acquire+0x1a3/0x4b0
> _raw_spin_lock_irqsave+0x48/0x60
> __queue_map_get+0x31/0x250
> bpf_prog_577904e86c81dead_func+0x12/0x4b4
> trace_call_bpf+0x262/0x5d0
> perf_trace_run_bpf_submit+0x91/0x1c0
> perf_trace_sched_switch+0x46c/0x700
> __schedule+0x11b5/0x24a0
> schedule+0xd4/0x270
> futex_wait_queue_me+0x25f/0x520
> futex_wait+0x1e0/0x5f0
> do_futex+0x395/0x1890
> __x64_sys_futex+0x1cb/0x480
> do_syscall_64+0x3b/0xc0
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> irq event stamp: 13640
> hardirqs last enabled at (13639): [<ffffffff95eb2bf4>]
> _raw_spin_unlock_irq+0x24/0x40
> hardirqs last disabled at (13640): [<ffffffff95eb2d4d>]
> _raw_spin_lock_irqsave+0x5d/0x60
> softirqs last enabled at (13464): [<ffffffff93e26de5>] __sys_bpf+0x3e15/0x4e80
> softirqs last disabled at (13462): [<ffffffff93e26da3>] __sys_bpf+0x3dd3/0x4e80
>
> other info that might help us debug this:
> Possible unsafe locking scenario:
>
> CPU0
> ----
> lock(&qs->lock);
> <Interrupt>
> lock(&qs->lock);
Hmm, so that lock() uses raw_spin_lock_irqsave(), which *should* be
disabling interrupts entirely for the critical section. But I guess a
Perf hardware event can still trigger? Which seems like it would
potentially wreak havoc with lots of things, not just this queue map
function?
No idea how to protect against this, though. Hoping Arnaldo knows? :)
-Toke
next prev parent reply other threads:[~2023-09-07 10:26 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-07 5:02 Possible deadlock in bpf queue map Hsin-Wei Hung
2023-09-07 10:26 ` Toke Høiland-Jørgensen [this message]
2023-09-07 11:13 ` Kumar Kartikeya Dwivedi
2023-09-07 13:04 ` Toke Høiland-Jørgensen
2023-09-07 15:40 ` Alexei Starovoitov
2023-09-07 16:05 ` Toke Høiland-Jørgensen
2023-09-07 16:08 ` Alexei Starovoitov
2023-09-08 5:29 ` Hsin-Wei Hung
2023-09-08 0:56 ` Hou Tao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87o7ienuss.fsf@toke.dk \
--to=toke@kernel$(echo .)org \
--cc=acme@kernel$(echo .)org \
--cc=andrii@kernel$(echo .)org \
--cc=ast@kernel$(echo .)org \
--cc=bpf@vger$(echo .)kernel.org \
--cc=daniel@iogearbox$(echo .)net \
--cc=hsinweih@uci$(echo .)edu \
--cc=john.fastabend@gmail$(echo .)com \
--cc=kafai@fb$(echo .)com \
--cc=kpsingh@kernel$(echo .)org \
--cc=netdev@vger$(echo .)kernel.org \
--cc=songliubraving@fb$(echo .)com \
--cc=yhs@fb$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox