From: Sasha Levin <sashal@kernel•org>
To: linux-kernel@vger•kernel.org, stable@vger•kernel.org
Cc: Song Liu <songliubraving@fb•com>,
Alexei Starovoitov <ast@kernel•org>,
Peter Zijlstra <peterz@infradead•org>,
Daniel Borkmann <daniel@iogearbox•net>,
Sasha Levin <sashal@kernel•org>,
netdev@vger•kernel.org, bpf@vger•kernel.org
Subject: [PATCH AUTOSEL 5.4 123/350] bpf/stackmap: Fix deadlock with rq_lock in bpf_get_stack()
Date: Tue, 10 Dec 2019 16:03:48 -0500 [thread overview]
Message-ID: <20191210210735.9077-84-sashal@kernel.org> (raw)
In-Reply-To: <20191210210735.9077-1-sashal@kernel.org>
From: Song Liu <songliubraving@fb•com>
[ Upstream commit eac9153f2b584c702cea02c1f1a57d85aa9aea42 ]
bpf stackmap with build-id lookup (BPF_F_STACK_BUILD_ID) can trigger A-A
deadlock on rq_lock():
rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[...]
Call Trace:
try_to_wake_up+0x1ad/0x590
wake_up_q+0x54/0x80
rwsem_wake+0x8a/0xb0
bpf_get_stack+0x13c/0x150
bpf_prog_fbdaf42eded9fe46_on_event+0x5e3/0x1000
bpf_overflow_handler+0x60/0x100
__perf_event_overflow+0x4f/0xf0
perf_swevent_overflow+0x99/0xc0
___perf_sw_event+0xe7/0x120
__schedule+0x47d/0x620
schedule+0x29/0x90
futex_wait_queue_me+0xb9/0x110
futex_wait+0x139/0x230
do_futex+0x2ac/0xa50
__x64_sys_futex+0x13c/0x180
do_syscall_64+0x42/0x100
entry_SYSCALL_64_after_hwframe+0x44/0xa9
This can be reproduced by:
1. Start a multi-thread program that does parallel mmap() and malloc();
2. taskset the program to 2 CPUs;
3. Attach bpf program to trace_sched_switch and gather stackmap with
build-id, e.g. with trace.py from bcc tools:
trace.py -U -p <pid> -s <some-bin,some-lib> t:sched:sched_switch
A sample reproducer is attached at the end.
This could also trigger deadlock with other locks that are nested with
rq_lock.
Fix this by checking whether irqs are disabled. Since rq_lock and all
other nested locks are irq safe, it is safe to do up_read() when irqs are
not disable. If the irqs are disabled, postpone up_read() in irq_work.
Fixes: 615755a77b24 ("bpf: extend stackmap to save binary_build_id+offset instead of address")
Signed-off-by: Song Liu <songliubraving@fb•com>
Signed-off-by: Alexei Starovoitov <ast@kernel•org>
Cc: Peter Zijlstra <peterz@infradead•org>
Cc: Alexei Starovoitov <ast@kernel•org>
Cc: Daniel Borkmann <daniel@iogearbox•net>
Link: https://lore.kernel.org/bpf/20191014171223.357174-1-songliubraving@fb.com
Reproducer:
============================ 8< ============================
char *filename;
void *worker(void *p)
{
void *ptr;
int fd;
char *pptr;
fd = open(filename, O_RDONLY);
if (fd < 0)
return NULL;
while (1) {
struct timespec ts = {0, 1000 + rand() % 2000};
ptr = mmap(NULL, 4096 * 64, PROT_READ, MAP_PRIVATE, fd, 0);
usleep(1);
if (ptr == MAP_FAILED) {
printf("failed to mmap\n");
break;
}
munmap(ptr, 4096 * 64);
usleep(1);
pptr = malloc(1);
usleep(1);
pptr[0] = 1;
usleep(1);
free(pptr);
usleep(1);
nanosleep(&ts, NULL);
}
close(fd);
return NULL;
}
int main(int argc, char *argv[])
{
void *ptr;
int i;
pthread_t threads[THREAD_COUNT];
if (argc < 2)
return 0;
filename = argv[1];
for (i = 0; i < THREAD_COUNT; i++) {
if (pthread_create(threads + i, NULL, worker, NULL)) {
fprintf(stderr, "Error creating thread\n");
return 0;
}
}
for (i = 0; i < THREAD_COUNT; i++)
pthread_join(threads[i], NULL);
return 0;
}
============================ 8< ============================
Signed-off-by: Sasha Levin <sashal@kernel•org>
---
kernel/bpf/stackmap.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index 052580c33d268..173e983619d77 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -287,7 +287,7 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
bool irq_work_busy = false;
struct stack_map_irq_work *work = NULL;
- if (in_nmi()) {
+ if (irqs_disabled()) {
work = this_cpu_ptr(&up_read_work);
if (work->irq_work.flags & IRQ_WORK_BUSY)
/* cannot queue more up_read, fallback */
@@ -295,8 +295,9 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
}
/*
- * We cannot do up_read() in nmi context. To do build_id lookup
- * in nmi context, we need to run up_read() in irq_work. We use
+ * We cannot do up_read() when the irq is disabled, because of
+ * risk to deadlock with rq_lock. To do build_id lookup when the
+ * irqs are disabled, we need to run up_read() in irq_work. We use
* a percpu variable to do the irq_work. If the irq_work is
* already used by another lookup, we fall back to report ips.
*
--
2.20.1
next prev parent reply other threads:[~2019-12-10 21:31 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20191210210735.9077-1-sashal@kernel.org>
2019-12-10 21:02 ` [PATCH AUTOSEL 5.4 048/350] ath10k: fix offchannel tx failure when no ath10k_mac_tx_frm_has_freq Sasha Levin
2019-12-10 21:02 ` [PATCH AUTOSEL 5.4 056/350] selftests/bpf: Correct path to include msg + path Sasha Levin
2019-12-10 21:02 ` [PATCH AUTOSEL 5.4 059/350] rtw88: fix NSS of hw_cap Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 077/350] mwifiex: pcie: Fix memory leak in mwifiex_pcie_init_evt_ring Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 079/350] selftests/bpf: Fix btf_dump padding test case Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 080/350] libbpf: Fix struct end padding in btf_dump Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 081/350] libbpf: Fix passing uninitialized bytes to setsockopt Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 082/350] net/smc: increase device refcount for added link group Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 083/350] team: call RCU read lock when walking the port_list Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 106/350] ath10k: Correct error handling of dma_map_single() Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 107/350] rtw88: coex: Set 4 slot mode for A2DP Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 119/350] Bluetooth: missed cpu_to_le16 conversion in hci_init4_req Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 120/350] Bluetooth: Workaround directed advertising bug in Broadcom controllers Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 121/350] Bluetooth: hci_core: fix init for HCI_USER_CHANNEL Sasha Levin
2019-12-10 21:03 ` Sasha Levin [this message]
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 132/350] selftests/bpf: Make a copy of subtest name Sasha Levin
2019-12-10 21:03 ` [PATCH AUTOSEL 5.4 134/350] net: hns3: log and clear hardware error after reset complete Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 139/350] rtl8xxxu: fix RTL8723BU connection failure issue after warm reboot Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 161/350] i40e: initialize ITRN registers with correct values Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 163/350] i40e: Wrong 'Advertised FEC modes' after set FEC to AUTO Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 164/350] net: phy: dp83867: enable robust auto-mdix Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 178/350] net: dsa: LAN9303: select REGMAP when LAN9303 enable Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 185/350] net: hns3: add struct netdev_queue debug info for TX timeout Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 189/350] net/mlx5e: Verify that rule has at least one fwd/drop action Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 193/350] libbpf: Fix error handling in bpf_map__reuse_fd() Sasha Levin
2019-12-10 21:04 ` [PATCH AUTOSEL 5.4 194/350] Bluetooth: Fix advertising duplicated flags Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 200/350] ixgbe: protect TX timestamping from API misuse Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 211/350] rfkill: allocate static minor Sasha Levin
2019-12-11 7:51 ` Greg Kroah-Hartman
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 212/350] bnx2x: Fix PF-VF communication over multi-cos queues Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 217/350] rtlwifi: fix memory leak in rtl92c_set_fw_rsvdpagepkt() Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 229/350] perf tools: Splice events onto evlist even on error Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 233/350] ice: Check for null pointer dereference when setting rings Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 235/350] perf parse: If pmu configuration fails free terms Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 237/350] net: avoid potential false sharing in neighbor related code Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 242/350] libbpf: Fix negative FD close() in xsk_setup_xdp_prog() Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 243/350] s390/bpf: Use kvcalloc for addrs array Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 248/350] ath10k: fix get invalid tx rate for Mesh metric Sasha Levin
2019-12-10 21:05 ` [PATCH AUTOSEL 5.4 250/350] selftests: net: Fix printf format warnings on arm Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 255/350] ice: delay less Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 267/350] bpf, testing: Workaround a verifier failure for test_progs Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 270/350] net: dsa: sja1105: Disallow management xmit during switch reset Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 271/350] r8169: respect EEE user setting when restarting network Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 274/350] net: ethernet: ti: Add dependency for TI_DAVINCI_EMAC Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 276/350] net: bcmgenet: Add RGMII_RXID support Sasha Levin
2019-12-10 21:49 ` Florian Fainelli
2019-12-19 22:26 ` Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 281/350] qtnfmac: fix debugfs support for multiple cards Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 282/350] qtnfmac: fix invalid channel information output Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 284/350] qtnfmac: fix using skb after free Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 291/350] iwlwifi: mvm: fix unaligned read of rx_pkt_status Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 295/350] tun: fix data-race in gro_normal_list() Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 306/350] selftests, bpf: Fix test_tc_tunnel hanging Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 307/350] selftests, bpf: Workaround an alu32 sub-register spilling issue Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 308/350] bnxt_en: Return proper error code for non-existent NVM variable Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 309/350] net: phy: avoid matching all-ones clause 45 PHY IDs Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 313/350] net-af_xdp: Use correct number of channels from ethtool Sasha Levin
2019-12-10 21:06 ` [PATCH AUTOSEL 5.4 314/350] brcmfmac: remove monitor interface when detaching Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 319/350] bnxt_en: Improve RX buffer error handling Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 320/350] iwlwifi: check kasprintf() return value Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 323/350] net: ethernet: ti: ale: clean ale tbl on init and intf restart Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 324/350] mt76: fix possible out-of-bound access in mt7615_fill_txs/mt7603_fill_txs Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 326/350] bpf: Switch bpf_map ref counter to atomic64_t so bpf_map_inc() never fails Sasha Levin
2019-12-10 21:28 ` [oss-drivers] " Jakub Kicinski
2019-12-12 16:25 ` Daniel Borkmann
2019-12-19 23:25 ` Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 327/350] libbpf: Fix call relocation offset calculation bug Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 332/350] mac80211: consider QoS Null frames for STA_NULLFUNC_ACKED Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 338/350] ice: Only disable VF state when freeing each VF resources Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 339/350] ice: Fix setting coalesce to handle DCB configuration Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 342/350] net: phy: initialise phydev speed and duplex sanely Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 343/350] tools, bpf: Fix build for 'make -s tools/bpf O=<dir>' Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 346/350] bpf: Provide better register bounds after jmp32 instructions Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 348/350] ibmvnic: Fix completion structure initialization Sasha Levin
2019-12-10 21:07 ` [PATCH AUTOSEL 5.4 349/350] net: wireless: intel: iwlwifi: fix GRO_NORMAL packet stalling Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191210210735.9077-84-sashal@kernel.org \
--to=sashal@kernel$(echo .)org \
--cc=ast@kernel$(echo .)org \
--cc=bpf@vger$(echo .)kernel.org \
--cc=daniel@iogearbox$(echo .)net \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=netdev@vger$(echo .)kernel.org \
--cc=peterz@infradead$(echo .)org \
--cc=songliubraving@fb$(echo .)com \
--cc=stable@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox