From: Bert Karwatzki <spasswolf@web•de>
To: Mateusz Guzik <mjguzik@gmail•com>,
Christian Brauner <brauner@kernel•org>
Cc: linux-kernel@vger•kernel.org, linux-next@vger•kernel.org,
linux-rt-devel@lists•linux.dev, linux-fsdevel@vger•kernel.org,
adobriyan@gmail•com, jack@suse•cz, viro@zeniv•linux.org.uk,
Sebastian Andrzej Siewior <bigeasy@linutronix•de>,
Thomas Gleixner <tglx@linutronix•de>,
spasswolf@web•de
Subject: Re: context switch within RCU read-side critical section in next-20260518+ with PREEMPT_RT
Date: Thu, 21 May 2026 11:20:39 +0200 [thread overview]
Message-ID: <4f548d61b2dd12e01f401ce4b8c865f238f7b23c.camel@web.de> (raw)
In-Reply-To: <s7cu3dpioidx6mepmai6eyj2pxjs4skbw7v534zbzs6g2fwcis@cvokidcxy3xa>
Am Donnerstag, dem 21.05.2026 um 11:09 +0200 schrieb Mateusz Guzik:
> On Thu, May 21, 2026 at 10:53:03AM +0200, Mateusz Guzik wrote:
> > Christian, can you fold this in please.
> >
> > diff --git a/fs/filesystems.c b/fs/filesystems.c
> > index 771fc31a69b8..8f17c0abbc95 100644
> > --- a/fs/filesystems.c
> > +++ b/fs/filesystems.c
> > @@ -289,6 +289,7 @@ static __cold noinline int regen_filesystems_string(void)
> > * Did someone beat us to it?
> > */
> > if (old && old->gen == file_systems_gen) {
> > + spin_unlock(&file_systems_lock);
> > kfree(new);
> > return 0;
> > }
> > @@ -297,6 +298,7 @@ static __cold noinline int regen_filesystems_string(void)
> > * Did the list change in the meantime?
> > */
> > if (gen != file_systems_gen) {
> > + spin_unlock(&file_systems_lock);
> > kfree(new);
> > goto retry;
> > }
> >
> >
>
> Even better, I got the above fixup + some polish listed below:
> - removed an extra space in newlen calculation
> - the WARN_ON_ONCE case needs to free 'new', not 'old'
> - there is no READ_ONCE anymore in filesystems_proc_show()
>
> goes into the "fs: cache the string generated by reading /proc/filesystems"
> commit.
>
> diff --git a/fs/filesystems.c b/fs/filesystems.c
> index 771fc31a69b8..712316a1e3e0 100644
> --- a/fs/filesystems.c
> +++ b/fs/filesystems.c
> @@ -269,7 +269,7 @@ static __cold noinline int regen_filesystems_string(void)
> hlist_for_each_entry_rcu(p, &file_systems, list) {
> if (!(p->fs_flags & FS_REQUIRES_DEV))
> newlen += strlen("nodev");
> - newlen += strlen("\t") + strlen(p->name) + strlen("\n");
> + newlen += strlen("\t") + strlen(p->name) + strlen("\n");
> }
> spin_unlock(&file_systems_lock);
>
> @@ -289,6 +289,7 @@ static __cold noinline int regen_filesystems_string(void)
> * Did someone beat us to it?
> */
> if (old && old->gen == file_systems_gen) {
> + spin_unlock(&file_systems_lock);
> kfree(new);
> return 0;
> }
> @@ -297,6 +298,7 @@ static __cold noinline int regen_filesystems_string(void)
> * Did the list change in the meantime?
> */
> if (gen != file_systems_gen) {
> + spin_unlock(&file_systems_lock);
> kfree(new);
> goto retry;
> }
> @@ -321,13 +323,12 @@ static __cold noinline int regen_filesystems_string(void)
> * generation above and messes it up.
> */
> spin_unlock(&file_systems_lock);
> - if (old)
> - kfree_rcu(old, rcu);
> + kfree(new);
> return -EINVAL;
> }
>
> /*
> - * Paired with consume fence in READ_ONCE() in filesystems_proc_show()
> + * Paired with consume fence in rcu_dereference() in filesystems_proc_show()
> */
> smp_store_release(&file_systems_string, new);
> spin_unlock(&file_systems_lock);
>
So it was commit 36b3306779ea
("fs: cache the string generated by reading /proc/filesystems")
which caused the problem. If I had finished the bisection properly instead
of cutting I probably would have noticed this...
So I tested
diff --git a/fs/filesystems.c b/fs/filesystems.c
index 771fc31a69b8..8f17c0abbc95 100644
--- a/fs/filesystems.c
+++ b/fs/filesystems.c
@@ -289,6 +289,7 @@ static __cold noinline int regen_filesystems_string(void)
* Did someone beat us to it?
*/
if (old && old->gen == file_systems_gen) {
+ spin_unlock(&file_systems_lock);
kfree(new);
return 0;
}
@@ -297,6 +298,7 @@ static __cold noinline int regen_filesystems_string(void)
* Did the list change in the meantime?
*/
if (gen != file_systems_gen) {
+ spin_unlock(&file_systems_lock);
kfree(new);
goto retry;
}
with next-20260519 (no RT, no LOCKDEP) and got no crash so far (4 boots only though (next-20260619
crashed in 2 out of 3 boots without RT)) but I get this warning on every boot:
[ 2.793416] [ T331] ------------[ cut here ]------------
[ 2.793433] [ T331] DEBUG_LOCKS_WARN_ON(lock->magic != lock)
[ 2.793434] [ T331] WARNING: kernel/locking/mutex.c:625 at __mutex_lock+0x586/0x10c0, CPU#17: (udev-worker)/331
[ 2.793463] [ T331] Modules linked in: amdgpu(+) hid_generic usbhid drm_client_lib i2c_algo_bit drm_buddy hid drm_ttm_helper ttm drm_exec
drm_suballoc_helper mfd_core drm_panel_backlight_quirks gpu_sched amdxcp drm_display_helper drm_kms_helper ahci libahci xhci_pci libata xhci_hcd drm nvme
scsi_mod igc usbcore nvme_core scsi_common video nvme_keyring i2c_piix4 cec nvme_auth usb_common crc16 i2c_smbus wmi gpio_amdpt gpio_generic
[ 2.793518] [ T331] CPU: 17 UID: 0 PID: 331 Comm: (udev-worker) Not tainted 7.1.0-rc4-next-20260519-rcunortlockdep-dirty #465 PREEMPT
[ 2.793534] [ T331] Hardware name: ASUS System Product Name/ROG STRIX B850-F GAMING WIFI, BIOS 1627 02/05/2026
[ 2.793547] [ T331] RIP: 0010:__mutex_lock+0x58d/0x10c0
[ 2.793555] [ T331] Code: 4c 8b 4d 88 85 c0 0f 84 f8 fa ff ff 44 8b 15 ca 9b 81 00 45 85 d2 0f 85 e8 fa ff ff 48 8d 3d 1a 57 82 00 48 c7 c6 a6 51 9e 83
<67> 48 0f b9 3a 4c 8b 4d 88 e9 cc fa ff ff 48 8b bd 78 ff ff ff e8
[ 2.793579] [ T331] RSP: 0018:ffffa497016c3510 EFLAGS: 00010246
[ 2.793588] [ T331] RAX: 0000000000000001 RBX: ffff88c33a4c2ad8 RCX: 0000000000000000
[ 2.793598] [ T331] RDX: 0000000000000001 RSI: ffffffff839e51a6 RDI: ffffffff83de3c00
[ 2.793609] [ T331] RBP: ffffa497016c35c0 R08: ffffffffc0a55d92 R09: 0000000000000000
[ 2.793619] [ T331] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 2.793629] [ T331] R13: 0000000000000002 R14: ffffa497016c3550 R15: 0000000000268000
[ 2.793641] [ T331] FS: 00007f1f32e5b9c0(0000) GS:ffff88d23b2ca000(0000) knlGS:0000000000000000
[ 2.793653] [ T331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2.793662] [ T331] CR2: 000055cdfa28f588 CR3: 0000000112e73000 CR4: 0000000000f50ef0
[ 2.793673] [ T331] PKRU: 55555554
[ 2.793678] [ T331] Call Trace:
[ 2.793683] [ T331] <TASK>
[ 2.793687] [ T331] ? lock_acquire+0xbe/0x2d0
[ 2.793696] [ T331] ? init_mqd+0x122/0x190 [amdgpu]
[ 2.793809] [ T331] ? lock_release+0xc6/0x2a0
[ 2.793816] [ T331] ? init_mqd+0x122/0x190 [amdgpu]
[ 2.793902] [ T331] init_mqd+0x122/0x190 [amdgpu]
[ 2.793961] [ T331] init_mqd_hiq+0xd/0x20 [amdgpu]
[ 2.794015] [ T331] kq_initialize.constprop.0+0x2b8/0x370 [amdgpu]
[ 2.794071] [ T331] kernel_queue_init+0x3f/0x60 [amdgpu]
[ 2.794125] [ T331] pm_init+0x6b/0x100 [amdgpu]
[ 2.794178] [ T331] start_cpsch+0x1d6/0x270 [amdgpu]
[ 2.794234] [ T331] kgd2kfd_device_init.cold+0x7b9/0xa1a [amdgpu]
[ 2.794365] [ T331] amdgpu_amdkfd_device_init+0x190/0x260 [amdgpu]
[ 2.794444] [ T331] amdgpu_device_init.cold+0x1952/0x1c79 [amdgpu]
[ 2.794556] [ T331] amdgpu_driver_load_kms+0x14/0x80 [amdgpu]
[ 2.794622] [ T331] amdgpu_pci_probe+0x1cd/0x440 [amdgpu]
[ 2.794684] [ T331] pci_device_probe+0xc2/0x1a0
[ 2.794693] [ T331] really_probe+0xd9/0x370
[ 2.794701] [ T331] ? __device_attach_driver+0x130/0x130
[ 2.794710] [ T331] __driver_probe_device+0x80/0x150
[ 2.794718] [ T331] driver_probe_device+0x1a/0x80
[ 2.794726] [ T331] __driver_attach+0xb9/0x1f0
[ 2.794734] [ T331] bus_for_each_dev+0x7b/0xd0
[ 2.794742] [ T331] bus_add_driver+0x11d/0x200
[ 2.794749] [ T331] driver_register+0x6d/0xc0
[ 2.794756] [ T331] ? ledtrig_usb_exit+0x880/0x880 [usb_common]
[ 2.794767] [ T331] do_one_initcall+0x57/0x3a0
[ 2.794774] [ T331] ? __kmalloc_cache_noprof+0x323/0x3f0
[ 2.794785] [ T331] do_init_module+0x5b/0x210
[ 2.794793] [ T331] init_module_from_file+0xd4/0x130
[ 2.794802] [ T331] idempotent_init_module+0x100/0x300
[ 2.794811] [ T331] __x64_sys_finit_module+0x6c/0xe0
[ 2.794819] [ T331] ? kmem_cache_free+0x1e9/0x420
[ 2.794827] [ T331] do_syscall_64+0xf8/0x6b0
[ 2.794834] [ T331] ? lock_release+0xc6/0x2a0
[ 2.794842] [ T331] ? kmem_cache_free+0x279/0x420
[ 2.794849] [ T331] ? do_sys_openat2+0x80/0xc0
[ 2.794857] [ T331] ? __x64_sys_openat+0x4f/0xa0
[ 2.794866] [ T331] ? do_syscall_64+0x1ef/0x6b0
[ 2.794873] [ T331] ? do_syscall_64+0x1ef/0x6b0
[ 2.794880] [ T331] ? do_syscall_64+0x1ef/0x6b0
[ 2.794888] [ T331] ? do_syscall_64+0x1ef/0x6b0
[ 2.794895] [ T331] ? do_syscall_64+0x1ef/0x6b0
[ 2.794903] [ T331] ? lockdep_hardirqs_on_prepare+0xd7/0x180
[ 2.794912] [ T331] ? do_syscall_64+0x38/0x6b0
[ 2.794919] [ T331] ? do_syscall_64+0xad/0x6b0
[ 2.794926] [ T331] entry_SYSCALL_64_after_hwframe+0x55/0x5d
[ 2.795281] [ T331] RIP: 0033:0x7f1f339b97b9
[ 2.795620] [ T331] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05
<48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 27 66 0d 00 f7 d8 64 89 01 48
[ 2.795936] [ T331] RSP: 002b:00007ffe5b4ce6d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 2.796219] [ T331] RAX: ffffffffffffffda RBX: 000055cdfa2b8a20 RCX: 00007f1f339b97b9
[ 2.796490] [ T331] RDX: 0000000000000004 RSI: 00007f1f320f644d RDI: 000000000000003b
[ 2.796741] [ T331] RBP: 0000000000000004 R08: 0000000000000000 R09: 000055cdfa282d70
[ 2.796986] [ T331] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f1f320f644d
[ 2.797227] [ T331] R13: 0000000000020000 R14: 000055cdfa286d60 R15: 0000000000000000
[ 2.797461] [ T331] </TASK>
[ 2.797680] [ T331] irq event stamp: 160663
[ 2.797897] [ T331] hardirqs last enabled at (160663): [<ffffffff835c6cbf>] _raw_spin_unlock_irqrestore+0x3f/0x50
[ 2.798125] [ T331] hardirqs last disabled at (160662): [<ffffffff835c6a7f>] _raw_spin_lock_irqsave+0x4f/0x60
[ 2.798350] [ T331] softirqs last enabled at (160282): [<ffffffff82ac9888>] __irq_exit_rcu+0xc8/0x130
[ 2.798581] [ T331] softirqs last disabled at (160277): [<ffffffff82ac9888>] __irq_exit_rcu+0xc8/0x130
[ 2.798806] [ T331] ---[ end trace 0000000000000000 ]---
Bert Karwatzki
next prev parent reply other threads:[~2026-05-21 9:21 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-20 22:52 context switch within RCU read-side critical section in next-20260518+ with PREEMPT_RT Bert Karwatzki
2026-05-21 8:37 ` Thomas Gleixner
2026-05-21 8:53 ` Mateusz Guzik
2026-05-21 9:08 ` Sebastian Andrzej Siewior
2026-05-21 9:17 ` Mateusz Guzik
2026-05-21 9:09 ` Mateusz Guzik
2026-05-21 9:20 ` Bert Karwatzki [this message]
2026-05-21 9:25 ` Mateusz Guzik
2026-05-21 9:57 ` Bert Karwatzki
2026-05-21 10:17 ` Thomas Gleixner
2026-05-21 10:21 ` Bert Karwatzki
2026-05-21 10:33 ` Mateusz Guzik
2026-05-21 11:50 ` Bert Karwatzki
2026-05-21 12:01 ` Mateusz Guzik
2026-05-28 17:59 ` Bert Karwatzki
2026-05-29 17:20 ` Mateusz Guzik
2026-05-21 10:05 ` Thomas Gleixner
2026-05-21 10:13 ` Bert Karwatzki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4f548d61b2dd12e01f401ce4b8c865f238f7b23c.camel@web.de \
--to=spasswolf@web$(echo .)de \
--cc=adobriyan@gmail$(echo .)com \
--cc=bigeasy@linutronix$(echo .)de \
--cc=brauner@kernel$(echo .)org \
--cc=jack@suse$(echo .)cz \
--cc=linux-fsdevel@vger$(echo .)kernel.org \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linux-next@vger$(echo .)kernel.org \
--cc=linux-rt-devel@lists$(echo .)linux.dev \
--cc=mjguzik@gmail$(echo .)com \
--cc=tglx@linutronix$(echo .)de \
--cc=viro@zeniv$(echo .)linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox