From: "Paul E. McKenney" <paulmck@kernel•org>
To: Tomas Glozar <tglozar@redhat•com>
Cc: Valentin Schneider <vschneid@redhat•com>,
Chen Yu <yu.c.chen@intel•com>,
Peter Zijlstra <peterz@infradead•org>,
linux-kernel@vger•kernel.org, sfr@canb•auug.org.au,
linux-next@vger•kernel.org, kernel-team@meta•com
Subject: Re: [BUG almost bisected] Splat in dequeue_rt_stack() and build error
Date: Thu, 10 Oct 2024 08:01:35 -0700 [thread overview]
Message-ID: <35e44f60-0a2f-49a7-b44b-c6537544a888@paulmck-laptop> (raw)
In-Reply-To: <CAP4=nvTeawTjhWR0jKNGweeQFvcTr8S=bNiLsSbaKiz=od+EOA@mail.gmail.com>
On Thu, Oct 10, 2024 at 01:24:11PM +0200, Tomas Glozar wrote:
> st 2. 10. 2024 v 11:01 odesílatel Tomas Glozar <tglozar@redhat•com> napsal:
> >
> > FYI I have managed to reproduce the bug on our infrastructure after 21
> > hours of 7*TREE03 and I will continue with trying to reproduce it with
> > the tracers we want.
> >
> > Tomas
>
> I successfully reproduced the bug also with the tracers active after a
> few 8-hour test runs on our infrastructure:
>
> [ 0.000000] Linux version 6.11.0-g2004cef11ea0-dirty (...) #1 SMP
> PREEMPT_DYNAMIC Wed Oct 9 12:13:40 EDT 2024
> [ 0.000000] Command line: debug_boot_weak_hash panic=-1 selinux=0
> initcall_debug debug console=ttyS0 rcutorture.n_barrier_cbs=4
> rcutorture.stat_interval=15 rcutorture.shutdown_secs=25200
> rcutorture.test_no_idle_hz=1 rcutorture.verbose=1
> rcutorture.onoff_interval=200 rcutorture.onoff_holdoff=30
> rcutree.gp_preinit_delay=12 rcutree.gp_init_delay=3
> rcutree.gp_cleanup_delay=3 rcutree.kthread_prio=2 threadirqs
> rcutree.use_softirq=0
> trace_event=sched:sched_switch,sched:sched_wakeup
> ftrace_filter=dl_server_start,dl_server_stop trace_buf_size=2k
> ftrace=function torture.ftrace_dump_at_shutdown=1
> ...
> [13550.127541] WARNING: CPU: 1 PID: 155 at
> kernel/sched/deadline.c:1971 enqueue_dl_entity+0x554/0x5d0
> [13550.128982] Modules linked in:
> [13550.129528] CPU: 1 UID: 0 PID: 155 Comm: rcu_torture_rea Tainted: G
> W 6.11.0-g2004cef11ea0-dirty #1
> [13550.131419] Tainted: [W]=WARN
> [13550.131979] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9 04/01/2014
> [13550.133230] RIP: 0010:enqueue_dl_entity+0x554/0x5d0
> ...
> [13550.151286] Call Trace:
> [13550.151749] <TASK>
> [13550.152141] ? __warn+0x88/0x130
> [13550.152717] ? enqueue_dl_entity+0x554/0x5d0
> [13550.153485] ? report_bug+0x18e/0x1a0
> [13550.154149] ? handle_bug+0x54/0x90
> [13550.154792] ? exc_invalid_op+0x18/0x70
> [13550.155484] ? asm_exc_invalid_op+0x1a/0x20
> [13550.156249] ? enqueue_dl_entity+0x554/0x5d0
> [13550.157055] dl_server_start+0x36/0xf0
> [13550.157709] enqueue_task_fair+0x220/0x6b0
> [13550.158447] activate_task+0x26/0x60
> [13550.159131] attach_task+0x35/0x50
> [13550.159756] sched_balance_rq+0x663/0xe00
> [13550.160511] sched_balance_newidle.constprop.0+0x1a5/0x360
> [13550.161520] pick_next_task_fair+0x2f/0x340
> [13550.162290] __schedule+0x203/0x900
> [13550.162958] ? enqueue_hrtimer+0x35/0x90
> [13550.163703] schedule+0x27/0xd0
> [13550.164299] schedule_hrtimeout_range_clock+0x99/0x120
> [13550.165239] ? __pfx_hrtimer_wakeup+0x10/0x10
> [13550.165954] torture_hrtimeout_us+0x7b/0xe0
> [13550.166624] rcu_torture_reader+0x139/0x200
> [13550.167284] ? __pfx_rcu_torture_timer+0x10/0x10
> [13550.168019] ? __pfx_rcu_torture_reader+0x10/0x10
> [13550.168764] kthread+0xd6/0x100
> [13550.169262] ? __pfx_kthread+0x10/0x10
> [13550.169860] ret_from_fork+0x34/0x50
> [13550.170424] ? __pfx_kthread+0x10/0x10
> [13550.171020] ret_from_fork_asm+0x1a/0x30
> [13550.171657] </TASK>
>
> Unfortunately, the following rcu stalls appear to have resulted in
> abnormal termination of the VM, which led to the ftrace buffer not
> being dumped into the console. Currently re-running the same test with
> the addition of "ftrace_dump_on_oops panic_on_warn=1" and hoping for
> the best.
Another approach would be rcupdate.rcu_cpu_stall_suppress=1.
We probably need to disable RCU CPU stall warnings automatically while
dumping ftrace buffers, but the asynchronous nature of printk() makes
it difficult to work out when to automatically re-enable them...
Thoughts?
Thanx, Paul
next prev parent reply other threads:[~2024-10-10 15:01 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-21 21:57 [BUG almost bisected] Splat in dequeue_rt_stack() and build error Paul E. McKenney
2024-08-22 23:01 ` Paul E. McKenney
2024-08-23 7:47 ` Peter Zijlstra
2024-08-23 12:46 ` Paul E. McKenney
2024-08-23 21:51 ` Paul E. McKenney
2024-08-24 6:54 ` Peter Zijlstra
2024-08-24 15:26 ` Paul E. McKenney
2024-08-25 2:10 ` Paul E. McKenney
2024-08-25 19:36 ` Paul E. McKenney
2024-08-26 11:44 ` Valentin Schneider
2024-08-26 16:31 ` Paul E. McKenney
2024-08-27 10:03 ` Valentin Schneider
2024-08-27 15:41 ` Valentin Schneider
2024-08-27 17:33 ` Paul E. McKenney
2024-08-27 18:35 ` Paul E. McKenney
2024-08-27 20:30 ` Valentin Schneider
2024-08-27 20:36 ` Paul E. McKenney
2024-08-28 12:35 ` Valentin Schneider
2024-08-28 13:03 ` Paul E. McKenney
2024-08-28 13:40 ` Paul E. McKenney
2024-08-28 13:44 ` Chen Yu
2024-08-28 14:32 ` Valentin Schneider
2024-08-28 16:35 ` Paul E. McKenney
2024-08-28 18:17 ` Valentin Schneider
2024-08-28 18:39 ` Paul E. McKenney
2024-08-29 10:28 ` Paul E. McKenney
2024-08-29 13:50 ` Valentin Schneider
2024-08-29 14:13 ` Paul E. McKenney
2024-09-08 16:32 ` Paul E. McKenney
2024-09-13 14:08 ` Paul E. McKenney
2024-09-13 16:55 ` Valentin Schneider
2024-09-13 18:00 ` Paul E. McKenney
2024-09-30 19:09 ` Paul E. McKenney
2024-09-30 20:44 ` Valentin Schneider
2024-10-01 10:10 ` Paul E. McKenney
2024-10-01 12:52 ` Valentin Schneider
2024-10-01 16:47 ` Paul E. McKenney
2024-10-02 9:01 ` Tomas Glozar
2024-10-02 12:07 ` Paul E. McKenney
2024-10-10 11:24 ` Tomas Glozar
2024-10-10 15:01 ` Paul E. McKenney [this message]
2024-10-10 23:28 ` Paul E. McKenney
2024-10-14 18:55 ` Paul E. McKenney
2024-10-21 19:25 ` Paul E. McKenney
2024-11-14 18:16 ` Paul E. McKenney
2024-12-15 18:31 ` Paul E. McKenney
2024-12-16 14:38 ` Tomas Glozar
2024-12-16 19:36 ` Paul E. McKenney
2024-12-17 16:42 ` Paul E. McKenney
2024-10-22 6:33 ` Tomas Glozar
2024-10-03 8:40 ` Peter Zijlstra
2024-10-03 8:47 ` Peter Zijlstra
2024-10-03 9:27 ` Peter Zijlstra
2024-10-03 12:28 ` Peter Zijlstra
2024-10-03 12:45 ` Paul E. McKenney
2024-10-03 14:22 ` Peter Zijlstra
2024-10-03 16:04 ` Paul E. McKenney
2024-10-03 18:50 ` Peter Zijlstra
2024-10-03 19:12 ` Paul E. McKenney
2024-10-04 13:22 ` Paul E. McKenney
2024-10-04 13:35 ` Peter Zijlstra
2024-10-06 20:44 ` Paul E. McKenney
2024-10-07 9:34 ` Peter Zijlstra
2024-10-08 11:11 ` Peter Zijlstra
2024-10-08 16:24 ` Paul E. McKenney
2024-10-08 22:34 ` Paul E. McKenney
2024-10-03 12:44 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=35e44f60-0a2f-49a7-b44b-c6537544a888@paulmck-laptop \
--to=paulmck@kernel$(echo .)org \
--cc=kernel-team@meta$(echo .)com \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linux-next@vger$(echo .)kernel.org \
--cc=peterz@infradead$(echo .)org \
--cc=sfr@canb$(echo .)auug.org.au \
--cc=tglozar@redhat$(echo .)com \
--cc=vschneid@redhat$(echo .)com \
--cc=yu.c.chen@intel$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox