From: "Paul E. McKenney" <paulmck@linux•vnet.ibm.com>
To: Anton Blanchard <anton@samba•org>
Cc: paulus@samba•org, linuxppc-dev@lists•ozlabs.org
Subject: Re: [PATCH] powerpc: irq work racing with timer interrupt can result in timer interrupt hang
Date: Fri, 9 May 2014 06:41:13 -0700 [thread overview]
Message-ID: <20140509134113.GP8754@linux.vnet.ibm.com> (raw)
In-Reply-To: <20140509174712.55fe72d0@kryten>
On Fri, May 09, 2014 at 05:47:12PM +1000, Anton Blanchard wrote:
> I am seeing an issue where a CPU running perf eventually hangs.
> Traces show timer interrupts happening every 4 seconds even
> when a userspace task is running on the CPU.
Is this by chance every 4.2 seconds? The reason I ask is that
Paul Clarke and I are seeing an interrupt every 4.2 seconds when
he runs NO_HZ_FULL, and are trying to get rid of it. ;-)
Thanx, Paul
> /proc/timer_list
> also shows pending hrtimers have not run in over an hour,
> including the scheduler.
>
> Looking closer, decrementers_next_tb is getting set to
> 0xffffffffffffffff, and at that point we will never take
> a timer interrupt again.
>
> In __timer_interrupt() we set decrementers_next_tb to
> 0xffffffffffffffff and rely on ->event_handler to update it:
>
> *next_tb = ~(u64)0;
> if (evt->event_handler)
> evt->event_handler(evt);
>
> In this case ->event_handler is hrtimer_interrupt. This will eventually
> call back through the clockevents code with the next event to be
> programmed:
>
> static int decrementer_set_next_event(unsigned long evt,
> struct clock_event_device *dev)
> {
> /* Don't adjust the decrementer if some irq work is pending */
> if (test_irq_work_pending())
> return 0;
> __get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
>
> If irq work came in between these two points, we will return
> before updating decrementers_next_tb and we never process a timer
> interrupt again.
>
> This looks to have been introduced by 0215f7d8c53f (powerpc: Fix races
> with irq_work). Fix it by removing the early exit and relying on
> code later on in the function to force an early decrementer:
>
> /* We may have raced with new irq work */
> if (test_irq_work_pending())
> set_dec(1);
>
> Signed-off-by: Anton Blanchard <anton@samba•org>
> Cc: stable@vger•kernel.org # 3.14+
> ---
>
> diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
> index 122a580..4f0b676 100644
> --- a/arch/powerpc/kernel/time.c
> +++ b/arch/powerpc/kernel/time.c
> @@ -813,9 +888,6 @@ static void __init clocksource_init(void)
> static int decrementer_set_next_event(unsigned long evt,
> struct clock_event_device *dev)
> {
> - /* Don't adjust the decrementer if some irq work is pending */
> - if (test_irq_work_pending())
> - return 0;
> __get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
> set_dec(evt);
>
>
next prev parent reply other threads:[~2014-05-09 13:41 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-09 7:47 [PATCH] powerpc: irq work racing with timer interrupt can result in timer interrupt hang Anton Blanchard
2014-05-09 9:52 ` Preeti U Murthy
2014-05-10 4:26 ` Benjamin Herrenschmidt
2014-05-10 15:36 ` Preeti U Murthy
2014-05-10 22:25 ` Benjamin Herrenschmidt
2014-05-11 8:15 ` Preeti U Murthy
2014-05-11 8:37 ` Benjamin Herrenschmidt
2014-05-11 8:43 ` Preeti U Murthy
2014-05-11 9:03 ` Benjamin Herrenschmidt
2014-05-11 9:07 ` Preeti U Murthy
2014-05-09 13:41 ` Paul E. McKenney [this message]
2014-05-09 21:50 ` Gabriel Paubert
2014-05-09 22:08 ` Paul E. McKenney
2014-05-10 6:33 ` Paul Mackerras
2014-05-10 16:33 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140509134113.GP8754@linux.vnet.ibm.com \
--to=paulmck@linux$(echo .)vnet.ibm.com \
--cc=anton@samba$(echo .)org \
--cc=linuxppc-dev@lists$(echo .)ozlabs.org \
--cc=paulus@samba$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox