From: Preeti U Murthy <preeti@linux•vnet.ibm.com>
To: Anton Blanchard <anton@samba•org>, benh@kernel•crashing.org
Cc: paulmck@linux•vnet.ibm.com, paulus@samba•org,
linuxppc-dev@lists•ozlabs.org
Subject: Re: [PATCH] powerpc: irq work racing with timer interrupt can result in timer interrupt hang
Date: Fri, 09 May 2014 15:22:33 +0530 [thread overview]
Message-ID: <536CA561.8010803@linux.vnet.ibm.com> (raw)
In-Reply-To: <20140509174712.55fe72d0@kryten>
Hi Anton,
On 05/09/2014 01:17 PM, Anton Blanchard wrote:
> I am seeing an issue where a CPU running perf eventually hangs.
> Traces show timer interrupts happening every 4 seconds even
> when a userspace task is running on the CPU. /proc/timer_list
> also shows pending hrtimers have not run in over an hour,
> including the scheduler.
>
> Looking closer, decrementers_next_tb is getting set to
> 0xffffffffffffffff, and at that point we will never take
> a timer interrupt again.
>
> In __timer_interrupt() we set decrementers_next_tb to
> 0xffffffffffffffff and rely on ->event_handler to update it:
>
> *next_tb = ~(u64)0;
> if (evt->event_handler)
> evt->event_handler(evt);
>
> In this case ->event_handler is hrtimer_interrupt. This will eventually
> call back through the clockevents code with the next event to be
> programmed:
>
> static int decrementer_set_next_event(unsigned long evt,
> struct clock_event_device *dev)
> {
> /* Don't adjust the decrementer if some irq work is pending */
> if (test_irq_work_pending())
> return 0;
> __get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
>
> If irq work came in between these two points, we will return
> before updating decrementers_next_tb and we never process a timer
> interrupt again.
>
> This looks to have been introduced by 0215f7d8c53f (powerpc: Fix races
> with irq_work). Fix it by removing the early exit and relying on
> code later on in the function to force an early decrementer:
>
> /* We may have raced with new irq work */
> if (test_irq_work_pending())
> set_dec(1);
>
There is another scenario we are missing. Its not necessary that on a
timer interrupt the event handler will call back through the
set_next_event().
If there are no pending timers then the event handler will not bother
programming the tick device and simply return.IOW, set_next_event() will
not be called. In that case we will miss taking care of pending irq work
altogether.
__timer_interrupt() -> event_handler -> next_time = KTIME_MAX ->
__timer_interrupt().
In __timer_interrupt() we do not check for pending irq anywhere after
the call to the event handler and we hence miss servicing irqs in the
above scenario.
How about you also move the check:
if (test_irq_pending())
set_dec(1)
in __timer_interrupt() outside the _else_ loop? This will ensure that no
matter what, before exiting timer interrupt handler we check for pending
irq work.
Regards
Preeti U Murthy
> Signed-off-by: Anton Blanchard <anton@samba•org>
> Cc: stable@vger•kernel.org # 3.14+
> ---
>
> diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
> index 122a580..4f0b676 100644
> --- a/arch/powerpc/kernel/time.c
> +++ b/arch/powerpc/kernel/time.c
> @@ -813,9 +888,6 @@ static void __init clocksource_init(void)
> static int decrementer_set_next_event(unsigned long evt,
> struct clock_event_device *dev)
> {
> - /* Don't adjust the decrementer if some irq work is pending */
> - if (test_irq_work_pending())
> - return 0;
> __get_cpu_var(decrementers_next_tb) = get_tb_or_rtc() + evt;
> set_dec(evt);
How about if you move the test_irq_work_pending
Why do we have test_irq_work_pending() later in the function
decrementer_set_next_event()?
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists•ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>
next prev parent reply other threads:[~2014-05-09 9:57 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-09 7:47 [PATCH] powerpc: irq work racing with timer interrupt can result in timer interrupt hang Anton Blanchard
2014-05-09 9:52 ` Preeti U Murthy [this message]
2014-05-10 4:26 ` Benjamin Herrenschmidt
2014-05-10 15:36 ` Preeti U Murthy
2014-05-10 22:25 ` Benjamin Herrenschmidt
2014-05-11 8:15 ` Preeti U Murthy
2014-05-11 8:37 ` Benjamin Herrenschmidt
2014-05-11 8:43 ` Preeti U Murthy
2014-05-11 9:03 ` Benjamin Herrenschmidt
2014-05-11 9:07 ` Preeti U Murthy
2014-05-09 13:41 ` Paul E. McKenney
2014-05-09 21:50 ` Gabriel Paubert
2014-05-09 22:08 ` Paul E. McKenney
2014-05-10 6:33 ` Paul Mackerras
2014-05-10 16:33 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=536CA561.8010803@linux.vnet.ibm.com \
--to=preeti@linux$(echo .)vnet.ibm.com \
--cc=anton@samba$(echo .)org \
--cc=benh@kernel$(echo .)crashing.org \
--cc=linuxppc-dev@lists$(echo .)ozlabs.org \
--cc=paulmck@linux$(echo .)vnet.ibm.com \
--cc=paulus@samba$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox