* CONFIG_NO_HZ causing poor console responsiveness
@ 2010-06-29 19:54 Timur Tabi
2010-06-30 6:49 ` Richard Cochran
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Timur Tabi @ 2010-06-29 19:54 UTC (permalink / raw)
To: Linuxppc-dev Development
I'm adding support for a new e500-based board (the P1022DS), and in
the process I've discovered that enabling CONFIG_NO_HZ (Tickless
System / Dynamic Ticks) causes significant responsiveness problems on
the serial console. When I type on the console, I see delays of up to
a half-second for almost every character. It acts as if there's a
background process eating all the CPU.
I don't have time to debug this thoroughly at the moment. The problem
occurs in the latest kernel, but it appears not to occur in 2.6.32.
Has anyone else seen anything like this?
--
Timur Tabi
Linux kernel developer at Freescale
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: CONFIG_NO_HZ causing poor console responsiveness
2010-06-29 19:54 CONFIG_NO_HZ causing poor console responsiveness Timur Tabi
@ 2010-06-30 6:49 ` Richard Cochran
2010-07-01 21:55 ` Timur Tabi
2010-07-02 5:47 ` Benjamin Herrenschmidt
2 siblings, 0 replies; 7+ messages in thread
From: Richard Cochran @ 2010-06-30 6:49 UTC (permalink / raw)
To: Timur Tabi; +Cc: Linuxppc-dev Development
On Tue, Jun 29, 2010 at 02:54:17PM -0500, Timur Tabi wrote:
> I'm adding support for a new e500-based board (the P1022DS), and in
> the process I've discovered that enabling CONFIG_NO_HZ (Tickless
> System / Dynamic Ticks) causes significant responsiveness problems on
> the serial console. When I type on the console, I see delays of up to
> a half-second for almost every character. It acts as if there's a
> background process eating all the CPU.
>
> I don't have time to debug this thoroughly at the moment. The problem
> occurs in the latest kernel, but it appears not to occur in 2.6.32.
>
> Has anyone else seen anything like this?
Yes, with a P2020RDB and kernel 2.6.35-rc1. Serial console is really
slow on an otherwise idle machine. I assumed it was a hardware
problem.
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: CONFIG_NO_HZ causing poor console responsiveness
2010-06-29 19:54 CONFIG_NO_HZ causing poor console responsiveness Timur Tabi
2010-06-30 6:49 ` Richard Cochran
@ 2010-07-01 21:55 ` Timur Tabi
2010-07-02 3:46 ` Mike Galbraith
2010-07-02 5:47 ` Benjamin Herrenschmidt
2 siblings, 1 reply; 7+ messages in thread
From: Timur Tabi @ 2010-07-01 21:55 UTC (permalink / raw)
To: Linuxppc-dev Development; +Cc: Mike Galbraith
On Tue, Jun 29, 2010 at 2:54 PM, Timur Tabi <timur@freescale•com> wrote:
> I'm adding support for a new e500-based board (the P1022DS), and in
> the process I've discovered that enabling CONFIG_NO_HZ (Tickless
> System / Dynamic Ticks) causes significant responsiveness problems on
> the serial console. =A0When I type on the console, I see delays of up to
> a half-second for almost every character. =A0It acts as if there's a
> background process eating all the CPU.
I finally finished my git-bisect, and it wasn't that helpful. I had
to skip several commits because the kernel just wouldn't boot:
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
6bc6cf2b61336ed0c55a615eb4c0c8ed5daf3f08
8b911acdf08477c059d1c36c21113ab1696c612b
21406928afe43f1db6acab4931bb8c886f4d04ce
5ca9880c6f4ba4c84b517bc2fed5366adf63d191
a64692a3afd85fe048551ab89142fd5ca99a0dbd
f2e74eeac03ffb779d64b66a643c5e598145a28b
c6ee36c423c3ed1fb86bb3eabba9fc256a300d16
e12f31d3e5d36328c7fbd0fce40a95e70b59152c
13814d42e45dfbe845a0bbe5184565d9236896ae
b42e0c41a422a212ddea0666d5a3a0e3c35206db
39c0cbe2150cbd848a25ba6cdb271d1ad46818ad
beac4c7e4a1cc6d57801f690e5e82fa2c9c245c8
41acab8851a0408c1d5ad6c21a07456f88b54d40
6427462bfa50f50dc6c088c07037264fcc73eca1
c9494727cf293ae2ec66af57547a3e79c724fec2
We cannot bisect more!
These correspond to a batch of scheduler patches, most from Mike Galbraith.
I don't know what to do now. I can't test any of these commits. Even
if I could, they look like they're all part of one set, so I doubt I
could narrow it down to one commit anyway.
--=20
Timur Tabi
Linux kernel developer at Freescale
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: CONFIG_NO_HZ causing poor console responsiveness
2010-07-01 21:55 ` Timur Tabi
@ 2010-07-02 3:46 ` Mike Galbraith
2010-07-02 3:54 ` Tabi Timur-B04825
0 siblings, 1 reply; 7+ messages in thread
From: Mike Galbraith @ 2010-07-02 3:46 UTC (permalink / raw)
To: Timur Tabi; +Cc: Linuxppc-dev Development
On Thu, 2010-07-01 at 16:55 -0500, Timur Tabi wrote:
> On Tue, Jun 29, 2010 at 2:54 PM, Timur Tabi <timur@freescale•com> wrote:
> > I'm adding support for a new e500-based board (the P1022DS), and in
> > the process I've discovered that enabling CONFIG_NO_HZ (Tickless
> > System / Dynamic Ticks) causes significant responsiveness problems on
> > the serial console. When I type on the console, I see delays of up to
> > a half-second for almost every character. It acts as if there's a
> > background process eating all the CPU.
>
> I finally finished my git-bisect, and it wasn't that helpful. I had
> to skip several commits because the kernel just wouldn't boot:
>
> There are only 'skip'ped commits left to test.
> The first bad commit could be any of:
> 6bc6cf2b61336ed0c55a615eb4c0c8ed5daf3f08
> 8b911acdf08477c059d1c36c21113ab1696c612b
> 21406928afe43f1db6acab4931bb8c886f4d04ce
> 5ca9880c6f4ba4c84b517bc2fed5366adf63d191
> a64692a3afd85fe048551ab89142fd5ca99a0dbd
> f2e74eeac03ffb779d64b66a643c5e598145a28b
> c6ee36c423c3ed1fb86bb3eabba9fc256a300d16
> e12f31d3e5d36328c7fbd0fce40a95e70b59152c
> 13814d42e45dfbe845a0bbe5184565d9236896ae
> b42e0c41a422a212ddea0666d5a3a0e3c35206db
> 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad <== the crime scene
> beac4c7e4a1cc6d57801f690e5e82fa2c9c245c8
> 41acab8851a0408c1d5ad6c21a07456f88b54d40
> 6427462bfa50f50dc6c088c07037264fcc73eca1
> c9494727cf293ae2ec66af57547a3e79c724fec2
> We cannot bisect more!
>
> These correspond to a batch of scheduler patches, most from Mike Galbraith.
>
> I don't know what to do now. I can't test any of these commits. Even
> if I could, they look like they're all part of one set, so I doubt I
> could narrow it down to one commit anyway.
Hi Timur,
This has already fixed. Below is the final fix from tip.
commit 3310d4d38fbc514e7b18bd3b1eea8effdd63b5aa
Author: Peter Zijlstra <peterz@infradead•org>
Date: Thu Jun 17 18:02:37 2010 +0200
nohz: Fix nohz ratelimit
Chris Wedgwood reports that 39c0cbe (sched: Rate-limit nohz) causes a
serial console regression, unresponsiveness, and indeed it does. The
reason is that the nohz code is skipped even when the tick was already
stopped before the nohz_ratelimit(cpu) condition changed.
Move the nohz_ratelimit() check to the other conditions which prevent
long idle sleeps.
Reported-by: Chris Wedgwood <cw@f00f•org>
Tested-by: Brian Bloniarz <bmb@athenacr•com>
Signed-off-by: Mike Galbraith <efault@gmx•de>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello•nl>
Cc: Jiri Kosina <jkosina@suse•cz>
Cc: Linus Torvalds <torvalds@linux-foundation•org>
Cc: Greg KH <gregkh@suse•de>
Cc: Alan Cox <alan@lxorguk•ukuu.org.uk>
Cc: OGAWA Hirofumi <hirofumi@mail•parknet.co.jp>
Cc: Jef Driesen <jefdriesen@telenet•be>
LKML-Reference: <1276790557.27822.516.camel@twins>
Signed-off-by: Thomas Gleixner <tglx@linutronix•de>
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 1d7b9bc..783fbad 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -315,9 +315,6 @@ void tick_nohz_stop_sched_tick(int inidle)
goto end;
}
- if (nohz_ratelimit(cpu))
- goto end;
-
ts->idle_calls++;
/* Read jiffies and the time when jiffies were updated last */
do {
@@ -328,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
} while (read_seqretry(&xtime_lock, seq));
if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
- arch_needs_cpu(cpu)) {
+ arch_needs_cpu(cpu) || nohz_ratelimit(cpu)) {
next_jiffies = last_jiffies + 1;
delta_jiffies = 1;
} else {
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: CONFIG_NO_HZ causing poor console responsiveness
2010-07-02 3:46 ` Mike Galbraith
@ 2010-07-02 3:54 ` Tabi Timur-B04825
0 siblings, 0 replies; 7+ messages in thread
From: Tabi Timur-B04825 @ 2010-07-02 3:54 UTC (permalink / raw)
To: Mike Galbraith; +Cc: Linuxppc-dev Development
[-- Attachment #1: Type: text/plain, Size: 217 bytes --]
On Jul 1, 2010, at 10:46 PM, "Mike Galbraith" <efault@gmx•de> wrote:
>
> Hi Timur,
>
> This has already fixed. Below is the final fix from tip.
Than Mike. I thought I was using the latest code, but I guess not.
[-- Attachment #2: Type: text/html, Size: 509 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: CONFIG_NO_HZ causing poor console responsiveness
2010-06-29 19:54 CONFIG_NO_HZ causing poor console responsiveness Timur Tabi
2010-06-30 6:49 ` Richard Cochran
2010-07-01 21:55 ` Timur Tabi
@ 2010-07-02 5:47 ` Benjamin Herrenschmidt
2010-07-02 6:03 ` Li Yang-R58472
2 siblings, 1 reply; 7+ messages in thread
From: Benjamin Herrenschmidt @ 2010-07-02 5:47 UTC (permalink / raw)
To: Timur Tabi; +Cc: Linuxppc-dev Development
On Tue, 2010-06-29 at 14:54 -0500, Timur Tabi wrote:
> I'm adding support for a new e500-based board (the P1022DS), and in
> the process I've discovered that enabling CONFIG_NO_HZ (Tickless
> System / Dynamic Ticks) causes significant responsiveness problems on
> the serial console. When I type on the console, I see delays of up to
> a half-second for almost every character. It acts as if there's a
> background process eating all the CPU.
>
> I don't have time to debug this thoroughly at the moment. The problem
> occurs in the latest kernel, but it appears not to occur in 2.6.32.
>
> Has anyone else seen anything like this?
I noticed that on the bimini with 2.6.35-rc* though I didn't get to
track it down yet.
Cheers,
Ben.
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: CONFIG_NO_HZ causing poor console responsiveness
2010-07-02 5:47 ` Benjamin Herrenschmidt
@ 2010-07-02 6:03 ` Li Yang-R58472
0 siblings, 0 replies; 7+ messages in thread
From: Li Yang-R58472 @ 2010-07-02 6:03 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Tabi Timur-B04825; +Cc: Linuxppc-dev Development
>-----Original Message-----
>From: linuxppc-dev-bounces+leoli=3Dfreescale.com@lists•ozlabs.org
>[mailto:linuxppc-dev-bounces+leoli=3Dfreescale.com@lists•ozlabs.org] On
>Behalf Of Benjamin Herrenschmidt
>Sent: Friday, July 02, 2010 1:47 PM
>To: Tabi Timur-B04825
>Cc: Linuxppc-dev Development
>Subject: Re: CONFIG_NO_HZ causing poor console responsiveness
>
>On Tue, 2010-06-29 at 14:54 -0500, Timur Tabi wrote:
>> I'm adding support for a new e500-based board (the P1022DS), and in
>> the process I've discovered that enabling CONFIG_NO_HZ (Tickless
>> System / Dynamic Ticks) causes significant responsiveness problems on
>> the serial console. When I type on the console, I see delays of up =
to
>> a half-second for almost every character. It acts as if there's a
>> background process eating all the CPU.
>>
>> I don't have time to debug this thoroughly at the moment. The =
problem
>> occurs in the latest kernel, but it appears not to occur in 2.6.32.
>>
>> Has anyone else seen anything like this?
>
>I noticed that on the bimini with 2.6.35-rc* though I didn't get to =
track
>it down yet.
Patch found at the following location fixed this problem.
http://www.spinics.net/lists/linux-tip-commits/msg08279.html
Hope it has already been merged.
- Leo
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2010-07-02 6:04 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-06-29 19:54 CONFIG_NO_HZ causing poor console responsiveness Timur Tabi
2010-06-30 6:49 ` Richard Cochran
2010-07-01 21:55 ` Timur Tabi
2010-07-02 3:46 ` Mike Galbraith
2010-07-02 3:54 ` Tabi Timur-B04825
2010-07-02 5:47 ` Benjamin Herrenschmidt
2010-07-02 6:03 ` Li Yang-R58472
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox