From: Brian Bloniarz <bmb@athenacr•com>
To: Eric Dumazet <eric.dumazet@gmail•com>
Cc: Thomas Gleixner <tglx@linutronix•de>,
Stephen Hemminger <shemminger@vyatta•com>,
netdev@vger•kernel.org, Andi Kleen <andi@firstfloor•org>,
Peter Zijlstra <peterz@infradead•org>
Subject: Re: OFT - reserving CPU's for networking
Date: Fri, 30 Apr 2010 14:15:34 -0400 [thread overview]
Message-ID: <4BDB1E46.6050106@athenacr.com> (raw)
In-Reply-To: <1272571339.2209.76.camel@edumazet-laptop>
Eric Dumazet wrote:
> Le jeudi 29 avril 2010 à 21:19 +0200, Thomas Gleixner a écrit :
>
>> Say thanks to Intel/AMD for providing us timers which stop in lower
>> c-states.
>>
>> Not much we can do about the broadcast lock when several cores are
>> going idle and we need to setup a global timer to work around the
>> lapic timer stops in C2/C3 issue.
>>
>> Simply the C-state timer broadcasting does not scale. And it was never
>> meant to scale. It's a workaround for laptops to have functional NOHZ.
>>
>> There are several ways to work around that on larger machines:
>>
>> - Restrict c-states
>> - Disable NOHZ and highres timers
>> - idle=poll is definitely the worst of all possible solutions
>>
>>> I keep getting asked about taking some core's away from clock and scheduler
>>> to be reserved just for network processing. Seeing this kind of stuff
>>> makes me wonder if maybe that isn't a half bad idea.
>> This comes up every few month and we pointed out several times what
>> needs to be done to make this work w/o these weird hacks which put a
>> core offline and then start some magic undebugable binary blob on it.
>> We have not seen anyone working on this, but the "set cores aside and
>> let them do X" idea seems to stick in peoples heads.
>>
>> Seriously, that's not a solution. It's going to be some hacked up
>> nightmare which is completely unmaintainable.
>>
>> Aside of that I seriously doubt that you can do networking w/o time
>> and timers.
>>
>
> Thanks a lot !
>
> booting with processor.max_cstate=1 solves the problem
>
> (I already had a CONFIG_NO_HZ=no conf, but highres timer enabled)
>
> Even with _carefuly_ chosen crazy configuration (receiving a packet on a
> cpu, then transfert it to another cpu, with a full 16x16 matrix
> involved), generating 700.000 IPI per second on the machine seems fine
> now.
FYI you can also restrict c=states at runtime with PM QoS:
Documentation/power/pm_qos_interface.txt
On my machine, /sys/devices/system/cpu/cpu0/cpuidle/state2/latency
is 205usec, so configuring a PM QoS request for <= 205usec latency
should prevent it being entered:
#!/usr/bin/python
import os;
import struct;
import signal;
latency_rec_usec = 100
f = os.open("/dev/cpu_dma_latency", os.O_WRONLY);
os.write(f, struct.pack("=i", latency_rec_usec));
signal.pause();
next prev parent reply other threads:[~2010-04-30 18:15 UTC|newest]
Thread overview: 108+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-04-23 8:12 [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue Changli Gao
2010-04-23 9:27 ` Eric Dumazet
2010-04-23 22:02 ` jamal
2010-04-24 14:10 ` jamal
2010-04-26 14:03 ` Eric Dumazet
2010-04-26 14:55 ` Eric Dumazet
2010-04-26 21:06 ` jamal
[not found] ` <20100429174056.GA8044@gargoyle.fritz.box>
2010-04-29 17:56 ` Eric Dumazet
2010-04-29 18:10 ` OFT - reserving CPU's for networking Stephen Hemminger
2010-04-29 19:19 ` Thomas Gleixner
2010-04-29 20:02 ` Eric Dumazet
2010-04-30 18:15 ` Brian Bloniarz [this message]
2010-04-30 18:57 ` David Miller
2010-04-30 19:58 ` Thomas Gleixner
2010-04-30 21:01 ` Andi Kleen
2010-04-30 22:30 ` David Miller
2010-05-01 10:53 ` Andi Kleen
2010-05-01 22:03 ` David Miller
2010-05-01 22:58 ` Andi Kleen
2010-05-01 23:29 ` David Miller
2010-05-01 23:44 ` Ben Hutchings
2010-05-01 20:31 ` Martin Josefsson
2010-05-01 22:13 ` David Miller
[not found] ` <20100429182347.GA8512@gargoyle.fritz.box>
2010-04-29 19:12 ` [PATCH v6] net: batch skb dequeueing from softnet input_pkt_queue Eric Dumazet
[not found] ` <20100429214144.GA10663@gargoyle.fritz.box>
2010-04-30 5:25 ` Eric Dumazet
2010-04-30 23:38 ` David Miller
2010-05-01 11:00 ` Andi Kleen
2010-05-02 6:56 ` Eric Dumazet
2010-05-02 9:20 ` Andi Kleen
2010-05-02 10:54 ` Eric Dumazet
2010-05-02 14:13 ` Arjan van de Ven
2010-05-02 14:27 ` Eric Dumazet
2010-05-02 15:32 ` Eric Dumazet
2010-05-02 17:54 ` Arjan van de Ven
2010-05-02 19:22 ` Eric Dumazet
2010-05-02 22:06 ` Andi Kleen
2010-05-03 3:50 ` Arjan van de Ven
2010-05-03 5:17 ` Eric Dumazet
2010-05-03 10:22 ` Arjan van de Ven
2010-05-03 10:34 ` Andi Kleen
2010-05-03 14:09 ` Arjan van de Ven
2010-05-03 14:45 ` Brian Bloniarz
2010-05-04 1:10 ` Arjan van de Ven
2010-05-03 15:52 ` Andi Kleen
2010-05-04 1:11 ` Arjan van de Ven
2010-05-02 21:30 ` Andi Kleen
2010-05-02 15:46 ` Andi Kleen
2010-05-02 16:35 ` Eric Dumazet
2010-05-02 17:43 ` Arjan van de Ven
2010-05-02 17:47 ` Eric Dumazet
2010-05-02 21:25 ` Andi Kleen
2010-05-02 21:45 ` Eric Dumazet
2010-05-02 21:54 ` Andi Kleen
2010-05-02 22:08 ` Eric Dumazet
2010-05-03 20:15 ` jamal
2010-04-26 21:03 ` jamal
2010-04-23 10:26 ` Eric Dumazet
2010-04-27 22:08 ` David Miller
2010-04-27 22:18 ` [PATCH net-next-2.6] bnx2x: Remove two prefetch() Eric Dumazet
2010-04-27 22:19 ` David Miller
2010-04-28 13:14 ` Eilon Greenstein
2010-04-28 15:44 ` Eliezer Tamir
2010-04-28 16:53 ` David Miller
[not found] ` <w2ue8f3c3211004280842r9f2589e8qb8fd4b7933cd9756@mail.gmail.com>
2010-04-28 16:55 ` David Miller
2010-04-28 11:33 ` jamal
2010-04-28 12:33 ` Eric Dumazet
2010-04-28 12:36 ` jamal
2010-04-28 14:06 ` [PATCH net-next-2.6] net: speedup udp receive path Eric Dumazet
2010-04-28 14:19 ` Eric Dumazet
2010-04-28 14:34 ` Eric Dumazet
2010-04-28 21:36 ` David Miller
2010-04-28 22:22 ` [PATCH net-next-2.6] net: ip_queue_rcv_skb() helper Eric Dumazet
2010-04-28 22:39 ` David Miller
2010-04-28 23:44 ` [PATCH net-next-2.6] net: speedup udp receive path jamal
2010-04-29 0:00 ` jamal
2010-04-29 4:09 ` Eric Dumazet
2010-04-29 11:35 ` jamal
2010-04-29 12:12 ` Changli Gao
2010-04-29 12:45 ` Eric Dumazet
2010-04-29 13:17 ` jamal
2010-04-29 13:21 ` Eric Dumazet
2010-04-29 13:37 ` jamal
2010-04-29 13:49 ` Eric Dumazet
2010-04-29 13:56 ` jamal
2010-04-29 20:36 ` jamal
2010-04-29 21:01 ` [PATCH net-next-2.6] net: sock_def_readable() and friends RCU conversion Eric Dumazet
2010-04-30 13:55 ` Brian Bloniarz
2010-04-30 17:26 ` Eric Dumazet
2010-04-30 23:35 ` David Miller
2010-05-01 4:56 ` Eric Dumazet
2010-05-01 7:02 ` Eric Dumazet
2010-05-01 8:03 ` Eric Dumazet
2010-05-01 22:00 ` David Miller
2010-04-30 19:30 ` [PATCH net-next-2.6] net: speedup udp receive path jamal
2010-04-30 20:40 ` Eric Dumazet
2010-05-01 0:06 ` jamal
2010-05-01 5:57 ` Eric Dumazet
2010-05-01 6:14 ` Eric Dumazet
2010-05-01 10:24 ` Changli Gao
2010-05-01 10:47 ` Eric Dumazet
2010-05-01 11:29 ` jamal
2010-05-01 11:23 ` jamal
2010-05-01 11:42 ` Eric Dumazet
2010-05-01 11:56 ` jamal
2010-05-01 13:22 ` Eric Dumazet
2010-05-01 13:49 ` jamal
2010-05-03 20:10 ` jamal
2010-04-29 23:07 ` Changli Gao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4BDB1E46.6050106@athenacr.com \
--to=bmb@athenacr$(echo .)com \
--cc=andi@firstfloor$(echo .)org \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=peterz@infradead$(echo .)org \
--cc=shemminger@vyatta$(echo .)com \
--cc=tglx@linutronix$(echo .)de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox