From: Eric Dumazet <dada1@cosmosbay•com>
To: Simon Kirby <sim@netnation•com>
Cc: Robert Olsson <Robert.Olsson@data•slu.se>, netdev@oss•sgi.com
Subject: Re: Route cache performance
Date: Tue, 16 Aug 2005 04:23:35 +0200 [thread overview]
Message-ID: <43014E27.1070104@cosmosbay.com> (raw)
In-Reply-To: <20050815213855.GA17832@netnation.com>
Simon Kirby a écrit :
> Hi!
>
> Well, after a few years of other work :), I have finally got around to
> setting up some more permanent forwarding / route cache performance
> test boxes. I noticed the route trie option in the newer 2.6 kernels
> and figured it would be a good time to revisit things.
>
> Test setup:
>
> [Xeon w/e1000]---[Opteron w/dual e1000]---[Xeon w/e1000]
>
> The Xeons are 2.4 GHz boxes and the Opteron is a 140. At some point
> I intend to compare the performance of a 32 bit versus 64 bit kernel.
>
> I'm only able to get pktgen to spit out about 660 kpps from the test 2.4
> Xeon box with onboard e1000 (pause disabled), but already I notice some
> disappointing results. The old 2.4.27 kernel I last did tests with seems
> to do a much better job of forwarding small packets (static src/dst) than
> 2.4.31 and 2.6.12.
>
> On the (leftmost) sending box, 2.4.27, 2.4.31, and 2.6.12 all seem to do
> fairly well at transmission with pktgen. The 2.6 pktgen seems a little
> better (no transmission errors and a few more Mbps), so I've been using
> 2.6.12. With fixed dst packets and pause disabled via ethtool, about
> 660 kpps is sent continuously. juno (spoofed source, userspace) seems to
> do about 360 kpps. The routes and packets are set up to route through
> the Opteron box to the receiving (rightmost) box.
>
> I've noticed that e1000 changes integrated in 2.6.11-bk2 are resulting in
> the forwarding test box slowing down enough that it seems to be exposing
> "dst cache overflow", even though under slightly less load the gc seems
> to be able to keep up. Robert, if I read correctly it seems that the
> e1000 NAPI changes were some fixes you submitted?
>
> Something appears to be different in the rtcache GC or perhaps NAPI or
> some other interaction, because firing juno at 2.4 does not show any
> problems while I can't seem to get 2.6.12 to _not_ print "dst cache
> overflow". 2.6.11 (pre-bk2) seems a little better at start, but any kind
> of burst seems to make the route cache entries exceed gc and then the
> slower hash lookups seem to make it get stuck at max_size (and printing
> "dst cache overflow") until the attack stops, even with gc_min_interval
> set to 0 (really 0).
>
> Anyway, I'm still in early testing stages here but it seems it's still as
> easy as ever to destroy routers (and hosts?) with a fairly small stream
> of small packets which create new rtcache entries. These days, 184 Mbps
> is starting to fall under the "small" DoS attack category, too.
>
> I notice the hash table size is still only 4096 buckets for 512 MB, which
> isn't that wonderful when it hits a max_size of 65536 (w/512 MB)...
>
> Simon-
>
>
Hi Simon
I think one of the reason linux 2.6 has worst results is because HZ=1000 (instead of HZ=100 for linux 2.4)
So if rt_garbage_collect() has heavy work to do, it usually break out of the loop because of :
} while (!in_softirq() && time_before_eq(jiffies, now));
Could you please test latest 2.6.13-rc6 kernel on the Opteron machine, compiled with HZ=100, with the appended kernel argument :
rhash_entries=8191 ( or rhash_entries=16383 )
and
echo 1 >/proc/sys/net/ipv4/route/gc_interval
echo 2 >/proc/sys/net/ipv4/route/gc_elasticity
Could you also post some data from your router (like : rtstat -c 20 -i 1)
Eric
next prev parent reply other threads:[~2005-08-16 2:23 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2005-08-15 21:38 Route cache performance Simon Kirby
2005-08-16 2:23 ` Eric Dumazet [this message]
2005-08-23 19:08 ` Simon Kirby
2005-08-23 19:56 ` Robert Olsson
2005-08-24 0:01 ` Simon Kirby
2005-08-24 3:50 ` Robert Olsson
2005-08-25 18:11 ` Simon Kirby
2005-08-25 20:05 ` Alexey Kuznetsov
2005-08-25 21:22 ` Simon Kirby
2005-08-26 11:55 ` Alexey Kuznetsov
2005-08-26 19:49 ` Robert Olsson
2005-09-06 23:57 ` Simon Kirby
2005-09-07 1:19 ` Alexey Kuznetsov
2005-09-07 15:03 ` Robert Olsson
2005-09-07 16:55 ` Simon Kirby
2005-09-07 17:21 ` Robert Olsson
2005-09-07 14:45 ` Robert Olsson
2005-09-07 16:28 ` Simon Kirby
2005-09-07 16:49 ` Robert Olsson
2005-09-07 16:57 ` Simon Kirby
2005-09-07 19:59 ` Alexey Kuznetsov
2005-09-13 22:14 ` Simon Kirby
2005-09-14 8:04 ` Robert Olsson
2005-09-17 0:28 ` Simon Kirby
2005-09-17 9:04 ` Martin Josefsson
2005-09-17 15:17 ` jamal
2005-09-15 21:04 ` Alexey Kuznetsov
2005-09-15 21:30 ` Robert Olsson
2005-09-15 22:21 ` Alexey Kuznetsov
2005-09-16 12:18 ` Robert Olsson
2005-09-16 19:04 ` Alexey Kuznetsov
2005-09-16 19:22 ` Ben Greear
2005-09-16 19:57 ` Robert Olsson
-- strict thread matches above, loose matches on Subject: below --
2005-08-24 16:06 Simon Kirby
[not found] <20050301220743.GF2554@netnation.com>
[not found] ` <16940.9990.975632.115834@robur.slu.se>
2005-03-09 1:45 ` Simon Kirby
2005-03-09 12:05 ` Robert Olsson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=43014E27.1070104@cosmosbay.com \
--to=dada1@cosmosbay$(echo .)com \
--cc=Robert.Olsson@data$(echo .)slu.se \
--cc=netdev@oss$(echo .)sgi.com \
--cc=sim@netnation$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox