From: Jesper Dangaard Brouer <hawk@comx•dk>
To: Stephen Hemminger <shemminger@vyatta•com>,
netfilter-devel <netfilter-devel@vger•kernel.org>
Cc: netdev <netdev@vger•kernel.org>
Subject: Possible regression: Packet drops during iptables calls
Date: Tue, 14 Dec 2010 15:46:14 +0100 [thread overview]
Message-ID: <1292337974.9155.68.camel@firesoul.comx.local> (raw)
I'm experiencing RX packet drops during call to iptables, on my
production servers.
Further investigations showed, that its only the CPU executing the
iptables command that experience packet drops!? Thus, a quick fix was
to force the iptables command to run on one of the idle CPUs (This can
be achieved with the "taskset" command).
I have a 2x Xeon 5550 CPU system, thus 16 CPUs (with HT enabled). We
only use 8 CPUs due to a multiqueue limitation of 8 queues in the
1Gbit/s NICs (82576 chips). CPUs 0 to 7 is assigned for packet
processing via smp_affinity.
Can someone explain why the packet drops only occur on the CPU
executing the iptables command?
What can we do to solve this issue?
I should note that I have a very large ruleset on this machine, and
the production machine is routing around 800 Mbit/s, in each
direction. The issue occurs on a simple iptables rule listing.
I think (untested) the problem is related to kernel git commit:
commit 942e4a2bd680c606af0211e64eb216be2e19bf61
Author: Stephen Hemminger <shemminger@vyatta•com>
Date: Tue Apr 28 22:36:33 2009 -0700
netfilter: revised locking for x_tables
The x_tables are organized with a table structure and a per-cpu copies
of the counters and rules. On older kernels there was a reader/writer
lock per table which was a performance bottleneck. In 2.6.30-rc, this
was converted to use RCU and the counters/rules which solved the performance
problems for do_table but made replacing rules much slower because of
the necessary RCU grace period.
This version uses a per-cpu set of spinlocks and counters to allow to
table processing to proceed without the cache thrashing of a global
reader lock and keeps the same performance for table updates.
Signed-off-by: Stephen Hemminger <shemminger@vyatta•com>
Acked-by: Linus Torvalds <torvalds@linux-foundation•org>
Signed-off-by: David S. Miller <davem@davemloft•net>
--
Med venlig hilsen / Best regards
Jesper Brouer
ComX Networks A/S
Linux Network Kernel Developer
Cand. Scient Datalog / MSc.CS
Author of http://adsl-optimizer.dk
LinkedIn: http://www.linkedin.com/in/brouer
next reply other threads:[~2010-12-14 14:52 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-12-14 14:46 Jesper Dangaard Brouer [this message]
2010-12-14 15:31 ` Possible regression: Packet drops during iptables calls Eric Dumazet
2010-12-14 16:09 ` Jesper Dangaard Brouer
2010-12-14 16:24 ` Eric Dumazet
2010-12-16 14:04 ` Jesper Dangaard Brouer
2010-12-16 14:12 ` Eric Dumazet
2010-12-16 14:24 ` Jesper Dangaard Brouer
2010-12-16 14:29 ` Eric Dumazet
2010-12-16 15:02 ` Eric Dumazet
2010-12-16 16:07 ` [PATCH net-next-2.6] netfilter: ip_tables: dont block BH while reading counters Eric Dumazet
2010-12-16 16:53 ` [PATCH v2 " Eric Dumazet
2010-12-16 17:31 ` Stephen Hemminger
2010-12-16 17:53 ` [PATCH v3 net-next-2.6] netfilter: x_tables: " Eric Dumazet
2010-12-16 17:57 ` Stephen Hemminger
2010-12-16 19:58 ` Eric Dumazet
2010-12-16 20:12 ` Stephen Hemminger
2010-12-16 20:40 ` Eric Dumazet
2010-12-16 17:57 ` Stephen Hemminger
2010-12-18 4:29 ` [PATCH v4 " Eric Dumazet
2010-12-20 13:42 ` Jesper Dangaard Brouer
2010-12-20 14:45 ` Eric Dumazet
2010-12-21 16:48 ` Jesper Dangaard Brouer
2011-01-08 16:45 ` Eric Dumazet
2011-01-09 21:31 ` Pablo Neira Ayuso
2010-12-16 14:13 ` Possible regression: Packet drops during iptables calls Eric Dumazet
2010-12-16 14:20 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1292337974.9155.68.camel@firesoul.comx.local \
--to=hawk@comx$(echo .)dk \
--cc=netdev@vger$(echo .)kernel.org \
--cc=netfilter-devel@vger$(echo .)kernel.org \
--cc=shemminger@vyatta$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox