From: "Paweł Staszewski" <pstaszewski@itcare•pl>
To: Changli Gao <xiaosuo@gmail•com>
Cc: Benny Amorsen <benny+usenet@amorsen•dk>,
zhigang gong <zhigang.gong@gmail•com>,
netdev@vger•kernel.org
Subject: Re: Strange packet drops with heavy firewalling
Date: Tue, 13 Apr 2010 14:33:42 +0200 [thread overview]
Message-ID: <4BC464A6.9000307@itcare.pl> (raw)
In-Reply-To: <u2y412e6f7f1004121618p6d6eff30q8a45a03faa59a912@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]
W dniu 2010-04-13 01:18, Changli Gao pisze:
> On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen<benny+usenet@amorsen•dk> wrote:
>
>> 99: 24 1306226 3 2 PCI-MSI-edge eth1-tx-0
>> 100: 15735 1648774 3 7 PCI-MSI-edge eth1-tx-1
>> 101: 8 11 9 1083022 PCI-MSI-edge eth1-tx-2
>> 102: 0 0 0 0 PCI-MSI-edge eth1-tx-3
>> 103: 18 15 6131 1095383 PCI-MSI-edge eth1-rx-0
>> 104: 217 32 46544 1335325 PCI-MSI-edge eth1-rx-1
>> 105: 154 1305595 218 16 PCI-MSI-edge eth1-rx-2
>> 106: 17 16 8229 1467509 PCI-MSI-edge eth1-rx-3
>> 107: 0 0 1 0 PCI-MSI-edge eth1
>> 108: 2 14 15 1003053 PCI-MSI-edge eth0-tx-0
>> 109: 8226 1668924 478 487 PCI-MSI-edge eth0-tx-1
>> 110: 3 1188874 17 12 PCI-MSI-edge eth0-tx-2
>> 111: 0 0 0 0 PCI-MSI-edge eth0-tx-3
>> 112: 203 185 5324 1015263 PCI-MSI-edge eth0-rx-0
>> 113: 4141 1600793 153 159 PCI-MSI-edge eth0-rx-1
>> 114: 16242 1210108 436 3124 PCI-MSI-edge eth0-rx-2
>> 115: 267 4173 19471 1321252 PCI-MSI-edge eth0-rx-3
>> 116: 0 1 0 0 PCI-MSI-edge eth0
>>
>>
>> irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
>> which to my mind should cause the same problem as before (where CPU1 and
>> CPU3 was handling all packets). Yet the box clearly works much better
>> than before.
>>
> irqbalanced? I don't think it can work properly. Try RPS in netdev and
> linux-next tree, and if cpu load isn't even, try this patch:
> http://patchwork.ozlabs.org/patch/49915/ .
>
>
>
Yes without irqbalance - and with irq affinity set by hand router will
work much better.
But I don't think that RPS will help him - I make some tests with RPS
and AFFINITY - results in attached file.
Test router make traffic management (hfsc) for almost 9k users
[-- Attachment #2: RPS_AFFINITY_TEST.txt --]
[-- Type: text/plain, Size: 5028 bytes --]
##############################################################################
eth0 -> CPU0
eth1 -> CPU5
RPS:
echo 00e0 > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo 000e > /sys/class/net/eth0/queues/rx-0/rps_cpus
------------------------------------------------------------------------------
PerfTop: 85205 irqs/sec kernel:97.1% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
214930.00 - 24.5% : _raw_spin_lock
63844.00 - 7.3% : u32_classify
48381.00 - 5.5% : e1000_clean
47754.00 - 5.5% : rb_next
37222.00 - 4.2% : e1000_intr_msi
26295.00 - 3.0% : hfsc_enqueue
17371.00 - 2.0% : rb_erase
15290.00 - 1.7% : _raw_spin_lock_irqsave
14958.00 - 1.7% : rb_insert_color
14439.00 - 1.6% : update_vf
14384.00 - 1.6% : e1000_xmit_frame
14356.00 - 1.6% : hfsc_dequeue
13804.00 - 1.6% : e1000_clean_tx_irq
13413.00 - 1.5% : ipt_do_table
9654.00 - 1.1% : ip_route_input
##############################################################################
eth0 -> CPU0
eth1 -> CPU5
NO RPS
------------------------------------------------------------------------------
PerfTop: 33800 irqs/sec kernel:96.9% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
19361.00 - 11.2% : e1000_clean
16424.00 - 9.5% : rb_next
13060.00 - 7.5% : e1000_intr_msi
7293.00 - 4.2% : u32_classify
6875.00 - 4.0% : ipt_do_table
5811.00 - 3.4% : _raw_spin_lock
5754.00 - 3.3% : e1000_xmit_frame
5671.00 - 3.3% : hfsc_dequeue
4503.00 - 2.6% : __alloc_skb
4156.00 - 2.4% : hfsc_enqueue
4090.00 - 2.4% : e1000_clean_tx_irq
3809.00 - 2.2% : e1000_clean_rx_irq
3424.00 - 2.0% : update_vf
3028.00 - 1.7% : rb_erase
2714.00 - 1.6% : ip_route_input
##############################################################################
eth0 -> CPU0,CPU1,CPU2,CPU4 -> affinity echo 0f > /proc/irq/30/smp_affinity
eth1 -> CPU5,CPU6,CPU7,CPU8 -> affinity echo f0 > /proc/irq/31/smp_affinity
NO RPS
------------------------------------------------------------------------------
PerfTop: 42362 irqs/sec kernel:96.0% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
33815.00 - 10.6% : rb_next
21357.00 - 6.7% : u32_classify
14525.00 - 4.6% : _raw_spin_lock
14346.00 - 4.5% : e1000_clean
12798.00 - 4.0% : hfsc_enqueue
10526.00 - 3.3% : ipt_do_table
9999.00 - 3.1% : hfsc_dequeue
9976.00 - 3.1% : e1000_intr_msi
9787.00 - 3.1% : rb_erase
8259.00 - 2.6% : e1000_xmit_frame
8015.00 - 2.5% : rb_insert_color
7948.00 - 2.5% : update_vf
6868.00 - 2.2% : e1000_clean_tx_irq
6822.00 - 2.1% : e1000_clean_rx_irq
6368.00 - 2.0% : __alloc_skb
##############################################################################
eth0 -> CPU0,CPU1,CPU2,CPU4 -> affinity echo 0f > /proc/irq/30/smp_affinity
eth1 -> CPU5,CPU6,CPU7,CPU8 -> affinity echo f0 > /proc/irq/31/smp_affinity
RPS:
echo 0f > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo f0 > /sys/class/net/eth1/queues/rx-0/rps_cpus
------------------------------------------------------------------------------
PerfTop: 81051 irqs/sec kernel:96.9% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
167110.00 - 22.3% : _raw_spin_lock
58221.00 - 7.8% : u32_classify
46379.00 - 6.2% : rb_next
35189.00 - 4.7% : e1000_clean
25614.00 - 3.4% : e1000_intr_msi
24094.00 - 3.2% : hfsc_enqueue
16231.00 - 2.2% : rb_erase
14298.00 - 1.9% : rb_insert_color
13751.00 - 1.8% : update_vf
13712.00 - 1.8% : ipt_do_table
13588.00 - 1.8% : hfsc_dequeue
13335.00 - 1.8% : e1000_xmit_frame
12449.00 - 1.7% : e1000_clean_tx_irq
11510.00 - 1.5% : net_tx_action
11428.00 - 1.5% : _raw_spin_lock_irqsave
next prev parent reply other threads:[~2010-04-13 12:33 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-04-09 9:56 Strange packet drops with heavy firewalling Benny Amorsen
2010-04-09 11:47 ` Eric Dumazet
2010-04-09 12:33 ` Benny Amorsen
2010-04-09 13:29 ` Eric Dumazet
2010-04-12 6:20 ` Benny Amorsen
[not found] ` <q2v40c9f5b21004120116p766df82dj88c6af4e4cad55f@mail.gmail.com>
2010-04-12 14:44 ` Benny Lyne Amorsen
[not found] ` <p2x40c9f5b21004120833jd7a749cak6ea69cebd28f8352@mail.gmail.com>
2010-04-12 17:06 ` Benny Amorsen
2010-04-12 23:18 ` Changli Gao
2010-04-13 5:56 ` Eric Dumazet
2010-04-13 7:56 ` Benny Amorsen
2010-04-15 13:23 ` Benny Amorsen
2010-04-15 13:42 ` Eric Dumazet
2010-04-13 12:33 ` Paweł Staszewski [this message]
2010-04-13 12:53 ` Eric Dumazet
2010-04-13 13:39 ` Paweł Staszewski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4BC464A6.9000307@itcare.pl \
--to=pstaszewski@itcare$(echo .)pl \
--cc=benny+usenet@amorsen$(echo .)dk \
--cc=netdev@vger$(echo .)kernel.org \
--cc=xiaosuo@gmail$(echo .)com \
--cc=zhigang.gong@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox