From: Eric Dumazet <eric.dumazet@gmail•com>
To: "Paweł Staszewski" <pstaszewski@itcare•pl>,
"netde >> Linux Kernel Network Developers"
<netdev@vger•kernel.org>
Subject: Re: kernel 4.15.0-rc9+ (net-next) high cpu load at 50Gbit/s - about 6Mpps
Date: Sun, 28 Jan 2018 10:59:35 -0800 [thread overview]
Message-ID: <1517165975.3715.85.camel@gmail.com> (raw)
In-Reply-To: <5891f536-f5cf-3e9a-40c7-d6765d038587@itcare.pl>
On Sun, 2018-01-28 at 19:26 +0100, Paweł Staszewski wrote:
>
> W dniu 27.01.2018 o 23:23, Paweł Staszewski pisze:
> > Hi
> >
> >
> > Today I made some real life traffic tests with kernel 4.15.0-rc9
> >
> > but when traffic reach 50Gbit/s and about 6Mpps cpou load rises fast
> > from 48% to 100% for all cpu cores.
> >
> > Here is some graph that presenting how cpu load rises when there was
> > more pps.
> >
> >
> > https://ibb.co/mhD5ob
> >
> >
> > here is perf record from that time:
> >
> > https://pastebin.com/3zqG1rvE
> >
> >
> > There is 8x 10G ixgbe 82599 interfaces teamed with teamd.
> >
> > No traffic queueing - only pfifo fast on all interfaces.
> >
> > No NAT or iptables forles other than INPUT (about 30rules)
> >
> > All nic's have same ethtool settings:
> >
> > ethtool -k eth0
> > Features for eth0:
> > Cannot get device udp-fragmentation-offload settings: Operation not
> > supported
> > rx-checksumming: on
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: on
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp-mangleid-segmentation: off
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: on
> > receive-hashing: on
> > highdma: on [fixed]
> > rx-vlan-filter: on
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: on
> > tx-gre-csum-segmentation: on
> > tx-ipxip4-segmentation: on
> > tx-ipxip6-segmentation: on
> > tx-udp_tnl-segmentation: on
> > tx-udp_tnl-csum-segmentation: on
> > tx-gso-partial: on
> > tx-sctp-segmentation: off [fixed]
> > tx-esp-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off
> > hw-tc-offload: off
> > esp-hw-offload: off [fixed]
> > esp-tx-csum-hw-offload: off [fixed]
> > rx-udp_tunnel-port-offload: on
> >
> >
> > ethtool -g eth0
> > Ring parameters for eth0:
> > Pre-set maximums:
> > RX: 4096
> > RX Mini: 0
> > RX Jumbo: 0
> > TX: 4096
> > Current hardware settings:
> > RX: 4096
> > RX Mini: 0
> > RX Jumbo: 0
> > TX: 2048
> >
> >
> > ethtool -c eth0
> > Coalesce parameters for eth0:
> > Adaptive RX: off TX: off
> > stats-block-usecs: 0
> > sample-interval: 0
> > pkt-rate-low: 0
> > pkt-rate-high: 0
> >
> > rx-usecs: 512
> > rx-frames: 0
> > rx-usecs-irq: 0
> > rx-frames-irq: 0
> >
> > tx-usecs: 0
> > tx-frames: 0
> > tx-usecs-irq: 0
> > tx-frames-irq: 0
> >
> > rx-usecs-low: 0
> > rx-frame-low: 0
> > tx-usecs-low: 0
> > tx-frame-low: 0
> >
> > rx-usecs-high: 0
> > rx-frame-high: 0
> > tx-usecs-high: 0
> > tx-frame-high: 0
> >
> >
> >
> >
>
>
>
> Peft top for kernel 4.15.0-rc9 below (all 40 cores 100% cpu load with
> 6.3Mpps)
>
> 20.96% [kernel] [k] queued_spin_lock_slowpath
> 5.51% [kernel] [k] ixgbe_poll
> 5.49% [kernel] [k] ixgbe_xmit_frame_ring
> 4.39% [kernel] [k] do_raw_spin_lock
> 4.29% [kernel] [k] sch_direct_xmit
> 4.11% [kernel] [k] fib_table_lookup
> 3.11% [team_mode_roundrobin] [k] rr_transmit
> 2.71% [kernel] [k] __dev_queue_xmit
> 2.62% [kernel] [k] __ptr_ring_peek
> 2.39% [kernel] [k] skb_release_data
> 2.18% [kernel] [k] dev_gro_receive
> 1.75% [kernel] [k] __qdisc_run
> 1.67% [kernel] [k] pfifo_fast_enqueue
> 1.57% [kernel] [k] netdev_pick_tx
> 1.56% [kernel] [k] page_frag_free
> 1.48% [kernel] [k] ip_finish_output2
> 1.38% [kernel] [k] __slab_free
> 1.36% [kernel] [k] skb_unref
> 1.34% [kernel] [k] ixgbe_maybe_stop_tx
> 1.30% [kernel] [k] vlan_do_receive
> 1.28% [kernel] [k] pfifo_fast_dequeue
> 1.23% [kernel] [k] virt_to_head_page
>
>
>
> Same configuration kernel 4.15.0-rc3 (50% cpu load on all 40 cores with
> 6.3Mpps)
>
> 7.81% [kernel] [k] ixgbe_xmit_frame_ring
> 7.61% [kernel] [k] ixgbe_poll
> 7.09% [kernel] [k] do_raw_spin_lock
> 5.63% [kernel] [k] fib_table_lookup
> 5.19% [kernel] [k] __dev_queue_xmit
> 4.38% [team_mode_roundrobin] [k] rr_transmit
> 3.10% [kernel] [k] netdev_pick_tx
> 2.79% [kernel] [k] skb_release_data
> 2.34% [kernel] [k] dev_gro_receive
> 1.99% [kernel] [k] page_frag_free
> 1.96% [kernel] [k] skb_unref
> 1.92% [kernel] [k] virt_to_head_page
> 1.90% [kernel] [k] ixgbe_maybe_stop_tx
> 1.82% [kernel] [k] vlan_do_receive
> 1.74% [kernel] [k] ip_finish_output2
> 1.73% [kernel] [k] __build_skb
> 1.68% [kernel] [k] __slab_free
> 1.67% [kernel] [k] __netif_receive_skb_core
> 1.60% [kernel] [k] inet_gro_receive
> 1.49% [kernel] [k] netif_skb_features
> 1.35% [kernel] [k] ip_rcv
> 1.33% [kernel] [k] ipt_do_table
> 1.30% [kernel] [k] compound_head
> 1.26% [kernel] [k] dev_hard_start_xmit
> 1.18% [kernel] [k] put_page
> 1.13% [kernel] [k] tcp_gro_receive
> 1.13% [kernel] [k] ip_forward
> 0.99% [kernel] [k] validate_xmit_skb
> 0.97% [kernel] [k] ip_route_input_rcu
> 0.88% [kernel] [k] inet_lookup_ifaddr_rcu
> 0.81% [kernel] [k] pfifo_fast_dequeue
> 0.77% [kernel] [k] vlan_dev_hard_start_xmit
> 0.64% [kernel] [k] ___slab_alloc
Please report :
1) "ethtool -l" information for your ethernet adapters.
2) IRQ configuration (grep eth /proc/interrupts)
3) tc -s qdisc show
next prev parent reply other threads:[~2018-01-28 18:59 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-27 22:23 kernel 4.15.0-rc9+ (net-next) high cpu load at 50Gbit/s - about 6Mpps Paweł Staszewski
2018-01-28 18:26 ` Paweł Staszewski
2018-01-28 18:59 ` Eric Dumazet [this message]
2018-01-28 21:40 ` Paweł Staszewski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1517165975.3715.85.camel@gmail.com \
--to=eric.dumazet@gmail$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=pstaszewski@itcare$(echo .)pl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox