From: Jesper Dangaard Brouer <brouer@redhat•com>
To: Eric Dumazet <eric.dumazet@gmail•com>
Cc: Rick Jones <rick.jones2@hpe•com>,
"netdev@vger•kernel.org\" <netdev@vger•kernel.org>,
brouer@redhat.com"@redhat.com
Subject: Re: Netperf UDP issue with connected sockets
Date: Thu, 17 Nov 2016 09:16:38 +0100 [thread overview]
Message-ID: <20161117091638.5fab8494@redhat.com> (raw)
In-Reply-To: <1479342849.8455.233.camel@edumazet-glaptop3.roam.corp.google.com>
On Wed, 16 Nov 2016 16:34:09 -0800
Eric Dumazet <eric.dumazet@gmail•com> wrote:
> On Wed, 2016-11-16 at 23:40 +0100, Jesper Dangaard Brouer wrote:
>
> > Using -R 1 does not seem to help remove __ip_select_ident()
> >
> > Samples: 56K of event 'cycles', Event count (approx.): 78628132661
> > Overhead Command Shared Object Symbol
> > + 9.11% netperf [kernel.vmlinux] [k] __ip_select_ident
> > + 6.98% netperf [kernel.vmlinux] [k] _raw_spin_lock
> > + 6.21% swapper [mlx5_core] [k] mlx5e_poll_tx_cq
> > + 5.03% netperf [kernel.vmlinux] [k] copy_user_enhanced_fast_string
> > + 4.69% netperf [kernel.vmlinux] [k] __ip_make_skb
> > + 4.63% netperf [kernel.vmlinux] [k] skb_set_owner_w
> > + 4.15% swapper [kernel.vmlinux] [k] __slab_free
> > + 3.80% netperf [mlx5_core] [k] mlx5e_sq_xmit
> > + 2.00% swapper [kernel.vmlinux] [k] sock_wfree
> > + 1.94% netperf netperf [.] send_data
> > + 1.92% netperf netperf [.] send_omni_inner
>
> Check "ss -nu" ?
>
> You will see if sockets are connected (present in ss output or not)
Tested different versions of netperf, commands used below signature:
netperf-2.6.0: connected "broken"
netperf-2.7.0: connected works
SVN-r709 : connected works
I noticed there is a Send-Q, and the perf-top2 is _raw_spin_lock, which
looks like it comes from __dev_queue_xmit(), but we know from
experience that this stall is actually caused by writing the
tailptr/doorbell in the HW. Thus, this could benefit a lot from
bulk/xmit_more into the qdisc layer.
> UDP being connected does not prevent __ip_select_ident() being used.
>
> if ((iph->frag_off & htons(IP_DF)) && !skb->ignore_df) {
>
> So you need IP_DF being set, and skb->ignore_df being 0
Thanks for explaining that! :-)
http://lxr.free-electrons.com/source/include/net/ip.h?v=4.8#L332
http://lxr.free-electrons.com/source/net/ipv4/ip_output.c?v=4.8#L449
Netperf UDP_STREAM default send 64K packets that get fragmented...
which actually is very unfortunate because people end-up testing a code
path in the kernel they didn't expect. That is why I use the
option "-- -m 1472".
> time to try IP_MTU_DISCOVER ;)
To Rick, maybe you can find a good solution or option with Eric's hint,
to send appropriate sized UDP packets with Don't Fragment (DF).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
Testing with ss -nua
$ /usr/local/stow/netperf-2.6.0-demo/bin/netperf -H 198.18.50.1 -t UDP_STREAM -l 3 -- -m 1472 -n -N > /dev/null & sleep 1; ss -una
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 11520 *:54589 *:*
$ /usr/local/stow/netperf-2.7.0-demo/bin/netperf -H 198.18.50.1 -t UDP_STREAM -l 3 -- -m 1472 -n -N > /dev/null & sleep 1; ss -una
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 18432 198.18.50.3:46803 198.18.50.1:51851
$ ~/tools/netperf2-svn/src/netperf -H 198.18.50.1 -t UDP_STREAM -l 3 -- -m 1472 -n -N > /dev/null & sleep 1; ss -una
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 43776 198.18.50.3:42965 198.18.50.1:51948
next prev parent reply other threads:[~2016-11-17 8:16 UTC|newest]
Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-03 14:59 High perf top ip_idents_reserve doing netperf UDP_STREAM Jesper Dangaard Brouer
2014-09-03 15:17 ` Eric Dumazet
2016-11-16 12:16 ` Netperf UDP issue with connected sockets Jesper Dangaard Brouer
2016-11-16 17:46 ` Rick Jones
2016-11-16 22:40 ` Jesper Dangaard Brouer
2016-11-16 22:50 ` Rick Jones
2016-11-17 0:34 ` Eric Dumazet
2016-11-17 8:16 ` Jesper Dangaard Brouer [this message]
2016-11-17 13:20 ` Eric Dumazet
2016-11-17 13:42 ` Jesper Dangaard Brouer
2016-11-17 14:17 ` Eric Dumazet
2016-11-17 14:57 ` Jesper Dangaard Brouer
2016-11-17 16:21 ` Eric Dumazet
2016-11-17 18:30 ` Jesper Dangaard Brouer
2016-11-17 18:51 ` Eric Dumazet
2016-11-17 21:19 ` Jesper Dangaard Brouer
2016-11-17 21:44 ` Eric Dumazet
2016-11-17 23:08 ` Rick Jones
2016-11-18 0:37 ` Julian Anastasov
2016-11-18 0:42 ` Rick Jones
2016-11-18 17:12 ` Jesper Dangaard Brouer
2016-11-21 16:03 ` Jesper Dangaard Brouer
2016-11-21 18:10 ` Eric Dumazet
2016-11-29 6:58 ` [WIP] net+mlx4: auto doorbell Eric Dumazet
2016-11-30 11:38 ` Jesper Dangaard Brouer
2016-11-30 15:56 ` Eric Dumazet
2016-11-30 19:17 ` Jesper Dangaard Brouer
2016-11-30 19:30 ` Eric Dumazet
2016-11-30 22:30 ` Jesper Dangaard Brouer
2016-11-30 22:40 ` Eric Dumazet
2016-12-01 0:27 ` Eric Dumazet
2016-12-01 1:16 ` Tom Herbert
2016-12-01 2:32 ` Eric Dumazet
2016-12-01 2:50 ` Eric Dumazet
2016-12-02 18:16 ` Eric Dumazet
2016-12-01 5:03 ` Tom Herbert
2016-12-01 19:24 ` Willem de Bruijn
2016-11-30 13:50 ` Saeed Mahameed
2016-11-30 15:44 ` Eric Dumazet
2016-11-30 16:27 ` Saeed Mahameed
2016-11-30 17:28 ` Eric Dumazet
2016-12-01 12:05 ` Jesper Dangaard Brouer
2016-12-01 14:24 ` Eric Dumazet
2016-12-01 16:04 ` Jesper Dangaard Brouer
2016-12-01 17:04 ` Eric Dumazet
2016-12-01 19:17 ` Jesper Dangaard Brouer
2016-12-01 20:11 ` Eric Dumazet
2016-12-01 20:20 ` David Miller
2016-12-01 22:10 ` Eric Dumazet
2016-12-02 14:23 ` Eric Dumazet
2016-12-01 21:32 ` Alexander Duyck
2016-12-01 22:04 ` Eric Dumazet
2016-11-17 17:34 ` Netperf UDP issue with connected sockets David Laight
2016-11-17 22:39 ` Alexander Duyck
2016-11-17 17:42 ` Rick Jones
2016-11-28 18:33 ` Rick Jones
2016-11-28 18:40 ` Rick Jones
2016-11-30 10:43 ` Jesper Dangaard Brouer
2016-11-30 17:42 ` Rick Jones
2016-11-30 18:11 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161117091638.5fab8494@redhat.com \
--to=brouer@redhat$(echo .)com \
--cc="netdev@vger$(echo .)kernel.org\" <netdev@vger$(echo .)kernel.org>, brouer@redhat.com"@redhat.com \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=rick.jones2@hpe$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox