From: Matthew Rosato <mjrosato@linux•vnet.ibm.com>
To: Wei Xu <wexu@redhat•com>
Cc: Jason Wang <jasowang@redhat•com>,
netdev@vger•kernel.org, davem@davemloft•net, mst@redhat•com
Subject: Re: Regression in throughput between kvm guests over virtual bridge
Date: Wed, 25 Oct 2017 16:31:55 -0400 [thread overview]
Message-ID: <4bb941da-059c-3c3c-26bf-5f41598d2fa6@linux.vnet.ibm.com> (raw)
In-Reply-To: <20171023135729.xeacprxsg5qizkoa@Wei-Dev>
On 10/23/2017 09:57 AM, Wei Xu wrote:
> On Wed, Oct 18, 2017 at 04:17:51PM -0400, Matthew Rosato wrote:
>> On 10/12/2017 02:31 PM, Wei Xu wrote:
>>> On Thu, Oct 05, 2017 at 04:07:45PM -0400, Matthew Rosato wrote:
>>>>
>>>> Ping... Jason, any other ideas or suggestions?
>>>
>>> Hi Matthew,
>>> Recently I am doing similar test on x86 for this patch, here are some,
>>> differences between our testbeds.
>>>
>>> 1. It is nice you have got improvement with 50+ instances(or connections here?)
>>> which would be quite helpful to address the issue, also you've figured out the
>>> cost(wait/wakeup), kindly reminder did you pin uperf client/server along the whole
>>> path besides vhost and vcpu threads?
>>
>> Was not previously doing any pinning whatsoever, just reproducing an
>> environment that one of our testers here was running. Reducing guest
>> vcpu count from 4->1, still see the regression. Then, pinned each vcpu
>> thread and vhost thread to a separate host CPU -- still made no
>> difference (regression still present).
>>
>>>
>>> 2. It might be useful to short the traffic path as a reference, What I am running
>>> is briefly like:
>>> pktgen(host kernel) -> tap(x) -> guest(DPDK testpmd)
>>>
>>> The bridge driver(br_forward(), etc) might impact performance due to my personal
>>> experience, so eventually I settled down with this simplified testbed which fully
>>> isolates the traffic from both userspace and host kernel stack(1 and 50 instances,
>>> bridge driver, etc), therefore reduces potential interferences.
>>>
>>> The down side of this is that it needs DPDK support in guest, has this ever be
>>> run on s390x guest? An alternative approach is to directly run XDP drop on
>>> virtio-net nic in guest, while this requires compiling XDP inside guest which needs
>>> a newer distro(Fedora 25+ in my case or Ubuntu 16.10, not sure).
>>>
>>
>> I made an attempt at DPDK, but it has not been run on s390x as far as
>> I'm aware and didn't seem trivial to get working.
>>
>> So instead I took your alternate suggestion & did:
>> pktgen(host) -> tap(x) -> guest(xdp_drop)
>
> It is really nice of you for having tried this, I also tried this on x86 with
> two ubuntu 16.04 guests, but unfortunately I couldn't reproduce it as well,
> but I did get lower throughput with 50 instances than one instance(1-4 vcpus),
> is this the same on s390x?
For me, the total throughput is higher from 50 instances than for 1
instance when host kernel is 4.13. However, when running a 50 instance
uperf load I cannot reproduce the regression, either. Throughput is a
little bit better when host is 4.13 vs 4.12 for a 50 instance run.
>
>>
>> When running this setup, I am not able to reproduce the regression. As
>> mentioned previously, I am also unable to reproduce when running one end
>> of the uperf connection from the host - I have only ever been able to
>> reproduce when both ends of the uperf connection are running within a guest.
>
> Did you see improvement when running uperf from the host if no regression?
>
> It would be pretty nice to run pktgen from the VM as Jason suggested in another
> mail(pktgen(vm1) -> tap1 -> bridge -> tap2 -> vm2), this is super close to your
> original test case and can help to determine if we can get some clue with tcp or
> bridge driver.
>
> Also I am interested in your hardware platform, how many NUMA nodes do you have?
> what about your binding(vcpu/vhost/pktgen). For my case, I got a server with 4
> NUMA nodes and 12 cpus for each sockets, and I am explicitly launching qemu from
> cpu0, then bind vhost(Rx/Tx) to cpu 2&3, and vcpus start from cpu 4(3 vcpus for
> each).
I'm running in an LPAR on a z13. The particular LPAR I am using to
reproduce has 20 CPUs and 40G of memory assigned, all in 1 NUMA node. I
was initially recreating an issue uncovered by someone elses test, and
thus was doing no cpu binding -- But have attempted binding vhost and
vcpu threads to individual host CPUs and it seemed to have no impact on
the noted regression. When doing said binding, I did: qemu-guestA ->
cpu0(or 0-3 when running 4vcpu), qemu-guestA-vhost -> cpu4, qemu-guestB
-> cpu8(or 8-11 when running 4vcpu), qemu-guestB-vhost -> cpu12.
prev parent reply other threads:[~2017-10-25 20:32 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-12 17:56 Regression in throughput between kvm guests over virtual bridge Matthew Rosato
2017-09-13 1:16 ` Jason Wang
2017-09-13 8:13 ` Jason Wang
2017-09-13 16:59 ` Matthew Rosato
2017-09-14 4:21 ` Jason Wang
2017-09-15 3:36 ` Matthew Rosato
2017-09-15 8:55 ` Jason Wang
2017-09-15 19:19 ` Matthew Rosato
2017-09-18 3:13 ` Jason Wang
2017-09-18 4:14 ` [PATCH] vhost_net: conditionally enable tx polling kbuild test robot
2017-09-18 7:36 ` Regression in throughput between kvm guests over virtual bridge Jason Wang
2017-09-18 18:11 ` Matthew Rosato
2017-09-20 6:27 ` Jason Wang
2017-09-20 19:38 ` Matthew Rosato
2017-09-22 4:03 ` Jason Wang
2017-09-25 20:18 ` Matthew Rosato
2017-10-05 20:07 ` Matthew Rosato
2017-10-11 2:41 ` Jason Wang
2017-10-12 18:31 ` Wei Xu
2017-10-18 20:17 ` Matthew Rosato
2017-10-23 2:06 ` Jason Wang
2017-10-23 2:13 ` Michael S. Tsirkin
2017-10-25 20:21 ` Matthew Rosato
2017-10-26 9:44 ` Wei Xu
2017-10-26 17:53 ` Matthew Rosato
2017-10-31 7:07 ` Wei Xu
2017-10-31 7:00 ` Jason Wang
2017-11-03 4:30 ` Matthew Rosato
2017-11-04 23:35 ` Wei Xu
2017-11-08 1:02 ` Matthew Rosato
2017-11-11 20:59 ` Matthew Rosato
2017-11-12 18:34 ` Wei Xu
2017-11-14 20:11 ` Matthew Rosato
2017-11-20 19:25 ` Matthew Rosato
2017-11-27 16:21 ` Wei Xu
2017-11-28 1:36 ` Jason Wang
2017-11-28 2:44 ` Matthew Rosato
2017-11-28 18:00 ` Wei Xu
2017-11-28 3:51 ` Wei Xu
2017-11-12 15:40 ` Wei Xu
2017-10-23 13:57 ` Wei Xu
2017-10-25 20:31 ` Matthew Rosato [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4bb941da-059c-3c3c-26bf-5f41598d2fa6@linux.vnet.ibm.com \
--to=mjrosato@linux$(echo .)vnet.ibm.com \
--cc=davem@davemloft$(echo .)net \
--cc=jasowang@redhat$(echo .)com \
--cc=mst@redhat$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=wexu@redhat$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox