Subject: Re: Peak TCP performance

From: Rick Jones <[email-masked]>
Date: Fri, 11 Oct 2013 10:46:02 -0700

On 10/10/2013 09:21 PM, Kyle Hubert wrote:
> On Thu, Oct 10, 2013 at 11:44 PM, Eric Dumazet <[email-masked]> wrote:
>>> Also, my copy of ethtool does not recognize tx-nocache-copy. However,
>>> I do have control over the net device. Is there something there I can
>>> set, or is tx-nocache-copy also a new feature? I'll start digging.
>>>
>>
>> nocache-copy was added in 3.0, but I do find its not a gain for current
>> cpus.
>>
>> You could get a fresh copy of ethtool sources :
>>
>> git clone git://git.kernel.org/pub/scm/network/ethtool/ethtool.git
>> cd ethtool
>> ./autogen.sh  ...
>
> That did the trick. Thanks for the help! Is there somewhere I can read
> up on this feature? A lot of the netdev features are opaque to me.
> Also, I can set NETIF_F_NOCACHE_COPY in the netdev->features to set
> this by default, yes?
>
> This at least mirrors the performance improvement that I see when
> forwarding, however I still see reserved CPU time. Is there a way to
> push it even farther?

Thought I would point-out that unless you do concrete steps to make it 
behave otherwise, netperf will constantly present the same set of 
cache-clean buffers to the transport.  The size of those buffers will be 
determined by some heuristics and will depend on the socket buffer size 
at the time the data socket is create, which itself will depend on 
whether or not you have used a test-specific -s option.  And the 
test-specific -m option will come into play.

If I am recalling correctly, the number of buffers will be one more than:

initialSO_SNDBUF / send_size

though you can control that with global -W option.

happy benchmarking,

rick jones