From: Bill Fink <billfink@mindspring•com>
To: Rick Jones <rick.jones2@hp•com>
Cc: jagana@us•ibm.com, herbert@gondor•apana.org.au,
gaagaan@gmail•com, Robert.Olsson@data•slu.se,
mcarlson@broadcom•com, rdreier@cisco•com,
peter.p.waskiewicz.jr@intel•com, hadi@cyberus•ca,
kaber@trash•net, jeff@garzik•org, general@lists•openfabrics.org,
mchan@broadcom•com, tgraf@suug•ch, netdev@vger•kernel.org,
johnpol@2ka•mipt.ru, shemminger@linux-foundation•org,
David Miller <davem@davemloft•net>,
sri@us•ibm.com
Subject: [ofa-general] Re: [PATCH 0/9 Rev3] Implement batching skb API and support in IPoIB
Date: Thu, 23 Aug 2007 23:18:20 -0400 [thread overview]
Message-ID: <20070823231820.2ae52cc0.billfink@mindspring.com> (raw)
In-Reply-To: <46CE0BA1.60206@hp.com>
On Thu, 23 Aug 2007, Rick Jones wrote:
> jamal wrote:
> > [TSO already passed - iirc, it has been
> > demostranted to really not add much to throughput (cant improve much
> > over closeness to wire speed) but improve CPU utilization].
>
> In the one gig space sure, but in the 10 Gig space, TSO on/off does make a
> difference for throughput.
Not too much.
TSO enabled:
[root@lang2 ~]# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
[root@lang2 ~]# nuttcp -w10m 192.168.88.16
11813.4375 MB / 10.00 sec = 9906.1644 Mbps 99 %TX 80 %RX
TSO disabled:
[root@lang2 ~]# ethtool -K eth2 tso off
[root@lang2 ~]# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
[root@lang2 ~]# nuttcp -w10m 192.168.88.16
11818.2500 MB / 10.00 sec = 9910.0176 Mbps 100 %TX 78 %RX
Pretty negligible difference it seems.
This is with a 2.6.20.7 kernel, Myricom 10-GigE NICs, and 9000 byte
jumbo frames, in a LAN environment.
For grins, I also did a couple of tests with an MSS of 1460 to
emulate a standard 1500 byte Ethernet MTU.
TSO enabled:
[root@lang2 ~]# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
[root@lang2 ~]# nuttcp -M1460 -w10m 192.168.88.16
5102.8503 MB / 10.06 sec = 4253.9124 Mbps 39 %TX 99 %RX
TSO disabled:
[root@lang2 ~]# ethtool -K eth2 tso off
[root@lang2 ~]# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
[root@lang2 ~]# nuttcp -M1460 -w10m 192.168.88.16
5399.5625 MB / 10.00 sec = 4527.9070 Mbps 99 %TX 76 %RX
Here you can see there is a major difference in the TX CPU utilization
(99 % with TSO disabled versus only 39 % with TSO enabled), although
the TSO disabled case was able to squeeze out a little extra performance
from its extra CPU utilization. Interestingly, with TSO enabled, the
receiver actually consumed more CPU than with TSO disabled, so I guess
the receiver CPU saturation in that case (99 %) was what restricted
its performance somewhat (this was consistent across a few test runs).
-Bill
next prev parent reply other threads:[~2007-08-24 3:18 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-08-17 6:06 [PATCH 0/9 Rev3] Implement batching skb API and support in IPoIB Krishna Kumar2
2007-08-21 7:18 ` David Miller
2007-08-21 12:30 ` [ofa-general] " jamal
2007-08-21 18:51 ` David Miller
2007-08-21 21:09 ` jamal
2007-08-21 22:50 ` David Miller
2007-08-22 4:11 ` [ofa-general] " Krishna Kumar2
2007-08-22 4:22 ` David Miller
2007-08-22 7:03 ` Krishna Kumar2
2007-08-22 9:14 ` David Miller
2007-08-23 2:43 ` Krishna Kumar2
2007-08-22 17:09 ` [ofa-general] " Rick Jones
2007-08-22 20:21 ` David Miller
2007-08-23 22:04 ` [ofa-general] " jamal
2007-08-23 22:25 ` jamal
2007-08-23 22:35 ` [ofa-general] " Rick Jones
2007-08-23 22:41 ` jamal
2007-08-24 3:18 ` Bill Fink [this message]
2007-08-24 12:14 ` jamal
2007-08-24 18:08 ` Bill Fink
2007-08-24 21:25 ` David Miller
2007-08-24 23:11 ` Herbert Xu
2007-08-25 23:45 ` Bill Fink
2007-08-24 18:46 ` [ofa-general] " Rick Jones
2007-08-25 0:42 ` John Heffner
2007-08-26 8:41 ` [ofa-general] " Bill Fink
2007-08-27 1:32 ` John Heffner
2007-08-27 2:04 ` David Miller
2007-08-27 23:23 ` jamal
2007-09-14 7:20 ` [ofa-general] " Bill Fink
2007-09-14 13:44 ` TSO, TCP Cong control etc jamal
2007-09-14 17:24 ` [PATCH 0/9 Rev3] Implement batching skb API and support in IPoIB David Miller
2007-08-23 22:30 ` David Miller
2007-08-23 22:38 ` [ofa-general] " jamal
2007-08-24 3:34 ` Stephen Hemminger
2007-08-24 12:36 ` jamal
2007-08-24 16:25 ` Rick Jones
-- strict thread matches above, loose matches on Subject: below --
2007-08-08 9:31 [ofa-general] " Krishna Kumar
2007-08-08 10:49 ` [ofa-general] " David Miller
2007-08-08 11:09 ` Krishna Kumar2
2007-08-08 22:01 ` [ofa-general] " David Miller
2007-08-09 4:19 ` Krishna Kumar2
2007-08-08 13:42 ` Herbert Xu
2007-08-08 15:14 ` jamal
2007-08-08 20:55 ` Stephen Hemminger
2007-08-08 22:40 ` jamal
2007-08-08 22:22 ` David Miller
2007-08-08 22:53 ` jamal
2007-08-09 3:19 ` Krishna Kumar2
2007-08-14 9:02 ` Krishna Kumar2
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070823231820.2ae52cc0.billfink@mindspring.com \
--to=billfink@mindspring$(echo .)com \
--cc=Robert.Olsson@data$(echo .)slu.se \
--cc=davem@davemloft$(echo .)net \
--cc=gaagaan@gmail$(echo .)com \
--cc=general@lists$(echo .)openfabrics.org \
--cc=hadi@cyberus$(echo .)ca \
--cc=herbert@gondor$(echo .)apana.org.au \
--cc=jagana@us$(echo .)ibm.com \
--cc=jeff@garzik$(echo .)org \
--cc=johnpol@2ka$(echo .)mipt.ru \
--cc=kaber@trash$(echo .)net \
--cc=mcarlson@broadcom$(echo .)com \
--cc=mchan@broadcom$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
--cc=peter.p.waskiewicz.jr@intel$(echo .)com \
--cc=rdreier@cisco$(echo .)com \
--cc=rick.jones2@hp$(echo .)com \
--cc=shemminger@linux-foundation$(echo .)org \
--cc=sri@us$(echo .)ibm.com \
--cc=tgraf@suug$(echo .)ch \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox