* Why is LRO off by default on ixgbe? @ 2009-09-23 16:29 Ben Greear 2009-09-23 16:53 ` Stephen Hemminger 0 siblings, 1 reply; 8+ messages in thread From: Ben Greear @ 2009-09-23 16:29 UTC (permalink / raw) To: NetDev I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU). Why is LRO disabled by default? Thanks, Ben -- Ben Greear <greearb@candelatech•com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-23 16:29 Why is LRO off by default on ixgbe? Ben Greear @ 2009-09-23 16:53 ` Stephen Hemminger 2009-09-23 16:57 ` Ben Greear 0 siblings, 1 reply; 8+ messages in thread From: Stephen Hemminger @ 2009-09-23 16:53 UTC (permalink / raw) To: Ben Greear; +Cc: NetDev On Wed, 23 Sep 2009 09:29:59 -0700 Ben Greear <greearb@candelatech•com> wrote: > I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two > NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU). > > Why is LRO disabled by default? > > Thanks, > Ben LRO is turned off if bridging or routing because of End to End requirements. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-23 16:53 ` Stephen Hemminger @ 2009-09-23 16:57 ` Ben Greear 2009-09-23 17:07 ` Stephen Hemminger 0 siblings, 1 reply; 8+ messages in thread From: Ben Greear @ 2009-09-23 16:57 UTC (permalink / raw) To: Stephen Hemminger; +Cc: NetDev On 09/23/2009 09:53 AM, Stephen Hemminger wrote: > On Wed, 23 Sep 2009 09:29:59 -0700 > Ben Greear<greearb@candelatech•com> wrote: > >> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two >> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU). >> >> Why is LRO disabled by default? >> >> Thanks, >> Ben > > LRO is turned off if bridging or routing because of End to End requirements. That makes sense. If I know that all interfaces in question can handle TSO and LRO, I could manually enable LRO w/out risk, right? Thanks, Ben -- Ben Greear <greearb@candelatech•com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-23 16:57 ` Ben Greear @ 2009-09-23 17:07 ` Stephen Hemminger 2009-09-23 17:16 ` Ben Greear 0 siblings, 1 reply; 8+ messages in thread From: Stephen Hemminger @ 2009-09-23 17:07 UTC (permalink / raw) To: Ben Greear; +Cc: NetDev On Wed, 23 Sep 2009 09:57:51 -0700 Ben Greear <greearb@candelatech•com> wrote: > On 09/23/2009 09:53 AM, Stephen Hemminger wrote: > > On Wed, 23 Sep 2009 09:29:59 -0700 > > Ben Greear<greearb@candelatech•com> wrote: > > > >> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two > >> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU). > >> > >> Why is LRO disabled by default? > >> > >> Thanks, > >> Ben > > > > LRO is turned off if bridging or routing because of End to End requirements. > > That makes sense. > > If I know that all interfaces in question can handle TSO and LRO, > I could manually enable LRO w/out risk, right? > The problem is that LRO merges TCP packets, this breaks the end-to-end ack clocking and checksumming, and therefore is not enabled. That is why GRO is the replacement solution (preserves packet boundaries) ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-23 17:07 ` Stephen Hemminger @ 2009-09-23 17:16 ` Ben Greear 2009-09-25 21:37 ` Herbert Xu 0 siblings, 1 reply; 8+ messages in thread From: Ben Greear @ 2009-09-23 17:16 UTC (permalink / raw) To: Stephen Hemminger; +Cc: NetDev On 09/23/2009 10:07 AM, Stephen Hemminger wrote: > On Wed, 23 Sep 2009 09:57:51 -0700 > Ben Greear<greearb@candelatech•com> wrote: > >> On 09/23/2009 09:53 AM, Stephen Hemminger wrote: >>> On Wed, 23 Sep 2009 09:29:59 -0700 >>> Ben Greear<greearb@candelatech•com> wrote: >>> >>>> I just noticed that enabling LRO on ixgbe lets me reach about 9Gbps receive on two >>>> NICs concurrently in an NFS test, where I was only getting about 6Gbps w/out it (1500 MTU). >>>> >>>> Why is LRO disabled by default? >>>> >>>> Thanks, >>>> Ben >>> >>> LRO is turned off if bridging or routing because of End to End requirements. >> >> That makes sense. >> >> If I know that all interfaces in question can handle TSO and LRO, >> I could manually enable LRO w/out risk, right? >> > > The problem is that LRO merges TCP packets, this breaks the end-to-end > ack clocking and checksumming, and therefore is not enabled. > That is why GRO is the replacement solution (preserves packet boundaries) Ok. It seems GRO was enabled the whole time, but LRO is what gave me the extra performance boost. In this particular case, I'm not actually routing, though I do have ip-forward enabled, so I guess LRO will be OK as long as I'm careful... Thanks, Ben -- Ben Greear <greearb@candelatech•com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-23 17:16 ` Ben Greear @ 2009-09-25 21:37 ` Herbert Xu 2009-09-25 21:43 ` Herbert Xu 0 siblings, 1 reply; 8+ messages in thread From: Herbert Xu @ 2009-09-25 21:37 UTC (permalink / raw) To: Ben Greear; +Cc: shemminger, netdev Ben Greear <greearb@candelatech•com> wrote: > > Ok. It seems GRO was enabled the whole time, but LRO is what gave me the > extra performance boost. > > In this particular case, I'm not actually routing, though I do have ip-forward > enabled, so I guess LRO will be OK as long as I'm careful... What? A driver is either GRO or LRO, it can't be both. What kernel version was this? Thanks, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor•apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-25 21:37 ` Herbert Xu @ 2009-09-25 21:43 ` Herbert Xu 2009-09-25 21:53 ` Ben Greear 0 siblings, 1 reply; 8+ messages in thread From: Herbert Xu @ 2009-09-25 21:43 UTC (permalink / raw) To: Ben Greear; +Cc: shemminger, netdev On Fri, Sep 25, 2009 at 02:37:27PM -0700, Herbert Xu wrote: > > > Ok. It seems GRO was enabled the whole time, but LRO is what gave me the > > extra performance boost. > > > > In this particular case, I'm not actually routing, though I do have ip-forward > > enabled, so I guess LRO will be OK as long as I'm careful... > > What? A driver is either GRO or LRO, it can't be both. What > kernel version was this? Oh perhaps you mean LRO performed by hardware? That would make sense indeed. It's also one of the reasons why we hope the hardware folks would switch over to GRO so we can enable it for everyone :) Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor•apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Why is LRO off by default on ixgbe? 2009-09-25 21:43 ` Herbert Xu @ 2009-09-25 21:53 ` Ben Greear 0 siblings, 0 replies; 8+ messages in thread From: Ben Greear @ 2009-09-25 21:53 UTC (permalink / raw) To: Herbert Xu; +Cc: shemminger, netdev On 09/25/2009 02:43 PM, Herbert Xu wrote: > On Fri, Sep 25, 2009 at 02:37:27PM -0700, Herbert Xu wrote: >> >>> Ok. It seems GRO was enabled the whole time, but LRO is what gave me the >>> extra performance boost. >>> >>> In this particular case, I'm not actually routing, though I do have ip-forward >>> enabled, so I guess LRO will be OK as long as I'm careful... >> >> What? A driver is either GRO or LRO, it can't be both. What >> kernel version was this? > > Oh perhaps you mean LRO performed by hardware? That would make > sense indeed. It's also one of the reasons why we hope the hardware > folks would switch over to GRO so we can enable it for everyone :) I assume it's in hardware. I am using ixgbe driver, 82599 chipset, 2.6.31 kernel. GRO was showing as enabled the whole time in ethtool. Turning on LRO gave me an extra 6Gbps (from 12Gbps -> 18Gbps) receive throughput, so it certainly had some affect! Thanks, Ben -- Ben Greear <greearb@candelatech•com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2009-09-25 21:55 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-09-23 16:29 Why is LRO off by default on ixgbe? Ben Greear 2009-09-23 16:53 ` Stephen Hemminger 2009-09-23 16:57 ` Ben Greear 2009-09-23 17:07 ` Stephen Hemminger 2009-09-23 17:16 ` Ben Greear 2009-09-25 21:37 ` Herbert Xu 2009-09-25 21:43 ` Herbert Xu 2009-09-25 21:53 ` Ben Greear
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox