Subject: Re: [net-next PATCH] net: reduce cycles spend on ICMP replies that gets rate limited

From: Jesper Dangaard Brouer <[email-masked]>
Date: Sat, 7 Jan 2017 11:31:33 +0100

On Fri, 06 Jan 2017 22:10:42 -0500 (EST)
David Miller <[email-masked]> wrote:

> BTW Eric, you asked about kmalloc() allocation, you were CC:'d in the
> patch which did this :-)
> 
> commit 9a99d4a50cb8ce516adf0f2436138d4c8e6e4535
> Author: Cong Wang <[email-masked]>
> Date:   Sun Jun 2 15:00:52 2013 +0000
> 
>     icmp: avoid allocating large struct on stack
>     
>     struct icmp_bxm is a large struct, reduce stack usage
>     by allocating it on heap.
>     
>     Cc: Eric Dumazet <[email-masked]>
>     Cc: Joe Perches <[email-masked]>
>     Cc: David S. Miller <[email-masked]>
>     Signed-off-by: Cong Wang <[email-masked]>
>     Signed-off-by: David S. Miller <[email-masked]>

Did a quick revert, and tested again.  It is not the major bottleneck,
but we do save something.  The major bottleneck is still the call to
__ip_route_output_key_hash (invoked by icmp_route_lookup).

Single flow improvement from 1719182 pps to 1783368 pps.
 - 64186 pps
 - (1/1783368-1/1719182)*10^9 = -20.93 nanosec
   * 4GHz approx = 20.93*4 = 83.72 cycles

The optimal SLUB fast-path on this machine is 54 cycles(tsc) 13.557 ns,
thus the saving is actually higher than expected.  But low compared to
avoiding the icmp_route_lookup.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer