Subject: [LKP] [net/sock] b99259a614: netperf.Throughput_Mbps -6.6% regression

From: kernel test robot <[email-masked]>
Date: Tue, 18 Sep 2018 17:37:22 +0800

Greeting,

FYI, we noticed a -6.6% regression of netperf.Throughput_Mbps due to commit:

commit: b99259a61450bb6403bbbbf279f8754a441eae8b ("[PATCH] net/sock: move memory_allocated over to percpu_counter variables")
url: https://github.com/0day-ci/linux/commits/Olof-Johansson/net-sock-move-memory_allocated-over-to-percpu_counter-variables/20180907-095536

in testcase: netperf
on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 200%
	cluster: cs-localhost
	send_size: 5K
	test: TCP_SENDFILE
	ucode: 0x25
	cpufreq_governor: performance

test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_total_tps -49.3% improvement              |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
| test parameters  | cluster=cs-localhost                                                  |
|                  | cpufreq_governor=performance                                          |
|                  | ip=ipv4                                                               |
|                  | nr_threads=50%                                                        |
|                  | runtime=300s                                                          |
|                  | test=UDP_RR                                                           |
+------------------+-----------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_total_tps -13.5% improvement              |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
| test parameters  | cluster=cs-localhost                                                  |
|                  | cpufreq_governor=performance                                          |
|                  | ip=ipv4                                                               |
|                  | nr_threads=25%                                                        |
|                  | runtime=300s                                                          |
|                  | test=UDP_RR                                                           |
+------------------+-----------------------------------------------------------------------+

Details are as below:
-------------------------------------------------------------------------------------------------->

To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/200%/debian-x86_64-2018-04-03.cgz/300s/5K/lkp-hsw-d01/TCP_SENDFILE/netperf/0x25

commit: 
  6da410d97f (" mlx5e-fixes-2018-09-05")
  b99259a614 ("net/sock: move memory_allocated over to percpu_counter variables")

6da410d97ffa486e b99259a61450bb6403bbbbf279 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.RIP:drm_wait_one_vblank[drm]
          1:4          -25%            :4     dmesg.WARNING:at_drivers/gpu/drm/drm_vblank.c:#drm_wait_one_vblank[drm]
           :4           25%           1:4     dmesg.WARNING:at_ip_fsnotify/0x
          1:4          -25%            :4     kmsg.drm:drm_atomic_helper_wait_for_dependencies[drm_kms_helper]]*ERROR*[CONNECTOR:#:VGA-#]flip_done_timed_out
          1:4          -25%            :4     kmsg.drm:drm_atomic_helper_wait_for_dependencies[drm_kms_helper]]*ERROR*[CRTC:#:pipe_A]flip_done_timed_out
          1:4          -25%            :4     kmsg.drm:drm_atomic_helper_wait_for_dependencies[drm_kms_helper]]*ERROR*[PLANE:#:primary_A]flip_done_timed_out
          1:4          -25%            :4     kmsg.drm:drm_atomic_helper_wait_for_flip_done[drm_kms_helper]]*ERROR*[CRTC:#:pipe_A]flip_done_timed_out
          0:4            0%           0:4     perf-profile.children.cycles-pp.schedule_timeout
         %stddev     %change         %stddev
             \          |                \  
      7990            -6.6%       7460        netperf.Throughput_Mbps
    127841            -6.6%     119362        netperf.Throughput_total_Mbps
   1874683            -2.1%    1835777        netperf.time.involuntary_context_switches
    643.25            +1.3%     651.50        netperf.time.percent_of_cpu_this_job_got
      1792            +2.1%       1830        netperf.time.system_time
    144.93            -8.1%     133.19        netperf.time.user_time
   1099277            -7.7%    1014302        netperf.time.voluntary_context_switches
 9.363e+08            -6.6%  8.742e+08        netperf.workload
     30722            -4.2%      29428        vmstat.system.cs
     25888 ± 31%     +77.6%      45981 ± 25%  sched_debug.cfs_rq:/.min_vruntime.stddev
     25888 ± 31%     +77.6%      45981 ± 25%  sched_debug.cfs_rq:/.spread0.stddev
      7151 ±  4%     -20.0%       5720 ±  5%  slabinfo.anon_vma_chain.active_objs
      7409 ±  3%     -21.7%       5804 ±  6%  slabinfo.anon_vma_chain.num_objs
     38618 ± 10%     -31.7%      26365 ± 22%  turbostat.C1E
      0.13 ± 15%      -0.1        0.08 ± 44%  turbostat.C1E%
    161917 ±  4%     -24.6%     122121 ± 15%  cpuidle.C1.time
   3215895 ± 15%     -38.8%    1966692 ± 43%  cpuidle.C1E.time
     38639 ± 10%     -31.8%      26368 ± 22%  cpuidle.C1E.usage
 1.305e+12            -1.4%  1.287e+12        perf-stat.branch-instructions
      3.00            -0.0        2.97        perf-stat.branch-miss-rate%
  3.92e+10            -2.5%  3.822e+10        perf-stat.branch-misses
      0.69 ±  9%      -0.2        0.51 ± 14%  perf-stat.cache-miss-rate%
   2.1e+10           +17.9%  2.477e+10        perf-stat.cache-references
   9364894            -4.2%    8976073        perf-stat.context-switches
      1.31            +2.0%       1.33        perf-stat.cpi
      0.09            -0.0        0.08        perf-stat.dTLB-load-miss-rate%
  2.02e+09           -15.8%  1.701e+09        perf-stat.dTLB-load-misses
 2.201e+12            -3.0%  2.135e+12        perf-stat.dTLB-loads
  1.38e+12            -4.1%  1.325e+12        perf-stat.dTLB-stores
 4.827e+08 ±  8%     -19.0%  3.908e+08 ±  9%  perf-stat.iTLB-loads
 6.749e+12            -1.9%   6.62e+12        perf-stat.instructions
      0.76            -1.9%       0.75        perf-stat.ipc
      7207            +5.1%       7572        perf-stat.path-length
     18.44            -1.4       17.04        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.46            -1.4       17.06        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.25            -1.4       16.86        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.17            -1.4       16.80        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
     13.45            -1.2       12.24        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     12.68            -1.1       11.57        perf-profile.calltrace.cycles-pp.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
     11.95            -0.9       11.00 ±  2%  perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      9.53 ±  2%      -0.9        8.68        perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
      9.35 ±  2%      -0.8        8.54        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg
      9.50            -0.8        8.73 ±  2%  perf-profile.calltrace.cycles-pp.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile
     10.73 ±  2%      -0.5       10.21        perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
     10.56 ±  2%      -0.4       10.11        perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      2.99 ±  3%      -0.4        2.59 ±  4%  perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
      4.94 ±  2%      -0.3        4.60 ±  3%  perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.70 ± 14%      -0.3        0.43 ± 58%  perf-profile.calltrace.cycles-pp.__inode_security_revalidate.selinux_file_permission.security_file_permission.do_splice_direct.do_sendfile
      3.04 ±  2%      -0.3        2.77 ±  4%  perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      2.62 ±  3%      -0.2        2.39 ±  2%  perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct
      2.31 ±  2%      -0.2        2.09 ±  2%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor
      1.52 ±  5%      -0.2        1.31 ±  8%  perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      1.14 ±  4%      -0.2        0.93 ± 10%  perf-profile.calltrace.cycles-pp.__fget_light.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.88 ±  2%      -0.2        1.68        perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
      1.11 ± 16%      -0.2        0.92 ±  3%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      3.21            -0.2        3.02 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      0.74 ± 23%      -0.2        0.56 ±  6%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.19 ±  7%      -0.2        1.02 ±  5%  perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64
      1.20 ±  3%      -0.1        1.05 ±  5%  perf-profile.calltrace.cycles-pp.__might_fault.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
      1.19 ±  5%      -0.1        1.06 ±  6%  perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64
      0.80 ±  4%      -0.1        0.68        perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg
      0.90 ±  6%      -0.1        0.78 ±  6%  perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.splice_direct_to_actor.do_splice_direct.do_sendfile
      0.71 ±  2%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.sched_clock_cpu.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
      1.07 ±  3%      -0.1        0.96 ±  3%  perf-profile.calltrace.cycles-pp.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read
      0.74 ±  6%      -0.1        0.64 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage
      0.63 ±  3%      -0.1        0.55 ±  6%  perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock.sched_clock_cpu.tcp_write_xmit.__tcp_push_pending_frames
      0.70 ±  3%      -0.1        0.63 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
      0.78 ±  3%      -0.1        0.71 ±  5%  perf-profile.calltrace.cycles-pp.__radix_tree_lookup.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.generic_file_read_iter
      0.82 ±  3%      -0.1        0.76 ±  3%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      5.58            +0.1        5.71        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
      5.35            +0.1        5.49        perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
      0.91 ±  4%      +0.1        1.04 ±  3%  perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
     90.58            +0.4       90.99        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.5        0.55 ±  6%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.00            +0.6        0.60 ±  6%  perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.00            +0.6        0.65 ± 18%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      0.00            +0.7        0.68 ±  2%  perf-profile.calltrace.cycles-pp.find_next_bit.cpumask_next.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule
      0.00            +0.8        0.78 ± 13%  perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked
      0.00            +0.8        0.84        perf-profile.calltrace.cycles-pp.cpumask_next.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages
      0.00            +1.7        1.65 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule
     70.71            +2.0       72.70        perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     68.26            +2.2       70.47        perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.8        2.85 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages
     57.51            +3.2       60.70        perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     54.10            +3.6       57.72        perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64
     36.84            +4.9       41.77        perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64
     35.89            +5.0       40.88        perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile
     35.00            +5.0       40.01        perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct
     30.95            +5.0       36.00        perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor
     29.68            +5.2       34.87        perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor
     28.59            +5.3       33.84        perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe
     28.87            +5.3       34.14        perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe
     27.07            +5.3       32.36        perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage
     21.40            +5.7       27.15        perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage
     20.54            +5.8       26.38        perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage
      0.00            +5.9        5.93 ±  4%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked
      1.22 ±  4%      +6.2        7.47 ±  3%  perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      1.32 ±  3%      +6.3        7.60 ±  3%  perf-profile.calltrace.cycles-pp.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
     18.44            -1.4       17.04        perf-profile.children.cycles-pp.__sys_recvfrom
     18.46            -1.4       17.06        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     18.25            -1.4       16.86        perf-profile.children.cycles-pp.inet_recvmsg
     18.19            -1.4       16.81        perf-profile.children.cycles-pp.tcp_recvmsg
     15.06            -1.2       13.82        perf-profile.children.cycles-pp.copy_page_to_iter
     13.47            -1.2       12.25        perf-profile.children.cycles-pp.skb_copy_datagram_iter
     12.00            -1.0       11.05 ±  2%  perf-profile.children.cycles-pp.generic_file_splice_read
      9.58 ±  2%      -0.9        8.72        perf-profile.children.cycles-pp.copyout
     10.09            -0.9        9.23 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      9.42 ±  2%      -0.8        8.60        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      9.68            -0.8        8.90 ±  2%  perf-profile.children.cycles-pp.generic_file_read_iter
     11.60 ±  2%      -0.6       11.03        perf-profile.children.cycles-pp.__tcp_push_pending_frames
     11.47 ±  2%      -0.5       10.94        perf-profile.children.cycles-pp.tcp_write_xmit
      6.05 ±  2%      -0.5        5.57 ±  2%  perf-profile.children.cycles-pp.selinux_file_permission
      3.54 ±  3%      -0.3        3.24 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      2.50 ±  3%      -0.3        2.21 ±  5%  perf-profile.children.cycles-pp.file_has_perm
      3.05 ±  2%      -0.3        2.78 ±  4%  perf-profile.children.cycles-pp.do_splice_to
      2.06            -0.3        1.81 ±  2%  perf-profile.children.cycles-pp.fsnotify
      2.70 ±  2%      -0.2        2.45 ±  2%  perf-profile.children.cycles-pp.pagecache_get_page
      3.39 ±  3%      -0.2        3.15        perf-profile.children.cycles-pp.touch_atime
      2.10 ±  2%      -0.2        1.87 ±  2%  perf-profile.children.cycles-pp.lock_sock_nested
      1.29 ± 13%      -0.2        1.07 ±  2%  perf-profile.children.cycles-pp.__kfree_skb
      2.57 ±  4%      -0.2        2.35        perf-profile.children.cycles-pp.atime_needs_update
      2.37            -0.2        2.15 ±  3%  perf-profile.children.cycles-pp.__might_fault
      2.34 ±  2%      -0.2        2.13 ±  2%  perf-profile.children.cycles-pp.find_get_entry
      1.21 ±  5%      -0.2        1.00 ±  9%  perf-profile.children.cycles-pp.__fget_light
      3.67            -0.2        3.46 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      2.72 ±  4%      -0.2        2.52 ±  2%  perf-profile.children.cycles-pp.__inode_security_revalidate
      1.61 ±  4%      -0.2        1.41 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.86 ± 20%      -0.2        0.67 ±  5%  perf-profile.children.cycles-pp.skb_release_data
      0.47 ± 18%      -0.2        0.30 ± 28%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.99 ±  5%      -0.2        0.83 ±  5%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.41 ± 21%      -0.2        0.25 ± 33%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.52 ± 16%      -0.2        0.36 ± 24%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.56 ± 15%      -0.2        0.40 ± 22%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.49 ±  2%      -0.1        1.34 ±  4%  perf-profile.children.cycles-pp.current_time
      0.32 ± 25%      -0.1        0.18 ± 48%  perf-profile.children.cycles-pp.tick_sched_timer
      1.77 ±  4%      -0.1        1.65 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      1.10 ±  3%      -0.1        0.98 ±  2%  perf-profile.children.cycles-pp.radix_tree_lookup_slot
      0.70 ± 11%      -0.1        0.58 ±  9%  perf-profile.children.cycles-pp.nf_hook_slow
      0.96 ±  2%      -0.1        0.85 ±  4%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.88 ±  2%      -0.1        0.78 ±  3%  perf-profile.children.cycles-pp.sched_clock
      0.85 ±  2%      -0.1        0.75 ±  3%  perf-profile.children.cycles-pp.native_sched_clock
      0.46 ±  4%      -0.1        0.37 ±  8%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.37 ±  5%      -0.1        0.30 ±  8%  perf-profile.children.cycles-pp.selinux_ip_postroute_compat
      0.44 ±  3%      -0.1        0.38 ±  4%  perf-profile.children.cycles-pp.tcp_rate_check_app_limited
      0.12 ± 27%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__copy_skb_header
      0.20 ± 16%      -0.0        0.15 ± 12%  perf-profile.children.cycles-pp.lock_timer_base
      0.16 ± 13%      -0.0        0.11 ± 18%  perf-profile.children.cycles-pp.ip_rcv_core
      0.21 ±  7%      -0.0        0.17 ±  7%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.37 ±  3%      -0.0        0.33 ±  6%  perf-profile.children.cycles-pp.generic_splice_sendpage
      0.25 ±  7%      -0.0        0.21 ±  7%  perf-profile.children.cycles-pp.iov_iter_pipe
      0.06 ± 11%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.06 ± 13%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.get_nohz_timer_target
      0.09 ± 18%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.sk_stream_wait_memory
      0.09 ± 11%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.04 ± 60%      +0.1        0.11 ± 14%  perf-profile.children.cycles-pp.splice_from_pipe_next
      0.91 ±  4%      +0.1        1.04 ±  3%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.11 ±  7%      +0.2        0.34 ±  9%  perf-profile.children.cycles-pp.tcp_leave_memory_pressure
      0.33 ± 10%      +0.2        0.58 ±  4%  perf-profile.children.cycles-pp.__release_sock
      3.19 ±  2%      +0.4        3.62 ±  4%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
      0.55 ± 12%      +0.4        0.99 ± 10%  perf-profile.children.cycles-pp.tcp_data_queue
      3.07 ±  2%      +0.4        3.51 ±  5%  perf-profile.children.cycles-pp.tcp_rcv_established
      0.18 ± 27%      +0.4        0.63 ± 15%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
     90.71            +0.5       91.20        perf-profile.children.cycles-pp.do_syscall_64
      0.08 ± 13%      +0.5        0.60 ± 12%  perf-profile.children.cycles-pp.tcp_try_rmem_schedule
      0.00            +0.8        0.77 ±  2%  perf-profile.children.cycles-pp.find_next_bit
      0.00            +0.9        0.94 ± 10%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +1.1        1.14 ±  3%  perf-profile.children.cycles-pp.cpumask_next
      0.00            +1.9        1.92 ±  6%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     70.92            +2.0       72.87        perf-profile.children.cycles-pp.__x64_sys_sendfile64
     68.42            +2.2       70.62        perf-profile.children.cycles-pp.do_sendfile
     57.60            +3.2       60.79        perf-profile.children.cycles-pp.do_splice_direct
      0.25 ± 13%      +3.3        3.54 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     54.16            +3.6       57.77        perf-profile.children.cycles-pp.splice_direct_to_actor
     36.88            +5.0       41.83        perf-profile.children.cycles-pp.direct_splice_actor
     35.98            +5.0       40.97        perf-profile.children.cycles-pp.splice_from_pipe
     35.13            +5.0       40.13        perf-profile.children.cycles-pp.__splice_from_pipe
     31.00            +5.0       36.03        perf-profile.children.cycles-pp.pipe_to_sendpage
     29.75            +5.2       34.97        perf-profile.children.cycles-pp.sock_sendpage
     28.64            +5.3       33.89        perf-profile.children.cycles-pp.inet_sendpage
     28.94            +5.3       34.22        perf-profile.children.cycles-pp.kernel_sendpage
     27.21            +5.3       32.51        perf-profile.children.cycles-pp.tcp_sendpage
     21.48            +5.7       27.23        perf-profile.children.cycles-pp.tcp_sendpage_locked
     20.69            +5.8       26.53        perf-profile.children.cycles-pp.do_tcp_sendpages
      0.00            +6.7        6.67 ±  4%  perf-profile.children.cycles-pp.__percpu_counter_sum
      1.34 ±  3%      +7.0        8.32 ±  3%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      1.46 ±  3%      +7.0        8.46 ±  3%  perf-profile.children.cycles-pp.__sk_mem_schedule
      1.21 ±  3%      -0.8        0.40 ±  9%  perf-profile.self.cycles-pp.__sk_mem_raise_allocated
      9.27            -0.8        8.50        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      3.41 ±  3%      -0.3        3.08        perf-profile.self.cycles-pp.___might_sleep
      3.12 ±  3%      -0.3        2.86        perf-profile.self.cycles-pp.selinux_file_permission
      2.01 ±  2%      -0.2        1.76 ±  2%  perf-profile.self.cycles-pp.fsnotify
      2.52            -0.2        2.28        perf-profile.self.cycles-pp.generic_file_read_iter
      2.42            -0.2        2.20 ±  2%  perf-profile.self.cycles-pp.do_sendfile
      3.66            -0.2        3.45 ±  3%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.18 ±  5%      -0.2        0.98 ±  9%  perf-profile.self.cycles-pp.__fget_light
      1.62 ±  3%      -0.2        1.45 ±  5%  perf-profile.self.cycles-pp.tcp_write_xmit
      1.19 ±  2%      -0.2        1.02 ±  5%  perf-profile.self.cycles-pp.pipe_to_sendpage
      1.44 ±  5%      -0.2        1.28 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.89 ±  5%      -0.1        0.74 ±  4%  perf-profile.self.cycles-pp.__fsnotify_parent
      1.12 ±  2%      -0.1        0.98 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
      1.02 ±  3%      -0.1        0.90 ±  5%  perf-profile.self.cycles-pp.__x64_sys_sendfile64
      0.81 ±  7%      -0.1        0.70 ±  6%  perf-profile.self.cycles-pp.file_has_perm
      0.71 ±  7%      -0.1        0.61 ±  7%  perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.89 ±  6%      -0.1        0.79 ±  6%  perf-profile.self.cycles-pp.tcp_current_mss
      0.14 ±  9%      -0.1        0.04 ± 59%  perf-profile.self.cycles-pp.sched_clock
      0.75 ±  4%      -0.1        0.66 ±  6%  perf-profile.self.cycles-pp.skb_release_data
      0.97 ±  3%      -0.1        0.88 ±  6%  perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm
      0.82            -0.1        0.74 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.29 ±  8%      -0.1        0.21 ±  8%  perf-profile.self.cycles-pp.radix_tree_lookup_slot
      0.42 ±  5%      -0.1        0.35 ±  4%  perf-profile.self.cycles-pp.tcp_rate_check_app_limited
      1.57            -0.1        1.51 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.12 ± 27%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__copy_skb_header
      0.15 ±  7%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.selinux_ip_postroute_compat
      0.15 ± 14%      -0.0        0.11 ± 15%  perf-profile.self.cycles-pp.ip_rcv_core
      0.17 ±  9%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.ip_output
      0.32            -0.0        0.28 ±  4%  perf-profile.self.cycles-pp.generic_splice_sendpage
      0.14 ± 10%      -0.0        0.10 ± 14%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.23 ±  9%      -0.0        0.19 ±  2%  perf-profile.self.cycles-pp.iov_iter_pipe
      0.22 ±  9%      -0.0        0.18 ±  7%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.11 ± 19%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.sock_def_readable
      0.11 ±  4%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.bpf_fd_pass
      0.11 ±  7%      +0.0        0.15 ± 10%  perf-profile.self.cycles-pp.tcp_ack
      0.01 ±173%      +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.splice_from_pipe_next
      0.09 ±  7%      +0.2        0.32 ±  9%  perf-profile.self.cycles-pp.tcp_leave_memory_pressure
      0.15 ± 16%      +0.4        0.52 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.4        0.38 ± 12%  perf-profile.self.cycles-pp.cpumask_next
      0.00            +0.7        0.67 ±  2%  perf-profile.self.cycles-pp.find_next_bit
      0.00            +0.7        0.72 ±  6%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.24 ± 14%      +1.4        1.63 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +1.9        1.90 ±  6%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.00            +2.1        2.08 ±  3%  perf-profile.self.cycles-pp.__percpu_counter_sum

                              netperf.Throughput_Mbps                           
                                                                                
  8100 +-+------------------------------------------------------------------+   
       |           ++.+ .++.+++.+                             .++.          |   
  8000 +-+++.++.  +    +         +.++.++.++.++.++.++.+++.++.++    ++.++.++.+|   
  7900 +-+      ++                                                          |   
       |                                                                    |   
  7800 +-+                                                                  |   
       |                                                                    |   
  7700 +-+                                                                  |   
       |                                                                    |   
  7600 +-+                                                                  |   
  7500 +-+         O   O                                                    |   
       |         O  O O  OO OOO OO OO                                       |   
  7400 +-+    O O                                                           |   
       OO OO O                                                              |   
  7300 +-+------------------------------------------------------------------+   

                            netperf.Throughput_total_Mbps                       
                                                                                
  130000 +-+----------------------------------------------------------------+   
         |          +.+   +.+++.+                              +.+          |   
  128000 +-+++.++.  :  +.+      :  ++.+ +.++.+++.++.+ .+++.++.+   ++.++.++.+|   
         |+       ++            :  :   +             +                      |   
  126000 +-+                    :  :                                        |   
         |                      : :                                         |   
  124000 +-+                     ::                                         |   
         |                       ::                                         |   
  122000 +-+                     ::                                         |   
         |                       :                                          |   
  120000 +-+       OO  O OO O    O                                          |   
         |      O     O      OO O  OO O                                     |   
  118000 O-+OO    O                                                         |   
         |O    O                                                            |   
  116000 +-+----------------------------------------------------------------+   

                                   netperf.workload                             
                                                                                
  9.5e+08 +-+---------------------------------------------------------------+   
  9.4e+08 +-+        +.++ .++.+++                              +.+          |   
          |+.++.++   :   +      :  +.+++.++.++.+++.++.+++.++.++   +.+++.++.+|   
  9.3e+08 +-+     +.+           :  :                                        |   
  9.2e+08 +-+                    : :                                        |   
          |                      : :                                        |   
  9.1e+08 +-+                    : :                                        |   
    9e+08 +-+                    ::                                         |   
  8.9e+08 +-+                     :                                         |   
          |                       :                                         |   
  8.8e+08 +-+       OO OOO OO     O                                         |   
  8.7e+08 +-+    O            OOO  O OO                                     |   
          O  OO   O                                                         |   
  8.6e+08 +O+   O                                                           |   
  8.5e+08 +-+---------------------------------------------------------------+   

                              netperf.time.user_time                            
                                                                                
  152 +-+-------------------------------------------------------------------+   
  150 +-+                                                          +        |   
      |                                                            ::       |   
  148 +-+                                                          ::       |   
  146 +-+ +. +                          +              +          :  :+. +.+|   
  144 +-++  + +     .+ .++.+    .++.+   :+ +.++.++.+   :+ +.++. +.+  +  +   |   
  142 +-+      + .++  +     +.++     +.+  +         +.+  +     +            |   
      |         +                                                           |   
  140 +-+                                                                   |   
  138 +-+                                                                   |   
  136 +-+         O                                                         |   
  134 +-+       O  O OO O   O     O                                         |   
      |        O         O O  OO O  O                                       |   
  132 OO+OO OO                                                              |   
  130 +-+-------------------------------------------------------------------+   

                             netperf.time.system_time                           
                                                                                
  1840 +-+-O----------------------------------------------------------------+   
       |O O   O                                                             |   
  1830 O-+   O  O      O  O  OO O  OO                                       |   
       |           OO       O    O                                          |   
       |         O    O  O                                                  |   
  1820 +-+                                                                  |   
       |                                                                    |   
  1810 +-+                                                                  |   
       |                                                                    |   
  1800 +-+                                                                  |   
       |      +.+     +          +. +. +.+ .++.++. +. ++. +.++.        .+   |   
       | .++.+   +.  + +.   + +  : +  +   +       +  +   +     ++.+ .++  +.+|   
  1790 +-+         ++    + + + +:                                  +        |   
       |                  +     +                                           |   
  1780 +-+------------------------------------------------------------------+   

                        netperf.time.voluntary_context_switches                 
                                                                                
  1.12e+06 +-+--------------------------------------------------------------+   
           |      +    +    +   +      +       +    +.      .+ : +        .+|   
   1.1e+06 +-+++. :   + +  : + + +.++ + + .++.+ :.+ : + .+++  ::  +.+++.++  |   
  1.08e+06 +-+   + :.+   + :  +      +   +      +  +   +      +             |   
           |       +      +                                                 |   
  1.06e+06 +-+                                                              |   
           |                                                                |   
  1.04e+06 +-+                                                              |   
           |                                                                |   
  1.02e+06 +-+               OO  O OO  O                                    |   
     1e+06 +-+        OO OO O   O    O                                      |   
           |O      O O                                                      |   
    980000 O-+OO O                                                          |   
           |                                                                |   
    960000 +-+----O---------------------------------------------------------+   

[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-bdw-ep2: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/50%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ep2/UDP_RR/netperf

commit: 
  6da410d97f (" mlx5e-fixes-2018-09-05")
  b99259a614 ("net/sock: move memory_allocated over to percpu_counter variables")

6da410d97ffa486e b99259a61450bb6403bbbbf279 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          1:4          -25%            :4     dmesg.WARNING:at_ip_native_sched_clock/0x
         11:4         -149%           5:4     perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
          4:4          -54%           2:4     perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.switch_mm.__sched_text_start.schedule.schedule_timeout
         31:4         -378%          16:4     perf-profile.children.cycles-pp.schedule_timeout
          1:4          -12%           0:4     perf-profile.self.cycles-pp.schedule_timeout
         %stddev     %change         %stddev
             \          |                \  
   3140615 ±  2%     -49.3%    1593635        netperf.Throughput_total_tps
     71377 ±  2%     -49.3%      36218        netperf.Throughput_tps
      6877 ±  4%     +11.8%       7689        netperf.time.involuntary_context_switches
      2795           -10.5%       2501        netperf.time.percent_of_cpu_this_job_got
      7688            -6.4%       7194        netperf.time.system_time
    727.56 ±  3%     -53.7%     336.63 ±  6%  netperf.time.user_time
 9.335e+08 ±  3%     -48.8%  4.781e+08        netperf.time.voluntary_context_switches
 9.422e+08 ±  2%     -49.3%  4.781e+08        netperf.workload
    497.45           -10.7%     444.23        pmeter.Average_Active_Power
    143.48           -43.2%      81.53        pmeter.performance_per_watt
     25025 ± 50%     -50.2%      12452 ± 98%  proc-vmstat.numa_pte_updates
    823677            -1.0%     815119        proc-vmstat.pgfault
     14.68           +19.6       34.31 ±  2%  mpstat.cpu.soft%
     37.84           -21.0       16.88 ±  5%  mpstat.cpu.sys%
      4.73 ±  3%      -2.4        2.30 ±  6%  mpstat.cpu.usr%
    231917 ±  7%     +48.8%     345161 ± 22%  numa-numastat.node0.numa_hit
    458545 ±  3%     -25.1%     343375 ± 24%  numa-numastat.node1.local_node
    467255 ±  4%     -25.0%     350578 ± 22%  numa-numastat.node1.numa_hit
     53.00            -9.4%      48.00        vmstat.procs.r
  12208228 ±  3%     -48.6%    6279977        vmstat.system.cs
    197402            -5.8%     185888        vmstat.system.in
 9.422e+08 ±  2%     -49.3%  4.781e+08        softirqs.NET_RX
   2090351 ± 22%     -23.7%    1595486 ±  3%  softirqs.RCU
   6824307           -13.3%    5915651        softirqs.SCHED
   8055584           +10.9%    8929857        softirqs.TIMER
  7.34e+09           +40.3%   1.03e+10        cpuidle.C1.time
 1.833e+09 ±  3%     -47.4%  9.638e+08        cpuidle.C1.usage
 3.942e+08 ± 63%     -68.5%  1.243e+08 ± 29%  cpuidle.C6.time
    558983 ± 49%     -64.4%     199205 ± 18%  cpuidle.C6.usage
 1.824e+08 ±  3%     -47.9%   95056831        cpuidle.POLL.time
  34485351 ±  5%     -82.4%    6062003        cpuidle.POLL.usage
      1997           -14.5%       1708        turbostat.Avg_MHz
     71.97           -10.8       61.14        turbostat.Busy%
 1.833e+09 ±  3%     -47.4%  9.638e+08        turbostat.C1
     27.42 ±  2%     +11.2       38.62        turbostat.C1%
    556509 ± 50%     -64.8%     196082 ± 18%  turbostat.C6
      1.46 ± 64%      -1.0        0.45 ± 30%  turbostat.C6%
     27.21 ±  2%     +40.7%      38.28        turbostat.CPU%c1
    270.14           -13.5%     233.54        turbostat.PkgWatt
     15.33            +2.1%      15.65        turbostat.RAMWatt
    360260 ± 11%     -57.2%     154078 ±  6%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   2840994           -20.3%    2263123        sched_debug.cfs_rq:/.MIN_vruntime.max
    922773 ±  6%     -40.1%     552528 ±  3%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    360260 ± 11%     -57.2%     154078 ±  6%  sched_debug.cfs_rq:/.max_vruntime.avg
   2840994           -20.3%    2263123        sched_debug.cfs_rq:/.max_vruntime.max
    922773 ±  6%     -40.1%     552528 ±  3%  sched_debug.cfs_rq:/.max_vruntime.stddev
   2804769 ±  2%     -19.7%    2251378        sched_debug.cfs_rq:/.min_vruntime.avg
   2905503           -20.3%    2314293        sched_debug.cfs_rq:/.min_vruntime.max
   2731218 ±  3%     -18.3%    2232433        sched_debug.cfs_rq:/.min_vruntime.min
     29631 ± 28%     -61.3%      11469 ± 21%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.58 ±  4%      -9.9%       0.52 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
     10.43 ±  9%     -15.4%       8.83 ±  2%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     29606 ± 28%     -61.3%      11465 ± 21%  sched_debug.cfs_rq:/.spread0.stddev
    387.56           -21.9%     302.54 ±  4%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    356998 ± 25%     -45.1%     196064 ±  3%  sched_debug.cpu.avg_idle.max
     58785 ± 15%     -30.2%      41011 ±  2%  sched_debug.cpu.avg_idle.stddev
      0.46 ± 15%     -81.8%       0.08 ± 99%  sched_debug.cpu.cpu_load[1].min
      2.25 ±  8%     -42.6%       1.29 ± 10%  sched_debug.cpu.cpu_load[2].min
      4.17 ± 14%     -21.0%       3.29 ±  4%  sched_debug.cpu.cpu_load[3].min
  21059143 ±  3%     -49.0%   10741980        sched_debug.cpu.nr_switches.avg
  21578752 ±  2%     -49.7%   10846007        sched_debug.cpu.nr_switches.max
  20097461 ±  6%     -47.8%   10493962        sched_debug.cpu.nr_switches.min
    262381 ± 31%     -79.3%      54430 ± 13%  sched_debug.cpu.nr_switches.stddev
 5.784e+12 ±  2%     -27.2%  4.212e+12        perf-stat.branch-instructions
      2.76            -0.8        1.96        perf-stat.branch-miss-rate%
 1.599e+11 ±  2%     -48.2%  8.276e+10        perf-stat.branch-misses
      0.73 ±  8%      +0.4        1.14        perf-stat.cache-miss-rate%
 2.527e+09 ±  7%     +11.0%  2.805e+09        perf-stat.cache-misses
 3.482e+11           -29.1%   2.47e+11        perf-stat.cache-references
 3.733e+09 ±  3%     -48.8%  1.912e+09        perf-stat.context-switches
      1.93           +20.0%       2.32        perf-stat.cpi
 5.633e+13           -17.5%  4.646e+13        perf-stat.cpu-cycles
 7.772e+09 ±  4%     -43.6%  4.386e+09 ± 15%  perf-stat.dTLB-load-misses
 8.521e+12 ±  2%     -33.9%   5.63e+12        perf-stat.dTLB-loads
      0.00 ±  8%      +0.0        0.00 ±  7%  perf-stat.dTLB-store-miss-rate%
 1.095e+08 ±  9%     -13.5%   94683885 ±  7%  perf-stat.dTLB-store-misses
  5.49e+12 ±  2%     -48.1%  2.851e+12        perf-stat.dTLB-stores
     10.06 ±  5%      +3.6       13.65 ± 14%  perf-stat.iTLB-load-miss-rate%
 9.169e+09 ±  5%     -36.1%  5.859e+09 ± 16%  perf-stat.iTLB-load-misses
 8.203e+10 ±  3%     -54.9%  3.696e+10        perf-stat.iTLB-loads
 2.915e+13 ±  2%     -31.3%  2.002e+13        perf-stat.instructions
      0.52           -16.7%       0.43        perf-stat.ipc
    801554            -1.1%     792718        perf-stat.minor-faults
 7.603e+08 ±  5%    +108.2%  1.583e+09        perf-stat.node-load-misses
     93.23 ±  3%     -35.4       57.88 ±  2%  perf-stat.node-store-miss-rate%
 1.072e+09 ±  5%     -40.6%  6.373e+08 ±  2%  perf-stat.node-store-misses
  80515351 ± 49%    +476.1%  4.638e+08 ±  3%  perf-stat.node-stores
    801557            -1.1%     792720        perf-stat.page-faults
     30931           +35.4%      41869        perf-stat.path-length
     15.38            -7.4        7.94        perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     15.18            -7.3        7.85        perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.40            -6.5        6.93        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.14            -6.3        6.82        perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      8.62 ±  6%      -5.9        2.74        perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     10.50            -5.1        5.43        perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      8.98 ±  2%      -4.4        4.62        perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
      5.03 ± 12%      -4.2        0.88 ±  3%  perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      9.64            -3.9        5.70        perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish
      9.42            -3.9        5.49        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv
      4.54 ± 13%      -3.9        0.65 ±  4%  perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg
     44.22            -3.9       40.32        perf-profile.calltrace.cycles-pp.secondary_startup_64
      4.46 ± 13%      -3.9        0.60 ±  5%  perf-profile.calltrace.cycles-pp.ip_idents_reserve.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg
     43.77            -3.8       39.95        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     43.76            -3.8       39.94        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      7.84 ±  2%      -3.8        4.06        perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
     43.66            -3.8       39.89        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      8.92            -3.7        5.20        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb
      7.58 ±  2%      -3.7        3.91        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      7.46 ±  2%      -3.6        3.82        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      8.60            -3.6        5.00        perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
      8.49            -3.6        4.94        perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      6.10            -2.8        3.27        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.95            -2.7        3.20        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      4.93            -2.1        2.88        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      4.64            -1.9        2.69        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.65            -1.8        1.84        perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      4.00            -1.7        2.31        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      3.24 ±  2%      -1.6        1.63        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      3.27            -1.6        1.69        perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      2.52            -1.3        1.19        perf-profile.calltrace.cycles-pp.__entry_SYSCALL_64_trampoline
      2.48 ±  2%      -1.3        1.20        perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      2.23 ±  4%      -1.1        1.12 ±  8%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
      2.24            -1.1        1.18        perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      2.16 ±  2%      -1.0        1.17        perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      1.94 ±  2%      -1.0        0.96        perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      1.66 ±  5%      -0.8        0.84 ± 10%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb
      1.57 ±  5%      -0.8        0.78 ± 10%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
      1.55 ±  2%      -0.8        0.78        perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      1.65 ±  2%      -0.8        0.90        perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      1.52            -0.7        0.77        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      1.57 ±  2%      -0.7        0.85        perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb
      1.46            -0.7        0.79        perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle
      1.29 ±  5%      -0.6        0.65 ±  7%  perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      0.89 ±  2%      -0.6        0.25 ±100%  perf-profile.calltrace.cycles-pp.resched_curr.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function
      1.26 ±  5%      -0.6        0.62 ±  7%  perf-profile.calltrace.cycles-pp.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto
      1.46            -0.6        0.86        perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      1.20 ±  5%      -0.6        0.60 ±  7%  perf-profile.calltrace.cycles-pp.ip_route_output_key_hash_rcu.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg
      1.34            -0.6        0.77        perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      1.20            -0.6        0.63 ±  2%  perf-profile.calltrace.cycles-pp.switch_mm.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      1.00 ±  2%      -0.5        0.52        perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.06 ±  2%      -0.5        0.59        perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.10 ±  3%      -0.5        0.65        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.95 ±  3%      -0.4        0.54 ±  2%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
     30.84            +2.3       33.16        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     29.71            +2.7       32.39        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     49.06            +7.3       56.32        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     48.63            +7.5       56.11        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.67           +15.2       47.87        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.48           +15.3       47.77        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.55           +15.7       47.30        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     30.98           +16.0       47.02        perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     20.10           +23.1       43.17        perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     19.82           +23.2       43.02        perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     19.32           +23.4       42.76        perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     17.83           +24.2       42.03        perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg
     15.21           +25.5       40.72        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
     15.07           +25.6       40.64        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb
     14.83           +25.7       40.52        perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     14.73           +25.7       40.45        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     14.31           +25.9       40.17        perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     13.95           +26.0       39.97        perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     13.17           +26.4       39.61        perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     12.85           +26.6       39.43        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     12.43           +26.7       39.16        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     12.27           +26.8       39.07        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     11.92           +27.0       38.90        perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     11.37           +27.2       38.58        perf-profile.calltrace.cycles-pp.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     10.29           +27.7       38.01        perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver
      0.00           +30.3       30.29        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb
      0.00           +30.5       30.48        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb
      0.00           +31.8       31.84        perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv
      0.00           +32.0       31.98        perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish
     15.40            -7.4        7.96        perf-profile.children.cycles-pp.__x64_sys_recvfrom
     15.20            -7.3        7.85        perf-profile.children.cycles-pp.__sys_recvfrom
     13.61            -6.5        7.12        perf-profile.children.cycles-pp.__sched_text_start
     13.40            -6.5        6.94        perf-profile.children.cycles-pp.inet_recvmsg
     13.15            -6.3        6.83        perf-profile.children.cycles-pp.udp_recvmsg
      8.64 ±  6%      -5.9        2.75        perf-profile.children.cycles-pp.ip_make_skb
     10.53            -5.1        5.44        perf-profile.children.cycles-pp.__skb_recv_udp
      8.98 ±  2%      -4.4        4.63        perf-profile.children.cycles-pp.__skb_wait_for_more_packets
      5.04 ± 12%      -4.2        0.88 ±  3%  perf-profile.children.cycles-pp.__ip_make_skb
      9.65            -3.9        5.71        perf-profile.children.cycles-pp.sock_def_readable
      9.43            -3.9        5.49        perf-profile.children.cycles-pp.__wake_up_common_lock
      4.54 ± 13%      -3.9        0.65 ±  4%  perf-profile.children.cycles-pp.__ip_select_ident
     44.22            -3.9       40.32        perf-profile.children.cycles-pp.secondary_startup_64
     44.22            -3.9       40.32        perf-profile.children.cycles-pp.cpu_startup_entry
     44.20            -3.9       40.31        perf-profile.children.cycles-pp.do_idle
      4.46 ± 13%      -3.9        0.60 ±  5%  perf-profile.children.cycles-pp.ip_idents_reserve
     43.77            -3.8       39.95        perf-profile.children.cycles-pp.start_secondary
      8.93            -3.7        5.20        perf-profile.children.cycles-pp.__wake_up_common
      7.59 ±  2%      -3.7        3.92        perf-profile.children.cycles-pp.schedule
      8.61            -3.6        5.01        perf-profile.children.cycles-pp.autoremove_wake_function
      8.51            -3.6        4.95        perf-profile.children.cycles-pp.try_to_wake_up
      6.16            -2.9        3.31        perf-profile.children.cycles-pp.schedule_idle
      4.93            -2.1        2.88        perf-profile.children.cycles-pp.ttwu_do_activate
      4.64            -1.9        2.69        perf-profile.children.cycles-pp.enqueue_task_fair
      3.70            -1.8        1.87        perf-profile.children.cycles-pp.menu_select
      4.09            -1.7        2.36        perf-profile.children.cycles-pp.enqueue_entity
      3.32 ±  2%      -1.7        1.67        perf-profile.children.cycles-pp.dequeue_task_fair
      3.28            -1.6        1.69        perf-profile.children.cycles-pp.__ip_append_data
      2.98            -1.5        1.44 ±  2%  perf-profile.children.cycles-pp.__entry_SYSCALL_64_trampoline
      3.02 ±  2%      -1.5        1.51        perf-profile.children.cycles-pp.dequeue_entity
      2.93            -1.3        1.61        perf-profile.children.cycles-pp.update_load_avg
      2.78            -1.3        1.47        perf-profile.children.cycles-pp.pick_next_task_fair
      2.51            -1.3        1.22        perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      2.52            -1.1        1.38        perf-profile.children.cycles-pp.switch_mm_irqs_off
      2.24 ±  4%      -1.1        1.13 ±  8%  perf-profile.children.cycles-pp.__dev_queue_xmit
      1.97 ±  2%      -1.0        0.97        perf-profile.children.cycles-pp.tick_nohz_next_event
      2.16            -1.0        1.18        perf-profile.children.cycles-pp.sock_alloc_send_pskb
      1.74            -0.8        0.89        perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.67 ±  4%      -0.8        0.84 ± 10%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.58 ±  5%      -0.8        0.79 ± 11%  perf-profile.children.cycles-pp.loopback_xmit
      1.57 ±  2%      -0.8        0.79        perf-profile.children.cycles-pp.get_next_timer_interrupt
      1.46            -0.8        0.70        perf-profile.children.cycles-pp._raw_spin_lock
      1.65 ±  2%      -0.8        0.90        perf-profile.children.cycles-pp.alloc_skb_with_frags
      1.58 ±  2%      -0.7        0.85        perf-profile.children.cycles-pp.__alloc_skb
      1.49            -0.7        0.80        perf-profile.children.cycles-pp.set_next_entity
      1.36 ±  2%      -0.7        0.67        perf-profile.children.cycles-pp.load_new_mm_cr3
      1.29 ±  5%      -0.6        0.65 ±  7%  perf-profile.children.cycles-pp.ip_route_output_flow
      1.26 ±  5%      -0.6        0.63 ±  7%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      1.21 ±  5%      -0.6        0.60 ±  7%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      1.24 ±  2%      -0.6        0.65        perf-profile.children.cycles-pp.switch_mm
      1.17            -0.6        0.59        perf-profile.children.cycles-pp.update_curr
      1.10 ±  3%      -0.6        0.54 ±  3%  perf-profile.children.cycles-pp.nf_hook_slow
      1.08 ±  3%      -0.6        0.52        perf-profile.children.cycles-pp.__next_timer_interrupt
      1.09            -0.5        0.60 ±  2%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.95 ±  2%      -0.5        0.45 ±  3%  perf-profile.children.cycles-pp.___perf_sw_event
      1.01 ±  2%      -0.5        0.52        perf-profile.children.cycles-pp.move_addr_to_user
      1.06 ±  2%      -0.5        0.59        perf-profile.children.cycles-pp.ttwu_do_wakeup
      1.10 ±  3%      -0.5        0.65        perf-profile.children.cycles-pp.select_task_rq_fair
      1.02 ±  2%      -0.4        0.57        perf-profile.children.cycles-pp.check_preempt_curr
      0.83 ±  2%      -0.4        0.41 ±  2%  perf-profile.children.cycles-pp.__switch_to_asm
      0.87 ±  2%      -0.4        0.46 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      0.77 ±  5%      -0.4        0.37 ±  3%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.84            -0.4        0.45 ±  3%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.89 ±  2%      -0.4        0.50 ±  2%  perf-profile.children.cycles-pp.resched_curr
      0.87 ±  2%      -0.4        0.48        perf-profile.children.cycles-pp.__consume_stateless_skb
      0.75 ±  2%      -0.4        0.36 ±  2%  perf-profile.children.cycles-pp.__switch_to
      0.79            -0.4        0.42 ±  2%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.72 ±  7%      -0.4        0.36 ±  3%  perf-profile.children.cycles-pp.update_cfs_group
      0.64 ±  2%      -0.4        0.29        perf-profile.children.cycles-pp.ip_generic_getfrag
      0.73 ±  3%      -0.4        0.38 ±  2%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.72            -0.3        0.37 ±  3%  perf-profile.children.cycles-pp.finish_task_switch
      0.73            -0.3        0.39 ±  4%  perf-profile.children.cycles-pp.sock_wfree
      0.74            -0.3        0.40 ±  3%  perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb
      0.69 ± 10%      -0.3        0.35 ± 12%  perf-profile.children.cycles-pp.fib_table_lookup
      0.72 ±  2%      -0.3        0.39 ±  3%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.62 ±  5%      -0.3        0.30 ±  4%  perf-profile.children.cycles-pp.selinux_ip_postroute_compat
      0.73            -0.3        0.42        perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.67 ±  2%      -0.3        0.35 ±  3%  perf-profile.children.cycles-pp.__slab_free
      0.53 ±  2%      -0.3        0.23 ±  3%  perf-profile.children.cycles-pp._copy_from_iter_full
      0.61 ±  2%      -0.3        0.33 ±  2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.69 ±  3%      -0.3        0.40        perf-profile.children.cycles-pp.poll_idle
      0.61 ±  2%      -0.3        0.34 ±  5%  perf-profile.children.cycles-pp.account_entity_enqueue
      0.56 ±  2%      -0.3        0.30 ±  3%  perf-profile.children.cycles-pp.sched_clock
      0.47 ± 21%      -0.3        0.20 ± 30%  perf-profile.children.cycles-pp.netif_rx_internal
      0.51 ±  3%      -0.3        0.25 ±  4%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.54 ±  2%      -0.3        0.28 ±  2%  perf-profile.children.cycles-pp.native_sched_clock
      0.46 ±  4%      -0.2        0.21 ±  3%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.55            -0.2        0.30 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.48 ±  3%      -0.2        0.24        perf-profile.children.cycles-pp.__might_fault
      0.43 ±  4%      -0.2        0.18 ±  2%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.54            -0.2        0.30 ±  2%  perf-profile.children.cycles-pp.reweight_entity
      0.53            -0.2        0.29        perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.56 ±  2%      -0.2        0.33        perf-profile.children.cycles-pp.__update_load_avg_se
      0.72 ±  4%      -0.2        0.51        perf-profile.children.cycles-pp.find_next_bit
      0.46 ±  3%      -0.2        0.24        perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.41 ± 10%      -0.2        0.19 ±  7%  perf-profile.children.cycles-pp.avc_has_perm
      0.41 ±  2%      -0.2        0.20 ±  4%  perf-profile.children.cycles-pp.__fget_light
      0.43 ± 10%      -0.2        0.21 ± 11%  perf-profile.children.cycles-pp.read_tsc
      0.51 ±  3%      -0.2        0.30 ±  2%  perf-profile.children.cycles-pp.update_rq_clock
      0.48            -0.2        0.27 ±  3%  perf-profile.children.cycles-pp.selinux_sock_rcv_skb_compat
      0.45 ±  2%      -0.2        0.25        perf-profile.children.cycles-pp.native_write_msr
      0.46 ±  2%      -0.2        0.26 ±  4%  perf-profile.children.cycles-pp.sock_def_write_space
      0.38            -0.2        0.18 ±  2%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.49 ±  2%      -0.2        0.30 ±  2%  perf-profile.children.cycles-pp.__kmalloc_reserve
      0.42 ±  2%      -0.2        0.23 ±  2%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.46 ±  4%      -0.2        0.27 ±  3%  perf-profile.children.cycles-pp.copyout
      0.40 ±  2%      -0.2        0.21 ±  4%  perf-profile.children.cycles-pp.ip_local_out
      0.42            -0.2        0.23        perf-profile.children.cycles-pp.skb_set_owner_w
      0.36            -0.2        0.18 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.36 ±  3%      -0.2        0.18 ±  4%  perf-profile.children.cycles-pp.__ip_local_out
      0.33 ±  2%      -0.2        0.16 ±  5%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.32 ±  2%      -0.2        0.15 ±  3%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.31 ±  3%      -0.2        0.15 ±  3%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.39 ±  7%      -0.2        0.23 ±  4%  perf-profile.children.cycles-pp.ksize
      0.44            -0.2        0.28        perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.41 ±  7%      -0.2        0.26        perf-profile.children.cycles-pp.select_idle_sibling
      0.34 ±  2%      -0.2        0.19 ±  6%  perf-profile.children.cycles-pp.ipv4_mtu
      0.30 ±  2%      -0.1        0.15 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.35 ±  3%      -0.1        0.21 ±  2%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.28            -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__skb_try_recv_from_queue
      0.27 ±  5%      -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.29 ±  2%      -0.1        0.15 ±  5%  perf-profile.children.cycles-pp.__udp4_lib_lookup
      0.24 ±  3%      -0.1        0.10        perf-profile.children.cycles-pp.copyin
      0.32 ±  3%      -0.1        0.18 ±  3%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.25 ±  4%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.kmem_cache_free
      0.24 ±  2%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2
      0.28 ±  2%      -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.27 ±  4%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.sock_has_perm
      0.28            -0.1        0.15 ±  5%  perf-profile.children.cycles-pp.selinux_parse_skb
      0.35 ±  3%      -0.1        0.23 ±  3%  perf-profile.children.cycles-pp.available_idle_cpu
      0.25            -0.1        0.12 ±  4%  perf-profile.children.cycles-pp._copy_from_user
      0.27 ± 20%      -0.1        0.15 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.29 ±  2%      -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.20 ±  8%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.22 ±  4%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.account_entity_dequeue
      0.32            -0.1        0.20 ±  5%  perf-profile.children.cycles-pp.__enqueue_entity
      0.29 ±  2%      -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.skb_release_data
      0.22 ±  3%      -0.1        0.11        perf-profile.children.cycles-pp._copy_to_user
      0.20 ±  4%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.menu_reflect
      0.21 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.put_prev_entity
      0.23            -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.22 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.22 ±  3%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.21 ±  2%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.security_sk_classify_flow
      0.21 ±  3%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.sock_recvmsg
      0.18 ±  3%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__get_user_4
      0.51 ±  3%      -0.1        0.41 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.22            -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.ip_setup_cork
      0.19 ±  4%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.inet_sendmsg
      0.21 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      0.20 ±  2%      -0.1        0.10        perf-profile.children.cycles-pp.__calc_delta
      0.18 ±  2%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.18 ±  2%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.call_cpuidle
      0.23 ±  2%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.12 ±  4%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.19 ±  3%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.23 ±  3%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.activate_task
      0.18 ±  2%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.netif_skb_features
      0.18 ±  4%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.rb_erase_cached
      0.15 ±  2%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.rcu_needs_cpu
      0.17 ±  4%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.ip_finish_output
      0.16 ±  2%      -0.1        0.10        perf-profile.children.cycles-pp.ip_rcv_core
      0.13 ±  3%      -0.1        0.07 ±  5%  perf-profile.children.cycles-pp.selinux_ipv4_output
      0.13            -0.1        0.07        perf-profile.children.cycles-pp.selinux_sk_getsecid
      0.11 ±  6%      -0.1        0.05        perf-profile.children.cycles-pp.update_min_vruntime
      0.08 ±  5%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.ip_send_check
      0.12 ±  9%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.compute_score
      0.10 ±  5%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.deactivate_task
      0.11 ±  6%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ipv4_pktinfo_prepare
      0.11            -0.1        0.06 ±  9%  perf-profile.children.cycles-pp.dst_release
      0.08 ±  8%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__slab_alloc
      0.11 ±  4%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.pm_qos_request
      0.10 ±  4%      -0.1        0.05        perf-profile.children.cycles-pp.udp_rcv
      0.10 ±  8%      -0.1        0.05        perf-profile.children.cycles-pp.__update_idle_core
      0.13 ± 10%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.import_single_range
      0.10 ±  4%      -0.0        0.05        perf-profile.children.cycles-pp.rcu_eqs_exit
      0.11 ±  7%      -0.0        0.07        perf-profile.children.cycles-pp.rb_insert_color_cached
      0.08 ±  8%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.13            -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.irq_exit
      0.07 ±  5%      -0.0        0.05        perf-profile.children.cycles-pp.rebalance_domains
      0.07 ±  5%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.receiver_wake_function
      0.00            +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +0.3        0.28        perf-profile.children.cycles-pp.cpumask_next
     31.18            +2.3       33.50        perf-profile.children.cycles-pp.cpuidle_enter_state
     30.02            +2.7       32.70        perf-profile.children.cycles-pp.intel_idle
     49.12            +7.3       56.39        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     48.68            +7.5       56.17        perf-profile.children.cycles-pp.do_syscall_64
     32.69           +15.2       47.88        perf-profile.children.cycles-pp.__x64_sys_sendto
     32.49           +15.3       47.77        perf-profile.children.cycles-pp.__sys_sendto
     31.56           +15.7       47.30        perf-profile.children.cycles-pp.sock_sendmsg
     31.00           +16.0       47.03        perf-profile.children.cycles-pp.udp_sendmsg
     20.10           +23.1       43.18        perf-profile.children.cycles-pp.udp_send_skb
     19.82           +23.2       43.03        perf-profile.children.cycles-pp.ip_send_skb
     19.33           +23.4       42.77        perf-profile.children.cycles-pp.ip_output
     17.85           +24.2       42.05        perf-profile.children.cycles-pp.ip_finish_output2
     15.28           +25.5       40.76        perf-profile.children.cycles-pp.__local_bh_enable_ip
     15.10           +25.6       40.66        perf-profile.children.cycles-pp.do_softirq
     14.84           +25.7       40.52        perf-profile.children.cycles-pp.do_softirq_own_stack
     14.85           +25.7       40.54        perf-profile.children.cycles-pp.__softirqentry_text_start
     14.32           +25.9       40.18        perf-profile.children.cycles-pp.net_rx_action
     13.96           +26.0       39.98        perf-profile.children.cycles-pp.process_backlog
     13.18           +26.4       39.61        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     12.86           +26.6       39.44        perf-profile.children.cycles-pp.ip_rcv
     12.43           +26.7       39.16        perf-profile.children.cycles-pp.ip_local_deliver
     12.27           +26.8       39.08        perf-profile.children.cycles-pp.ip_local_deliver_finish
     11.93           +27.0       38.90        perf-profile.children.cycles-pp.__udp4_lib_rcv
     11.38           +27.2       38.59        perf-profile.children.cycles-pp.udp_queue_rcv_skb
     10.30           +27.7       38.02        perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
      1.43 ±  3%     +29.9       31.29        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.06 ± 87%     +30.2       30.31        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.15 ±  3%     +31.8       31.98        perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      0.00           +31.9       31.87        perf-profile.children.cycles-pp.__percpu_counter_sum
      4.44 ± 13%      -3.8        0.60 ±  5%  perf-profile.self.cycles-pp.ip_idents_reserve
      2.90            -1.5        1.39        perf-profile.self.cycles-pp.__entry_SYSCALL_64_trampoline
      2.17            -1.0        1.20        perf-profile.self.cycles-pp.__sched_text_start
      1.74            -0.8        0.89        perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.39            -0.7        0.67        perf-profile.self.cycles-pp._raw_spin_lock
      1.36 ±  2%      -0.7        0.67        perf-profile.self.cycles-pp.load_new_mm_cr3
      1.25            -0.6        0.67        perf-profile.self.cycles-pp.update_load_avg
      1.08 ±  2%      -0.5        0.59 ±  2%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.87 ±  2%      -0.5        0.41 ±  3%  perf-profile.self.cycles-pp.___perf_sw_event
      1.16            -0.4        0.71        perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.83 ±  2%      -0.4        0.41 ±  2%  perf-profile.self.cycles-pp.__switch_to_asm
      0.85            -0.4        0.43 ±  2%  perf-profile.self.cycles-pp.update_curr
      1.38 ±  2%      -0.4        0.98        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.89 ±  2%      -0.4        0.49        perf-profile.self.cycles-pp.menu_select
      0.71 ±  2%      -0.4        0.35 ±  2%  perf-profile.self.cycles-pp.__switch_to
      0.79            -0.4        0.43        perf-profile.self.cycles-pp.do_idle
      0.82 ±  3%      -0.4        0.46 ±  2%  perf-profile.self.cycles-pp.resched_curr
      0.68 ±  2%      -0.4        0.32 ±  3%  perf-profile.self.cycles-pp.udp_sendmsg
      0.70 ±  7%      -0.3        0.35 ±  3%  perf-profile.self.cycles-pp.update_cfs_group
      0.68 ± 11%      -0.3        0.35 ± 12%  perf-profile.self.cycles-pp.fib_table_lookup
      0.71            -0.3        0.41        perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.68 ±  3%      -0.3        0.38 ±  2%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.66 ±  2%      -0.3        0.35 ±  3%  perf-profile.self.cycles-pp.__slab_free
      0.62            -0.3        0.32 ±  2%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.67 ±  2%      -0.3        0.38        perf-profile.self.cycles-pp.select_task_rq_fair
      0.65            -0.3        0.36        perf-profile.self.cycles-pp.set_next_entity
      0.59            -0.3        0.32        perf-profile.self.cycles-pp.udp_recvmsg
      0.57            -0.3        0.30 ±  2%  perf-profile.self.cycles-pp.finish_task_switch
      0.65            -0.3        0.39        perf-profile.self.cycles-pp.enqueue_entity
      0.51 ±  2%      -0.3        0.26 ±  3%  perf-profile.self.cycles-pp.do_syscall_64
      0.49            -0.3        0.24 ±  3%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.52 ±  2%      -0.2        0.27 ±  2%  perf-profile.self.cycles-pp.native_sched_clock
      0.54 ±  3%      -0.2        0.29 ±  2%  perf-profile.self.cycles-pp.dequeue_entity
      0.54            -0.2        0.30 ±  2%  perf-profile.self.cycles-pp.reweight_entity
      0.45 ±  2%      -0.2        0.21 ±  7%  perf-profile.self.cycles-pp.__ip_append_data
      0.47 ±  3%      -0.2        0.24        perf-profile.self.cycles-pp.__next_timer_interrupt
      0.57 ±  3%      -0.2        0.34 ±  3%  perf-profile.self.cycles-pp.poll_idle
      0.42 ±  2%      -0.2        0.19 ±  3%  perf-profile.self.cycles-pp.__skb_wait_for_more_packets
      0.54 ±  2%      -0.2        0.32        perf-profile.self.cycles-pp.__update_load_avg_se
      0.44            -0.2        0.21 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.39 ±  2%      -0.2        0.16 ±  6%  perf-profile.self.cycles-pp.ip_output
      0.55            -0.2        0.33 ±  2%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.41 ±  3%      -0.2        0.20 ±  4%  perf-profile.self.cycles-pp.__fget_light
      0.40 ± 10%      -0.2        0.18 ±  6%  perf-profile.self.cycles-pp.avc_has_perm
      0.48            -0.2        0.28        perf-profile.self.cycles-pp.move_addr_to_user
      0.45            -0.2        0.24 ±  2%  perf-profile.self.cycles-pp.native_write_msr
      0.40 ± 10%      -0.2        0.20 ± 13%  perf-profile.self.cycles-pp.read_tsc
      0.36 ±  4%      -0.2        0.16 ±  4%  perf-profile.self.cycles-pp.__skb_recv_udp
      0.45 ±  2%      -0.2        0.25 ±  5%  perf-profile.self.cycles-pp.sock_def_write_space
      0.42 ±  2%      -0.2        0.23 ±  2%  perf-profile.self.cycles-pp.account_entity_enqueue
      0.38 ±  5%      -0.2        0.19 ±  2%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.39 ±  8%      -0.2        0.20 ±  3%  perf-profile.self.cycles-pp.ip_finish_output2
      0.41            -0.2        0.23        perf-profile.self.cycles-pp.skb_set_owner_w
      0.60 ±  4%      -0.2        0.42        perf-profile.self.cycles-pp.find_next_bit
      0.34            -0.2        0.17 ±  3%  perf-profile.self.cycles-pp.__ip_make_skb
      0.36 ±  3%      -0.2        0.19 ±  2%  perf-profile.self.cycles-pp.process_backlog
      0.45 ±  4%      -0.2        0.28        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.31 ±  2%      -0.2        0.15 ±  2%  perf-profile.self.cycles-pp.__alloc_skb
      0.33            -0.2        0.17 ±  4%  perf-profile.self.cycles-pp.pick_next_entity
      0.38 ±  6%      -0.2        0.22 ±  3%  perf-profile.self.cycles-pp.ksize
      0.38 ±  4%      -0.2        0.22 ±  3%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.31 ±  2%      -0.2        0.15 ±  2%  perf-profile.self.cycles-pp.__sys_sendto
      0.29 ±  3%      -0.2        0.14 ±  8%  perf-profile.self.cycles-pp._copy_to_iter
      0.30 ±  3%      -0.1        0.15        perf-profile.self.cycles-pp.___might_sleep
      0.27            -0.1        0.12 ±  5%  perf-profile.self.cycles-pp.sock_wfree
      0.40 ±  2%      -0.1        0.26 ±  3%  perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb
      0.27 ±  3%      -0.1        0.13 ±  3%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.31            -0.1        0.17 ±  6%  perf-profile.self.cycles-pp.ipv4_mtu
      0.34 ±  3%      -0.1        0.21 ±  2%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.27 ±  4%      -0.1        0.13 ±  3%  perf-profile.self.cycles-pp.sock_has_perm
      0.29            -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.net_rx_action
      0.24 ±  3%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_stage2
      0.28 ±  2%      -0.1        0.15 ±  4%  perf-profile.self.cycles-pp.loopback_xmit
      0.24 ±  3%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.inet_recvmsg
      0.27            -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.do_softirq
      0.27            -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.selinux_parse_skb
      0.27 ± 20%      -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.35 ±  4%      -0.1        0.23        perf-profile.self.cycles-pp.available_idle_cpu
      0.31 ±  2%      -0.1        0.19 ±  3%  perf-profile.self.cycles-pp.__enqueue_entity
      0.24            -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.selinux_socket_sock_rcv_skb
      0.29 ±  3%      -0.1        0.17 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.28 ±  3%      -0.1        0.17 ±  3%  perf-profile.self.cycles-pp.skb_release_data
      0.27 ±  2%      -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.__udp4_lib_rcv
      0.21 ±  5%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.account_entity_dequeue
      0.21 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.18 ±  4%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.23            -0.1        0.12 ±  4%  perf-profile.self.cycles-pp.udp_send_skb
      0.21 ±  3%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.26            -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.20 ±  4%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.put_prev_task_fair
      0.24 ±  3%      -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.__wake_up_common
      0.20 ±  4%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.18 ±  3%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.selinux_ip_postroute_compat
      0.18 ±  2%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__get_user_4
      0.20 ±  8%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.26 ±  5%      -0.1        0.16 ±  2%  perf-profile.self.cycles-pp.update_rq_clock
      0.26            -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.20 ±  2%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.19 ±  3%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.18 ±  2%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.call_cpuidle
      0.22 ±  3%      -0.1        0.13 ±  9%  perf-profile.self.cycles-pp.__list_add_valid
      0.19 ±  3%      -0.1        0.10        perf-profile.self.cycles-pp.__calc_delta
      0.17 ±  2%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.inet_sendmsg
      0.20 ±  4%      -0.1        0.11 ±  6%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.19 ±  3%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.12 ±  4%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.23 ±  3%      -0.1        0.15 ±  4%  perf-profile.self.cycles-pp.activate_task
      0.19 ±  4%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.check_preempt_curr
      0.18 ±  2%      -0.1        0.10        perf-profile.self.cycles-pp.__might_sleep
      0.18 ±  2%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__udp4_lib_lookup
      0.19 ±  6%      -0.1        0.11 ± 18%  perf-profile.self.cycles-pp.ktime_get
      0.14 ±  3%      -0.1        0.06 ± 15%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.16 ±  2%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.selinux_sock_rcv_skb_compat
      0.15 ±  4%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.selinux_ip_postroute
      0.13 ±  3%      -0.1        0.05        perf-profile.self.cycles-pp._copy_from_iter_full
      0.15 ±  3%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.rcu_needs_cpu
      0.16 ±  2%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.prepare_to_wait_exclusive
      0.17 ±  5%      -0.1        0.09 ±  4%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.14 ±  5%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.netif_skb_features
      0.10 ±  4%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.put_prev_entity
      0.14 ±  3%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__might_fault
      0.14 ±  3%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.menu_reflect
      0.16 ±  5%      -0.1        0.09        perf-profile.self.cycles-pp.rb_erase_cached
      0.16 ±  2%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.ip_rcv_core
      0.15 ±  3%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.ip_rcv
      0.09 ±  4%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.udp_rcv
      0.15            -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.ip_local_deliver
      0.13 ±  3%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.09            -0.1        0.03 ±100%  perf-profile.self.cycles-pp.selinux_ipv4_output
      0.09 ±  7%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.deactivate_task
      0.12 ±  3%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.12            -0.1        0.06        perf-profile.self.cycles-pp.selinux_sk_getsecid
      0.11 ±  4%      -0.1        0.05        perf-profile.self.cycles-pp.ip_make_skb
      0.10 ± 11%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.__update_idle_core
      0.08            -0.1        0.03 ±100%  perf-profile.self.cycles-pp.ip_send_check
      0.11 ±  4%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.pm_qos_request
      0.11 ±  7%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.compute_score
      0.11 ±  7%      -0.1        0.05 ±  9%  perf-profile.self.cycles-pp.sock_sendmsg
      0.11 ±  7%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.ip_generic_getfrag
      0.11 ±  3%      -0.1        0.06        perf-profile.self.cycles-pp.switch_mm
      0.11 ±  4%      -0.1        0.06 ±  9%  perf-profile.self.cycles-pp.nf_hook_slow
      0.08 ±  6%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.alloc_skb_with_frags
      0.10 ±  7%      -0.1        0.05        perf-profile.self.cycles-pp.__skb_try_recv_from_queue
      0.10            -0.1        0.05        perf-profile.self.cycles-pp.dst_release
      0.12 ± 10%      -0.0        0.07        perf-profile.self.cycles-pp.import_single_range
      0.11 ±  6%      -0.0        0.06        perf-profile.self.cycles-pp.rb_insert_color_cached
      0.10 ±  4%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.12 ±  5%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.09 ±  4%      -0.0        0.05        perf-profile.self.cycles-pp.ip_finish_output
      0.10            -0.0        0.06        perf-profile.self.cycles-pp.ip_send_skb
      0.09 ±  7%      -0.0        0.05        perf-profile.self.cycles-pp.schedule_idle
      0.09 ±  4%      -0.0        0.05        perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.09 ±  4%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.udp_queue_rcv_skb
      0.08 ±  6%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.09 ±  5%      -0.0        0.05        perf-profile.self.cycles-pp.dev_hard_start_xmit
      0.08 ±  5%      -0.0        0.05        perf-profile.self.cycles-pp.ip_setup_cork
      0.10 ±  7%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.do_softirq_own_stack
      0.07            -0.0        0.05        perf-profile.self.cycles-pp.receiver_wake_function
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.cpumask_next
      0.00            +1.0        1.01        perf-profile.self.cycles-pp.__percpu_counter_sum
     30.01            +2.7       32.69        perf-profile.self.cycles-pp.intel_idle
      0.06 ± 87%     +30.2       30.31        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

***************************************************************************************************
lkp-bdw-ep2: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/25%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ep2/UDP_RR/netperf

commit: 
  6da410d97f (" mlx5e-fixes-2018-09-05")
  b99259a614 ("net/sock: move memory_allocated over to percpu_counter variables")

6da410d97ffa486e b99259a61450bb6403bbbbf279 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          9:4          -43%           7:4     perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout
          3:4          -15%           3:4     perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.switch_mm.__sched_text_start.schedule.schedule_timeout
         26:4         -109%          21:4     perf-profile.children.cycles-pp.schedule_timeout
          1:4          -11%           1:4     perf-profile.self.cycles-pp.schedule_timeout
         %stddev     %change         %stddev
             \          |                \  
   1930589           -13.5%    1670226        netperf.Throughput_total_tps
     87754           -13.5%      75919        netperf.Throughput_tps
      1426            -3.9%       1371 ±  2%  netperf.time.percent_of_cpu_this_job_got
      3911            -2.6%       3809 ±  2%  netperf.time.system_time
    385.12           -17.0%     319.68 ±  4%  netperf.time.user_time
 5.792e+08           -13.5%  5.011e+08        netperf.time.voluntary_context_switches
 5.792e+08           -13.5%  5.011e+08        netperf.workload
 5.792e+08           -12.5%  5.068e+08        softirqs.NET_RX
   7590488           -13.1%    6595363        vmstat.system.cs
      2095 ± 50%    +740.9%      17617 ± 87%  numa-meminfo.node0.Shmem
     40809 ±  4%     -14.3%      34971 ± 11%  numa-meminfo.node1.SReclaimable
    415.39            -1.8%     408.09        pmeter.Average_Active_Power
    211.25           -11.9%     186.02        pmeter.performance_per_watt
     18997 ±  6%      +8.2%      20555 ±  3%  sched_debug.cpu.load.avg
  13008831           -13.3%   11273636        sched_debug.cpu.nr_switches.avg
 1.133e+09           -12.8%  9.877e+08        cpuidle.C1.usage
 8.879e+09 ± 18%     -43.5%  5.013e+09 ± 54%  cpuidle.C6.time
  25831364 ±  3%     -27.4%   18752863        cpuidle.POLL.usage
      7.97            +2.5       10.43        mpstat.cpu.soft%
     17.09            -2.4       14.66        mpstat.cpu.sys%
      2.34            -0.3        2.00 ±  2%  mpstat.cpu.usr%
      1037            -3.8%     998.50        turbostat.Avg_MHz
 1.133e+09           -12.8%  9.877e+08        turbostat.C1
     33.21 ± 18%     -14.4       18.80 ± 55%  turbostat.C6%
      0.96 ± 96%    +201.3%       2.88 ± 30%  turbostat.CPU%c3
    216.23            -2.2%     211.46        turbostat.PkgWatt
    523.00 ± 50%    +742.2%       4404 ± 87%  numa-vmstat.node0.nr_shmem
    471766 ± 12%     +27.2%     600222 ±  6%  numa-vmstat.node0.numa_hit
    454541 ± 12%     +28.6%     584644 ±  7%  numa-vmstat.node0.numa_local
     10202 ±  4%     -14.3%       8743 ± 11%  numa-vmstat.node1.nr_slab_reclaimable
    665328 ±  9%     -20.2%     530868 ±  7%  numa-vmstat.node1.numa_hit
    532036 ± 11%     -25.5%     396167 ±  9%  numa-vmstat.node1.numa_local
 3.597e+12            -6.9%  3.348e+12        perf-stat.branch-instructions
      2.70            -0.2        2.54        perf-stat.branch-miss-rate%
 9.714e+10           -12.4%  8.509e+10        perf-stat.branch-misses
 3.173e+11 ±  2%     -14.9%  2.701e+11        perf-stat.cache-references
 2.317e+09           -13.3%   2.01e+09        perf-stat.context-switches
      1.59            +3.4%       1.64        perf-stat.cpi
 2.871e+13            -4.5%  2.742e+13        perf-stat.cpu-cycles
      0.09            -0.0        0.09        perf-stat.dTLB-load-miss-rate%
 4.816e+09           -10.7%  4.302e+09        perf-stat.dTLB-load-misses
  5.29e+12            -8.4%  4.846e+12        perf-stat.dTLB-loads
      0.00 ± 27%      +0.0        0.00 ±  9%  perf-stat.dTLB-store-miss-rate%
  50665696 ± 27%     +87.6%   95071493 ±  9%  perf-stat.dTLB-store-misses
 3.418e+12           -11.6%  3.023e+12        perf-stat.dTLB-stores
     11.55 ±  5%      +1.9       13.47 ±  7%  perf-stat.iTLB-load-miss-rate%
 4.068e+10           -16.8%  3.384e+10 ±  2%  perf-stat.iTLB-loads
  1.81e+13            -7.6%  1.673e+13        perf-stat.instructions
      0.63            -3.3%       0.61        perf-stat.ipc
  7.63e+08 ± 20%     +87.8%  1.433e+09 ±  3%  perf-stat.node-load-misses
     86.73 ±  2%     -27.7       59.01        perf-stat.node-store-miss-rate%
 5.359e+08 ±  9%     +17.5%  6.296e+08        perf-stat.node-store-misses
  83392223 ± 25%    +424.4%  4.373e+08        perf-stat.node-stores
     31255            +6.8%      33385        perf-stat.path-length
     13.66 ±  2%      -2.4       11.21 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.45 ±  2%      -2.4       11.05 ±  6%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.88 ±  2%      -2.1        9.73 ±  6%  perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.67 ±  2%      -2.1        9.58 ±  6%  perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      9.09 ±  2%      -1.6        7.45 ±  7%  perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      7.67 ±  2%      -1.4        6.27 ±  7%  perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
      9.02 ±  2%      -1.3        7.69 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv
      9.25 ±  2%      -1.3        7.99 ±  6%  perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish
      8.48 ±  2%      -1.2        7.24 ±  7%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb
      8.14 ±  3%      -1.1        6.99 ±  7%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb
      8.03 ±  3%      -1.1        6.89 ±  7%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      6.50 ±  2%      -1.1        5.41 ±  7%  perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg
      6.11 ±  2%      -1.0        5.12 ±  7%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg
      5.97 ±  2%      -1.0        5.02 ±  7%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp
      5.49            -0.9        4.60 ±  5%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.35            -0.9        4.48 ±  5%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      4.79            -0.8        3.96 ±  6%  perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      4.50 ±  2%      -0.7        3.85 ±  7%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      4.24 ±  2%      -0.6        3.62 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.50 ±  3%      -0.5        2.99 ±  8%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      2.62 ±  2%      -0.5        2.16 ±  7%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      2.88            -0.4        2.46 ±  6%  perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      1.68            -0.4        1.29 ±  4%  perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
      2.78 ±  3%      -0.4        2.42 ±  7%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.84            -0.4        1.48 ±  6%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      0.64 ±  3%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      1.89 ±  3%      -0.3        1.56 ±  7%  perf-profile.calltrace.cycles-pp.__entry_SYSCALL_64_trampoline
      2.04 ±  4%      -0.3        1.73 ±  8%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
      1.29 ±  2%      -0.3        1.00 ±  4%  perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg
      1.22 ±  2%      -0.3        0.94 ±  4%  perf-profile.calltrace.cycles-pp.ip_idents_reserve.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg
      1.32            -0.3        1.04 ±  6%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle
      2.11 ±  2%      -0.3        1.83 ±  6%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg
      1.77 ±  3%      -0.3        1.50 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.49            -0.3        1.22 ±  5%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      1.58 ±  4%      -0.2        1.36 ±  7%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb
      1.60            -0.2        1.40 ±  4%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry
      1.50 ±  4%      -0.2        1.30 ±  7%  perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
      0.60 ±  2%      -0.2        0.41 ± 58%  perf-profile.calltrace.cycles-pp.selinux_socket_sock_rcv_skb.security_sock_rcv_skb.sk_filter_trim_cap.udp_queue_rcv_skb.__udp4_lib_rcv
      1.48 ±  2%      -0.2        1.29 ±  7%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg
      1.41 ±  2%      -0.2        1.23 ±  7%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb
      0.90            -0.2        0.73 ±  4%  perf-profile.calltrace.cycles-pp._copy_to_iter.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.09 ±  3%      -0.2        0.93 ±  8%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.81 ±  3%      -0.2        0.66 ±  4%  perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2
      1.02 ±  3%      -0.2        0.87 ±  7%  perf-profile.calltrace.cycles-pp.switch_mm.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets
      0.94            -0.2        0.79 ±  6%  perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72            -0.2        0.57 ±  4%  perf-profile.calltrace.cycles-pp.__switch_to_asm
      0.90 ±  3%      -0.2        0.75 ±  5%  perf-profile.calltrace.cycles-pp.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      0.70 ±  3%      -0.2        0.54 ±  4%  perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.71 ±  2%      -0.1        0.58 ±  9%  perf-profile.calltrace.cycles-pp.nf_hook_slow.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg
      0.94 ±  3%      -0.1        0.82 ±  8%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.99 ±  4%      -0.1        0.88 ±  5%  perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      1.07 ±  5%      -0.1        0.96 ±  5%  perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.73 ±  3%      -0.1        0.62 ±  4%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      1.02 ±  5%      -0.1        0.92 ±  5%  perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.68 ±  2%      -0.1        0.59 ±  7%  perf-profile.calltrace.cycles-pp.sk_filter_trim_cap.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver
      0.69 ±  4%      -0.1        0.60 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.64 ±  2%      -0.1        0.55 ±  6%  perf-profile.calltrace.cycles-pp.security_sock_rcv_skb.sk_filter_trim_cap.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish
     26.69 ±  2%      +1.9       28.61 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     26.51 ±  2%      +1.9       28.45 ±  5%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.75 ±  2%      +2.1       27.80 ±  5%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.20 ±  2%      +2.2       27.38 ±  5%  perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     18.57 ±  2%      +3.3       21.89 ±  5%  perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
     18.32 ±  2%      +3.4       21.68 ±  5%  perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto
     17.95 ±  2%      +3.4       21.32 ±  5%  perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg
     16.70 ±  2%      +3.6       20.34 ±  5%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg
     14.36 ±  2%      +4.0       18.32 ±  5%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb
     14.21 ±  2%      +4.0       18.22 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb
     13.99 ±  2%      +4.0       18.01 ±  5%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     14.04 ±  2%      +4.0       18.07 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     13.60 ±  2%      +4.1       17.66 ±  5%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     13.30 ±  2%      +4.1       17.42 ±  5%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
      0.00            +4.1        4.14 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb
     12.53 ±  2%      +4.2       16.77 ±  5%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     12.28 ±  2%      +4.2       16.52 ±  5%  perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     11.88 ±  3%      +4.3       16.14 ±  5%  perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     11.73 ±  3%      +4.3       16.00 ±  5%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     11.40 ±  2%      +4.4       15.77 ±  6%  perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
      0.00            +4.4        4.44 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb
     10.89 ±  2%      +4.4       15.34 ±  6%  perf-profile.calltrace.cycles-pp.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv
      9.96 ±  2%      +4.6       14.57 ±  6%  perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver
      0.00            +5.8        5.80 ±  5%  perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv
      0.00            +6.0        6.01 ±  5%  perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish
     13.67 ±  2%      -2.5       11.22 ±  6%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
     13.46 ±  2%      -2.4       11.06 ±  6%  perf-profile.children.cycles-pp.__sys_recvfrom
     11.88 ±  2%      -2.1        9.73 ±  6%  perf-profile.children.cycles-pp.inet_recvmsg
     11.69 ±  2%      -2.1        9.59 ±  6%  perf-profile.children.cycles-pp.udp_recvmsg
     11.45            -1.8        9.62 ±  6%  perf-profile.children.cycles-pp.__sched_text_start
      9.11 ±  2%      -1.6        7.47 ±  7%  perf-profile.children.cycles-pp.__skb_recv_udp
      7.67 ±  2%      -1.4        6.28 ±  7%  perf-profile.children.cycles-pp.__skb_wait_for_more_packets
      9.03 ±  2%      -1.3        7.70 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
      9.25 ±  3%      -1.3        7.99 ±  6%  perf-profile.children.cycles-pp.sock_def_readable
      8.48 ±  2%      -1.2        7.24 ±  7%  perf-profile.children.cycles-pp.__wake_up_common
      8.06 ±  3%      -1.2        6.90 ±  7%  perf-profile.children.cycles-pp.try_to_wake_up
      8.15 ±  3%      -1.1        7.00 ±  7%  perf-profile.children.cycles-pp.autoremove_wake_function
      6.11 ±  2%      -1.0        5.12 ±  7%  perf-profile.children.cycles-pp.schedule
      5.51            -0.9        4.62 ±  5%  perf-profile.children.cycles-pp.schedule_idle
      4.80            -0.8        3.96 ±  6%  perf-profile.children.cycles-pp.ip_make_skb
      4.50 ±  2%      -0.6        3.86 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      4.25 ±  2%      -0.6        3.63 ±  8%  perf-profile.children.cycles-pp.enqueue_task_fair
      3.66 ±  3%      -0.5        3.14 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      2.65 ±  2%      -0.5        2.18 ±  7%  perf-profile.children.cycles-pp.dequeue_task_fair
      2.32 ±  2%      -0.4        1.88 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      2.38 ±  2%      -0.4        1.95 ±  8%  perf-profile.children.cycles-pp.dequeue_entity
      2.90            -0.4        2.47 ±  6%  perf-profile.children.cycles-pp.__ip_append_data
      2.31 ±  3%      -0.4        1.91 ±  6%  perf-profile.children.cycles-pp.__entry_SYSCALL_64_trampoline
      1.68            -0.4        1.29 ±  4%  perf-profile.children.cycles-pp.__ip_make_skb
      2.81 ±  3%      -0.4        2.44 ±  7%  perf-profile.children.cycles-pp.menu_select
      2.60            -0.4        2.24 ±  5%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      2.42 ±  3%      -0.3        2.11 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      2.04 ±  4%      -0.3        1.74 ±  7%  perf-profile.children.cycles-pp.__dev_queue_xmit
      1.69 ±  2%      -0.3        1.40 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.29 ±  2%      -0.3        1.00 ±  4%  perf-profile.children.cycles-pp.__ip_select_ident
      1.23 ±  2%      -0.3        0.94 ±  4%  perf-profile.children.cycles-pp.ip_idents_reserve
      1.34            -0.3        1.06 ±  6%  perf-profile.children.cycles-pp.set_next_entity
      2.12 ±  2%      -0.3        1.84 ±  6%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      1.79 ±  3%      -0.3        1.52 ±  8%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.48 ±  2%      -0.3        1.23 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      1.24 ±  2%      -0.2        1.03 ±  6%  perf-profile.children.cycles-pp.load_new_mm_cr3
      1.58 ±  4%      -0.2        1.37 ±  7%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.51 ±  4%      -0.2        1.31 ±  7%  perf-profile.children.cycles-pp.loopback_xmit
      1.48 ±  2%      -0.2        1.29 ±  7%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      0.83 ±  3%      -0.2        0.66 ±  8%  perf-profile.children.cycles-pp.update_curr
      1.42 ±  2%      -0.2        1.24 ±  7%  perf-profile.children.cycles-pp.__alloc_skb
      0.73 ±  3%      -0.2        0.55 ±  4%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.91            -0.2        0.73 ±  4%  perf-profile.children.cycles-pp._copy_to_iter
      1.07 ±  2%      -0.2        0.90 ±  6%  perf-profile.children.cycles-pp.switch_mm
      1.10 ±  3%      -0.2        0.94 ±  8%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.81 ±  3%      -0.2        0.66 ±  4%  perf-profile.children.cycles-pp.sock_wfree
      0.95            -0.2        0.80 ±  6%  perf-profile.children.cycles-pp.move_addr_to_user
      0.90 ±  2%      -0.2        0.75 ±  5%  perf-profile.children.cycles-pp.__consume_stateless_skb
      0.74            -0.2        0.59 ±  4%  perf-profile.children.cycles-pp.__switch_to_asm
      0.78            -0.1        0.64 ±  6%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.49            -0.1        0.35 ±  5%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.66 ±  2%      -0.1        0.53 ±  5%  perf-profile.children.cycles-pp.__switch_to
      0.84 ±  3%      -0.1        0.70 ±  8%  perf-profile.children.cycles-pp.nf_hook_slow
      0.71 ±  3%      -0.1        0.58 ±  9%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.95 ±  3%      -0.1        0.82 ±  7%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.68 ±  3%      -0.1        0.55 ± 12%  perf-profile.children.cycles-pp.update_cfs_group
      0.57 ±  4%      -0.1        0.44 ±  7%  perf-profile.children.cycles-pp.prepare_to_wait_exclusive
      0.43 ±  7%      -0.1        0.31 ± 10%  perf-profile.children.cycles-pp.__enqueue_entity
      0.69 ±  3%      -0.1        0.56 ±  5%  perf-profile.children.cycles-pp.__slab_free
      0.43            -0.1        0.31 ±  7%  perf-profile.children.cycles-pp.ip_generic_getfrag
      0.74 ±  3%      -0.1        0.63 ±  4%  perf-profile.children.cycles-pp.poll_idle
      0.39 ±  2%      -0.1        0.28 ±  7%  perf-profile.children.cycles-pp._copy_from_iter_full
      1.07 ±  5%      -0.1        0.97 ±  5%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      1.04 ±  5%      -0.1        0.94 ±  5%  perf-profile.children.cycles-pp.check_preempt_curr
      0.54            -0.1        0.44 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      0.57 ±  2%      -0.1        0.47 ±  9%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.53 ±  2%      -0.1        0.43 ±  4%  perf-profile.children.cycles-pp.sock_def_write_space
      0.68 ±  2%      -0.1        0.59 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.57            -0.1        0.48 ±  5%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.53            -0.1        0.44 ±  4%  perf-profile.children.cycles-pp.copyout
      0.64 ±  2%      -0.1        0.55 ±  6%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.56            -0.1        0.47 ±  4%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.69 ±  5%      -0.1        0.60 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.53 ±  2%      -0.1        0.44 ±  4%  perf-profile.children.cycles-pp.sched_clock
      0.39 ±  5%      -0.1        0.30 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.35 ±  4%      -0.1        0.26 ±  6%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.42            -0.1        0.34 ±  5%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.61 ±  2%      -0.1        0.53 ±  6%  perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb
      0.38 ±  6%      -0.1        0.30 ± 14%  perf-profile.children.cycles-pp.___perf_sw_event
      0.50 ±  2%      -0.1        0.42 ±  5%  perf-profile.children.cycles-pp.native_sched_clock
      0.40 ±  2%      -0.1        0.33 ±  4%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.41 ±  3%      -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.sockfd_lookup_light
      0.45 ±  3%      -0.1        0.37 ± 11%  perf-profile.children.cycles-pp.selinux_ip_postroute_compat
      0.45            -0.1        0.37 ±  8%  perf-profile.children.cycles-pp.native_write_msr
      0.56 ±  4%      -0.1        0.48 ±  6%  perf-profile.children.cycles-pp.finish_task_switch
      0.43            -0.1        0.36 ±  9%  perf-profile.children.cycles-pp.ksize
      0.30 ±  6%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.__udp4_lib_lookup
      0.33 ±  3%      -0.1        0.27 ±  8%  perf-profile.children.cycles-pp.__fget_light
      0.31            -0.1        0.25 ± 11%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.20            -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.copyin
      0.35            -0.1        0.28 ±  8%  perf-profile.children.cycles-pp.__might_fault
      0.28 ±  9%      -0.1        0.22 ±  5%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.21 ±  5%      -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2
      0.49            -0.1        0.43 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.27 ±  3%      -0.1        0.21 ±  7%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.09 ±  8%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.kfree
      0.24 ±  5%      -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.rb_erase_cached
      0.44 ±  2%      -0.1        0.39 ±  5%  perf-profile.children.cycles-pp.reweight_entity
      0.26 ±  3%      -0.1        0.21 ±  7%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.24            -0.0        0.20 ±  5%  perf-profile.children.cycles-pp.security_socket_sendmsg
      0.21 ±  5%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.kmem_cache_free
      0.15 ±  7%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.07 ± 12%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.rb_next
      0.24            -0.0        0.20 ±  9%  perf-profile.children.cycles-pp.___might_sleep
      0.18 ± 12%      -0.0        0.14        perf-profile.children.cycles-pp.inet_sendmsg
      0.20 ±  4%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.19 ±  2%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.17 ±  7%      -0.0        0.13 ± 11%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.14 ± 10%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.compute_score
      0.22 ±  4%      -0.0        0.19 ± 10%  perf-profile.children.cycles-pp.selinux_parse_skb
      0.29 ±  4%      -0.0        0.25 ±  9%  perf-profile.children.cycles-pp.ipv4_mtu
      0.16 ±  4%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.__calc_delta
      0.24            -0.0        0.21 ±  8%  perf-profile.children.cycles-pp.move_addr_to_kernel
      0.22 ±  4%      -0.0        0.19 ±  3%  perf-profile.children.cycles-pp.sock_has_perm
      0.23 ±  3%      -0.0        0.19 ±  9%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.26 ±  3%      -0.0        0.22 ±  4%  perf-profile.children.cycles-pp.pick_next_entity
      0.21 ±  3%      -0.0        0.18 ±  7%  perf-profile.children.cycles-pp._copy_from_user
      0.19 ±  6%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.__skb_try_recv_from_queue
      0.20 ±  2%      -0.0        0.17 ±  5%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.18 ±  2%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.security_sk_classify_flow
      0.15 ±  7%      -0.0        0.11 ±  7%  perf-profile.children.cycles-pp.menu_reflect
      0.10 ±  5%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.selinux_socket_sendmsg
      0.19 ±  2%      -0.0        0.16 ±  7%  perf-profile.children.cycles-pp._copy_to_user
      0.19 ±  4%      -0.0        0.16 ±  8%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.17 ±  4%      -0.0        0.15 ± 11%  perf-profile.children.cycles-pp.__might_sleep
      0.17 ±  6%      -0.0        0.14        perf-profile.children.cycles-pp.sock_recvmsg
      0.10 ±  7%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.ipv4_pktinfo_prepare
      0.14 ±  7%      -0.0        0.12        perf-profile.children.cycles-pp.security_socket_recvmsg
      0.13 ±  3%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp.__get_user_4
      0.13 ±  3%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp.selinux_sk_getsecid
      0.11 ±  8%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.dst_release
      0.10 ±  8%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.selinux_ipv4_postroute
      0.11 ±  4%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp.pm_qos_request
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.udp_rcv
      0.09 ±  4%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.ip_send_check
      0.05            +0.0        0.07 ± 13%  perf-profile.children.cycles-pp.selinux_netlbl_sock_rcv_skb
      0.06 ± 11%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.07 ±  7%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.09 ±  4%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.15 ±  8%      +0.0        0.20 ±  8%  perf-profile.children.cycles-pp.irq_exit
      0.11 ± 31%      +0.1        0.16 ± 22%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.netlbl_enabled
      0.45 ±  5%      +0.1        0.57 ±  7%  perf-profile.children.cycles-pp.find_next_bit
      0.00            +0.2        0.17 ±  4%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +0.3        0.34 ±  5%  perf-profile.children.cycles-pp.cpumask_next
     26.71 ±  2%      +1.9       28.63 ±  5%  perf-profile.children.cycles-pp.__x64_sys_sendto
     26.51 ±  2%      +1.9       28.46 ±  5%  perf-profile.children.cycles-pp.__sys_sendto
     25.75 ±  2%      +2.1       27.81 ±  5%  perf-profile.children.cycles-pp.sock_sendmsg
     25.23 ±  2%      +2.2       27.40 ±  5%  perf-profile.children.cycles-pp.udp_sendmsg
     18.58 ±  2%      +3.3       21.90 ±  5%  perf-profile.children.cycles-pp.udp_send_skb
     18.33 ±  2%      +3.4       21.68 ±  5%  perf-profile.children.cycles-pp.ip_send_skb
     17.95 ±  2%      +3.4       21.33 ±  5%  perf-profile.children.cycles-pp.ip_output
     16.73 ±  2%      +3.6       20.36 ±  5%  perf-profile.children.cycles-pp.ip_finish_output2
     14.40 ±  2%      +4.0       18.36 ±  5%  perf-profile.children.cycles-pp.__local_bh_enable_ip
     14.24 ±  2%      +4.0       18.25 ±  5%  perf-profile.children.cycles-pp.do_softirq
     14.05 ±  2%      +4.0       18.08 ±  5%  perf-profile.children.cycles-pp.do_softirq_own_stack
     14.12 ±  2%      +4.0       18.16 ±  5%  perf-profile.children.cycles-pp.__softirqentry_text_start
     13.61 ±  2%      +4.1       17.67 ±  5%  perf-profile.children.cycles-pp.net_rx_action
     13.31 ±  2%      +4.1       17.43 ±  5%  perf-profile.children.cycles-pp.process_backlog
      1.50 ±  2%      +4.2        5.68 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +4.2        4.19 ±  5%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     12.28 ±  2%      +4.2       16.52 ±  5%  perf-profile.children.cycles-pp.ip_rcv
     12.53 ±  2%      +4.2       16.78 ±  5%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     11.89 ±  3%      +4.3       16.15 ±  5%  perf-profile.children.cycles-pp.ip_local_deliver
     11.73 ±  3%      +4.3       16.00 ±  5%  perf-profile.children.cycles-pp.ip_local_deliver_finish
     11.41 ±  2%      +4.4       15.78 ±  6%  perf-profile.children.cycles-pp.__udp4_lib_rcv
     10.90 ±  2%      +4.4       15.35 ±  6%  perf-profile.children.cycles-pp.udp_queue_rcv_skb
      9.97 ±  2%      +4.6       14.58 ±  6%  perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
      0.00            +5.8        5.84 ±  5%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.15 ±  3%      +5.9        6.01 ±  5%  perf-profile.children.cycles-pp.__sk_mem_raise_allocated
      2.25 ±  3%      -0.4        1.85 ±  6%  perf-profile.self.cycles-pp.__entry_SYSCALL_64_trampoline
      2.20            -0.3        1.89 ±  5%  perf-profile.self.cycles-pp.__sched_text_start
      1.69 ±  2%      -0.3        1.39 ±  5%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.22 ±  2%      -0.3        0.93 ±  4%  perf-profile.self.cycles-pp.ip_idents_reserve
      1.43 ±  2%      -0.3        1.16 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      1.24 ±  2%      -0.2        1.03 ±  6%  perf-profile.self.cycles-pp.load_new_mm_cr3
      1.01            -0.2        0.82 ±  7%  perf-profile.self.cycles-pp.do_idle
      0.45            -0.2        0.29 ±  5%  perf-profile.self.cycles-pp.__skb_wait_for_more_packets
      1.05 ±  2%      -0.2        0.90 ±  6%  perf-profile.self.cycles-pp.update_load_avg
      0.74            -0.2        0.59 ±  4%  perf-profile.self.cycles-pp.__switch_to_asm
      0.60 ±  2%      -0.1        0.45 ±  6%  perf-profile.self.cycles-pp.udp_sendmsg
      0.76            -0.1        0.62 ±  5%  perf-profile.self.cycles-pp.enqueue_entity
      1.35            -0.1        1.21 ±  5%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.63 ±  2%      -0.1        0.50 ±  5%  perf-profile.self.cycles-pp.__switch_to
      0.76            -0.1        0.62 ±  6%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.67 ±  3%      -0.1        0.54 ± 12%  perf-profile.self.cycles-pp.update_cfs_group
      0.56 ±  3%      -0.1        0.44 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.68 ±  3%      -0.1        0.56 ±  5%  perf-profile.self.cycles-pp.__slab_free
      0.42 ±  7%      -0.1        0.30 ± 11%  perf-profile.self.cycles-pp.__enqueue_entity
      0.66            -0.1        0.54 ±  5%  perf-profile.self.cycles-pp.set_next_entity
      0.34 ±  2%      -0.1        0.23 ±  3%  perf-profile.self.cycles-pp.ip_output
      0.59 ±  2%      -0.1        0.49 ±  4%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.53 ±  3%      -0.1        0.42 ±  9%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.51            -0.1        0.42 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.64            -0.1        0.55 ±  4%  perf-profile.self.cycles-pp.poll_idle
      0.54            -0.1        0.45 ±  5%  perf-profile.self.cycles-pp.move_addr_to_user
      0.63 ±  2%      -0.1        0.54 ±  6%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.40 ±  4%      -0.1        0.31 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.56            -0.1        0.47 ±  4%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.52 ±  2%      -0.1        0.43 ±  5%  perf-profile.self.cycles-pp.sock_def_write_space
      0.20 ±  6%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.check_preempt_curr
      0.68 ±  5%      -0.1        0.60 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.53 ±  5%      -0.1        0.45 ±  5%  perf-profile.self.cycles-pp.udp_recvmsg
      0.38 ±  5%      -0.1        0.30 ±  8%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.31 ±  2%      -0.1        0.23 ±  8%  perf-profile.self.cycles-pp.__skb_recv_udp
      0.51 ±  2%      -0.1        0.43 ± 10%  perf-profile.self.cycles-pp.dequeue_entity
      0.26 ±  4%      -0.1        0.18 ±  6%  perf-profile.self.cycles-pp.__ip_make_skb
      0.48 ±  2%      -0.1        0.41 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      0.29 ±  4%      -0.1        0.22 ±  5%  perf-profile.self.cycles-pp._copy_to_iter
      0.44            -0.1        0.37 ±  7%  perf-profile.self.cycles-pp.native_write_msr
      0.44            -0.1        0.37 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.40            -0.1        0.33 ±  4%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.25 ±  4%      -0.1        0.18 ±  8%  perf-profile.self.cycles-pp.__wake_up_common
      0.42            -0.1        0.35 ±  9%  perf-profile.self.cycles-pp.ksize
      0.32            -0.1        0.26 ±  9%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.28 ±  3%      -0.1        0.21 ±  5%  perf-profile.self.cycles-pp.sock_wfree
      0.32 ±  2%      -0.1        0.26 ±  8%  perf-profile.self.cycles-pp.__fget_light
      0.31            -0.1        0.25 ±  5%  perf-profile.self.cycles-pp.update_rq_clock
      0.49            -0.1        0.43 ±  6%  perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb
      0.21 ±  4%      -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.__sys_recvfrom
      0.29 ±  5%      -0.1        0.24 ±  6%  perf-profile.self.cycles-pp.try_to_wake_up
      0.09 ±  8%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.kfree
      0.20 ±  7%      -0.1        0.15 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_stage2
      0.43 ±  2%      -0.0        0.39 ±  6%  perf-profile.self.cycles-pp.reweight_entity
      0.35 ±  4%      -0.0        0.31 ±  6%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.17 ±  2%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.25 ±  4%      -0.0        0.21 ±  7%  perf-profile.self.cycles-pp.net_rx_action
      0.08 ±  5%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.tick_nohz_idle_exit
      0.26 ±  4%      -0.0        0.21 ±  7%  perf-profile.self.cycles-pp.__sys_sendto
      0.20 ±  4%      -0.0        0.16 ±  5%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.12 ±  4%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.udp_queue_rcv_skb
      0.27            -0.0        0.22 ±  7%  perf-profile.self.cycles-pp.loopback_xmit
      0.18 ±  7%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.inet_recvmsg
      0.14 ± 10%      -0.0        0.10 ±  8%  perf-profile.self.cycles-pp.compute_score
      0.19 ±  6%      -0.0        0.15 ±  8%  perf-profile.self.cycles-pp.selinux_socket_sock_rcv_skb
      0.15 ± 13%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.ip_local_deliver_finish
      0.20 ±  4%      -0.0        0.16 ±  7%  perf-profile.self.cycles-pp.rb_erase_cached
      0.22 ±  5%      -0.0        0.18 ±  8%  perf-profile.self.cycles-pp.__alloc_skb
      0.18 ±  2%      -0.0        0.14 ±  9%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.20 ±  2%      -0.0        0.16 ±  8%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.18 ±  3%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.17 ± 15%      -0.0        0.13        perf-profile.self.cycles-pp.inet_sendmsg
      0.29 ±  2%      -0.0        0.26 ±  4%  perf-profile.self.cycles-pp.process_backlog
      0.23 ±  3%      -0.0        0.19 ±  2%  perf-profile.self.cycles-pp.pick_next_entity
      0.15 ±  5%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.selinux_ip_postroute_compat
      0.16 ±  6%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__calc_delta
      0.11 ±  3%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.09 ±  5%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp._copy_from_iter_full
      0.09 ±  5%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.selinux_socket_sendmsg
      0.22 ±  5%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.sock_has_perm
      0.15 ±  5%      -0.0        0.12 ±  4%  perf-profile.self.cycles-pp.__udp4_lib_lookup
      0.15 ±  4%      -0.0        0.12 ± 10%  perf-profile.self.cycles-pp.__might_sleep
      0.09 ±  4%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.nf_hook_slow
      0.13 ±  6%      -0.0        0.10 ±  8%  perf-profile.self.cycles-pp.prepare_to_wait_exclusive
      0.15 ±  2%      -0.0        0.12 ±  5%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.08 ±  5%      -0.0        0.06 ± 15%  perf-profile.self.cycles-pp.ip_make_skb
      0.14 ±  6%      -0.0        0.11 ±  3%  perf-profile.self.cycles-pp.ip_rcv
      0.13            -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.selinux_ip_postroute
      0.10            -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.sock_sendmsg
      0.10            -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.__might_fault
      0.09 ±  9%      -0.0        0.07 ± 13%  perf-profile.self.cycles-pp.put_prev_task_fair
      0.12 ±  3%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.12 ±  8%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.menu_reflect
      0.06            -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.__ip_local_out
      0.13 ±  3%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.selinux_sock_rcv_skb_compat
      0.08 ±  5%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.__skb_try_recv_from_queue
      0.09 ±  4%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.selinux_ipv4_postroute
      0.10 ±  9%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.dst_release
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.schedule
      0.13 ±  6%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.__get_user_4
      0.11 ±  7%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.netif_skb_features
      0.11 ±  7%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.18 ±  4%      -0.0        0.17 ±  5%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.11 ±  3%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.switch_mm
      0.12 ±  3%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.selinux_sk_getsecid
      0.09 ±  4%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.udp_rcv
      0.07            -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.update_min_vruntime
      0.09 ±  5%      -0.0        0.07        perf-profile.self.cycles-pp.ip_send_check
      0.08 ± 10%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.sockfd_lookup_light
      0.06 ± 11%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.netlbl_enabled
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.hrtimer_get_next_event
      0.21 ±  7%      +0.1        0.29 ±  3%  perf-profile.self.cycles-pp.sock_def_readable
      0.00            +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.cpumask_next
      0.38 ±  6%      +0.1        0.50 ±  7%  perf-profile.self.cycles-pp.find_next_bit
      0.00            +0.2        0.17 ±  4%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.00            +1.1        1.07 ±  5%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.00            +4.2        4.19 ±  5%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Thanks,
Rong Chen