Subject: [lkp] [inet] 34630bc073: netperf.Throughput_tps 4.2% improvement

From: kernel test robot <[email-masked]>
Date: Fri, 4 Nov 2016 09:56:50 +0800

Greeting,

FYI, we noticed a 4.2% improvement of netperf.Throughput_tps due to commit:

commit 34630bc07341e30e42a974840a0d622b31f1845c ("inet: fix sleeping inside inet_wait_for_connect()")
https://github.com/0day-ci/linux Cong-Wang/inet-fix-sleeping-inside-inet_wait_for_connect/20161102-070925

in testcase: netperf
on test machine: 8 threads Intel(R) Atom(TM) CPU C2750 @ 2.40GHz with 16G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 200%
	cluster: cs-localhost
	test: TCP_CRR
	cpufreq_governor: performance

Netperf is a benchmark that can be use to measure various aspect of networking performance.

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Details are as below:
-------------------------------------------------------------------------------------------------->

To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-6/performance/ipv4/x86_64-rhel-7.2/200%/debian-x86_64-2016-08-31.cgz/300s/lkp-avoton3/TCP_CRR/netperf

commit: 
  b9b84fc07d ("net: mv643xx_eth: ensure coalesce settings survive read-modify-write")
  34630bc073 ("inet: fix sleeping inside inet_wait_for_connect()")

b9b84fc07dbc6ac7 34630bc07341e30e42a974840a 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      2915 ±  0%      +4.2%       3039 ±  0%  netperf.Throughput_tps
  20900055 ±  0%     -97.4%     540516 ±  8%  netperf.time.involuntary_context_switches
    517.25 ±  0%     +13.8%     588.50 ±  0%  netperf.time.percent_of_cpu_this_job_got
      1536 ±  0%     +13.9%       1750 ±  0%  netperf.time.system_time
   7199745 ±  1%    +100.6%   14442880 ±  0%  netperf.time.voluntary_context_switches
     18454 ±  1%     +28.3%      23680 ±  4%  softirqs.SCHED
      0.08 ±  0%     +34.4%       0.11 ±  7%  turbostat.CPU%c1
    187728 ±  0%     -44.8%     103607 ±  0%  vmstat.system.cs
      9065 ±  1%     +24.6%      11296 ±  4%  vmstat.system.in
    118058 ± 13%    +582.9%     806220 ± 17%  cpuidle.C1-AVN.time
      2948 ± 28%    +822.4%      27193 ± 17%  cpuidle.C1-AVN.usage
     24.75 ± 16%    +238.4%      83.75 ± 20%  cpuidle.POLL.usage
      2121 ±  6%     -16.2%       1778 ±  6%  slabinfo.cred_jar.num_objs
      5215 ± 23%     +42.3%       7423 ±  3%  slabinfo.kmalloc-32.active_objs
      5215 ± 23%     +42.3%       7423 ±  3%  slabinfo.kmalloc-32.num_objs
 2.561e+10 ±  0%      -1.4%  2.526e+10 ±  0%  perf-stat.branch-misses
      1.27 ±  3%     +14.3%       1.45 ±  2%  perf-stat.cache-miss-rate%
 2.882e+09 ±  3%     +13.4%  3.269e+09 ±  2%  perf-stat.cache-misses
  57816048 ±  0%     -45.0%   31803360 ±  0%  perf-stat.context-switches
      8221 ±  2%     +24.9%      10268 ±  5%  perf-stat.cpu-migrations
 2.315e+10 ±  0%      -1.1%  2.289e+10 ±  0%  perf-stat.iTLB-load-misses
    285020 ±  0%      +4.2%     296904 ±  0%  perf-stat.minor-faults
    285022 ±  0%      +4.2%     296904 ±  0%  perf-stat.page-faults
    231706 ± 10%    +143.8%     564885 ± 13%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    645748 ± 13%     +49.9%     967850 ±  0%  sched_debug.cfs_rq:/.MIN_vruntime.max
    297027 ± 10%     +52.4%     452531 ±  3%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    231706 ± 10%    +143.8%     564885 ± 13%  sched_debug.cfs_rq:/.max_vruntime.avg
    645748 ± 13%     +49.9%     967851 ±  0%  sched_debug.cfs_rq:/.max_vruntime.max
    297027 ± 10%     +52.4%     452532 ±  3%  sched_debug.cfs_rq:/.max_vruntime.stddev
    769174 ±  0%     +24.3%     955802 ±  0%  sched_debug.cfs_rq:/.min_vruntime.avg
    786747 ±  1%     +23.4%     970602 ±  0%  sched_debug.cfs_rq:/.min_vruntime.max
    758458 ±  0%     +24.1%     941243 ±  0%  sched_debug.cfs_rq:/.min_vruntime.min
      1.23 ±  3%     +24.9%       1.54 ±  2%  sched_debug.cfs_rq:/.nr_running.avg
      1.79 ±  4%     +18.6%       2.12 ±  6%  sched_debug.cfs_rq:/.nr_running.max
      0.41 ±  8%     +21.7%       0.50 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
    107.00 ±  2%     -20.2%      85.38 ± 10%  sched_debug.cfs_rq:/.runnable_load_avg.min
     11.17 ± 14%     +66.3%      18.58 ± 23%  sched_debug.cfs_rq:/.util_avg.stddev
    390011 ± 29%     -55.3%     174458 ± 12%  sched_debug.cpu.avg_idle.avg
    813413 ± 18%     -65.5%     280413 ± 22%  sched_debug.cpu.avg_idle.max
    257758 ± 22%     -77.5%      58027 ± 35%  sched_debug.cpu.avg_idle.stddev
      3.38 ± 25%     -45.6%       1.84 ± 20%  sched_debug.cpu.clock.stddev
      3.38 ± 25%     -45.6%       1.84 ± 20%  sched_debug.cpu.clock_task.stddev
    115535 ±  4%     +16.2%     134278 ±  7%  sched_debug.cpu.load.min
      0.58 ±  3%     +22.7%       0.71 ±  8%  sched_debug.cpu.nr_running.stddev
   3579054 ±  0%     -45.0%    1966781 ±  0%  sched_debug.cpu.nr_switches.avg
   3645312 ±  0%     -43.6%    2056851 ±  0%  sched_debug.cpu.nr_switches.max
   3525172 ±  0%     -45.9%    1908138 ±  0%  sched_debug.cpu.nr_switches.min
      0.00 ± 18%     -88.7%       0.00 ± 58%  sched_debug.rt_rq:/.rt_time.min
     15.67 ±  0%     -17.4%      12.94 ±  1%  perf-profile.calltrace.cycles-pp.SYSC_recvfrom.sys_recvfrom.entry_SYSCALL_64_fastpath
     16.23 ±  0%     +19.3%      19.36 ±  0%  perf-profile.calltrace.cycles-pp.SYSC_sendto.sys_sendto.entry_SYSCALL_64_fastpath
      2.41 ±  3%     +10.7%       2.67 ±  1%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_close.inet_release.sock_release
      0.69 ±  2%    +137.3%       1.64 ±  3%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      1.17 ±  4%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__inet_stream_connect.inet_stream_connect
      2.06 ±  1%     -13.5%       1.78 ±  2%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg
      9.07 ±  0%     -28.3%       6.51 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
      0.82 ±  1%    +109.5%       1.71 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
     12.40 ±  0%     +16.7%      14.46 ±  1%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      1.94 ±  3%     -25.4%       1.45 ±  1%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish
      2.04 ±  3%     -25.8%       1.51 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      1.78 ±  3%     -14.8%       1.52 ±  1%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
      1.87 ±  3%     -26.0%       1.39 ±  1%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv
      1.82 ±  3%     -25.9%       1.35 ±  1%  perf-profile.calltrace.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue
      2.00 ±  3%     -24.8%       1.51 ±  3%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.dev_queue_xmit.ip_finish_output2.ip_finish_output
      1.15 ±  5%     -65.9%       0.39 ± 57%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.57 ±  3%     -14.8%       1.34 ±  1%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
     14.95 ±  0%     -18.8%      12.14 ±  1%  perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.entry_SYSCALL_64_fastpath
     15.71 ±  0%     +20.0%      18.84 ±  0%  perf-profile.calltrace.cycles-pp.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto.entry_SYSCALL_64_fastpath
      7.37 ±  0%     -17.4%       6.08 ±  1%  perf-profile.calltrace.cycles-pp.ip_local_out.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check
      7.59 ±  0%     -16.9%       6.31 ±  1%  perf-profile.calltrace.cycles-pp.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established
      9.50 ±  0%     +13.6%      10.79 ±  1%  perf-profile.calltrace.cycles-pp.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push
      2.53 ±  2%     +10.4%       2.79 ±  1%  perf-profile.calltrace.cycles-pp.release_sock.tcp_close.inet_release.sock_release.sock_close
      0.82 ±  2%    +117.7%       1.79 ±  2%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
      1.24 ±  3%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__inet_stream_connect.inet_stream_connect.SYSC_connect
      2.17 ±  1%     -12.8%       1.89 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg
      1.28 ±  3%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.schedule_timeout.__inet_stream_connect.inet_stream_connect.SYSC_connect.sys_connect
      2.21 ±  1%     -13.0%       1.93 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      0.81 ± 25%     +54.9%       1.25 ±  2%  perf-profile.calltrace.cycles-pp.security_sock_rcv_skb.sk_filter_trim_cap.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      1.19 ± 23%     +53.6%       1.82 ± 14%  perf-profile.calltrace.cycles-pp.sk_filter_trim_cap.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
      2.67 ±  0%     -15.5%       2.26 ±  2%  perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
     15.30 ±  0%     -18.1%      12.53 ±  1%  perf-profile.calltrace.cycles-pp.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.entry_SYSCALL_64_fastpath
     15.96 ±  0%     +19.6%      19.08 ±  0%  perf-profile.calltrace.cycles-pp.sock_sendmsg.SYSC_sendto.sys_sendto.entry_SYSCALL_64_fastpath
     15.77 ±  0%     -17.4%      13.02 ±  1%  perf-profile.calltrace.cycles-pp.sys_recvfrom.entry_SYSCALL_64_fastpath
     16.29 ±  0%     +19.2%      19.41 ±  0%  perf-profile.calltrace.cycles-pp.sys_sendto.entry_SYSCALL_64_fastpath
      1.11 ±  2%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      1.81 ±  2%      +6.9%       1.93 ±  4%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      1.86 ±  1%     +34.3%       2.50 ±  1%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      0.26 ±173%    +211.4%       0.82 ± 37%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      0.55 ±  3%    +106.3%       1.14 ±  1%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
      1.12 ±  2%     -19.6%       0.90 ±  6%  perf-profile.calltrace.cycles-pp.tcp_finish_connect.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      2.58 ±  2%     -26.3%       1.90 ±  4%  perf-profile.calltrace.cycles-pp.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
     10.67 ±  0%     -24.3%       8.08 ±  2%  perf-profile.calltrace.cycles-pp.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
     12.42 ±  0%     +16.7%      14.49 ±  1%  perf-profile.calltrace.cycles-pp.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
      0.57 ±  1%    +141.7%       1.39 ±  3%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      8.66 ±  0%     -11.5%       7.67 ±  1%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg
      3.09 ±  1%     -23.5%       2.36 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      1.05 ±  2%      +9.1%       1.14 ±  3%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      2.23 ±  3%      +9.5%       2.44 ±  1%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.tcp_close
     10.91 ±  1%      +9.6%      11.96 ±  1%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
     14.75 ±  0%     -19.2%      11.91 ±  1%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom
      8.79 ±  0%     -30.0%       6.16 ±  2%  perf-profile.calltrace.cycles-pp.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
      0.78 ±  1%    +110.2%       1.65 ±  4%  perf-profile.calltrace.cycles-pp.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
     15.33 ±  0%     +20.3%      18.45 ±  0%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto
      8.73 ±  0%     -20.4%       6.95 ±  1%  perf-profile.calltrace.cycles-pp.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv
     10.37 ±  0%     +11.4%      11.56 ±  1%  perf-profile.calltrace.cycles-pp.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg
      2.29 ±  3%     +10.4%       2.53 ±  1%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_close.inet_release
      0.61 ±  2%    +140.6%       1.47 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.inet_sendmsg
      7.73 ±  0%     -23.5%       5.91 ±  2%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      2.06 ±  1%      +7.3%       2.21 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
     12.26 ±  0%     +16.7%      14.31 ±  1%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg
      3.04 ±  1%     -12.7%       2.65 ±  0%  perf-profile.calltrace.cycles-pp.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
      2.36 ±  3%     -14.1%       2.02 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
     15.69 ±  0%     -17.4%      12.96 ±  1%  perf-profile.children.cycles-pp.SYSC_recvfrom
     16.25 ±  0%     +19.3%      19.38 ±  1%  perf-profile.children.cycles-pp.SYSC_sendto
      1.66 ±  4%     +16.7%       1.93 ±  1%  perf-profile.children.cycles-pp.__kfree_skb
     15.12 ±  1%     +10.6%      16.72 ±  0%  perf-profile.children.cycles-pp.__release_sock
      5.51 ±  1%     -33.1%       3.69 ±  1%  perf-profile.children.cycles-pp.__schedule
     10.11 ±  0%     -18.0%       8.29 ±  2%  perf-profile.children.cycles-pp.__tcp_ack_snd_check
      3.83 ±  1%     -18.0%       3.14 ±  1%  perf-profile.children.cycles-pp.__wake_up_common
      3.63 ±  2%     -12.2%       3.18 ±  1%  perf-profile.children.cycles-pp.__wake_up_sync_key
      1.86 ±  3%     -11.7%       1.64 ±  1%  perf-profile.children.cycles-pp.activate_task
      3.66 ±  1%     -20.5%       2.91 ±  1%  perf-profile.children.cycles-pp.autoremove_wake_function
      3.56 ±  1%     -18.0%       2.92 ±  0%  perf-profile.children.cycles-pp.default_wake_function
      1.21 ±  5%     -10.4%       1.08 ±  1%  perf-profile.children.cycles-pp.enqueue_entity
      1.64 ±  2%     -10.8%       1.46 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
     14.98 ±  0%     -18.8%      12.17 ±  1%  perf-profile.children.cycles-pp.inet_recvmsg
     15.72 ±  0%     +20.0%      18.87 ±  0%  perf-profile.children.cycles-pp.inet_sendmsg
      2.03 ±  2%     -50.4%       1.01 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
     15.94 ±  1%     +10.2%      17.57 ±  1%  perf-profile.children.cycles-pp.release_sock
      5.86 ±  1%     -33.6%       3.89 ±  0%  perf-profile.children.cycles-pp.schedule
      5.25 ±  1%     -27.3%       3.81 ±  0%  perf-profile.children.cycles-pp.schedule_timeout
      2.68 ±  0%     -15.5%       2.27 ±  2%  perf-profile.children.cycles-pp.sk_wait_data
      1.35 ±  0%     +14.6%       1.55 ±  2%  perf-profile.children.cycles-pp.skb_release_all
     15.32 ±  0%     -18.1%      12.54 ±  1%  perf-profile.children.cycles-pp.sock_recvmsg
     15.97 ±  0%     +19.6%      19.10 ±  0%  perf-profile.children.cycles-pp.sock_sendmsg
     15.79 ±  0%     -17.3%      13.05 ±  1%  perf-profile.children.cycles-pp.sys_recvfrom
     16.32 ±  0%     +19.2%      19.46 ±  0%  perf-profile.children.cycles-pp.sys_sendto
      1.14 ±  2%     -19.3%       0.92 ±  6%  perf-profile.children.cycles-pp.tcp_finish_connect
      2.75 ±  1%     -24.1%       2.08 ±  4%  perf-profile.children.cycles-pp.tcp_prequeue
     10.69 ±  0%     -24.3%       8.09 ±  2%  perf-profile.children.cycles-pp.tcp_prequeue_process
     12.50 ±  0%     +16.6%      14.58 ±  1%  perf-profile.children.cycles-pp.tcp_push
      1.82 ±  3%     +18.1%       2.15 ±  3%  perf-profile.children.cycles-pp.tcp_rearm_rto
     14.78 ±  0%     -19.2%      11.95 ±  1%  perf-profile.children.cycles-pp.tcp_recvmsg
     15.36 ±  0%     +20.3%      18.48 ±  0%  perf-profile.children.cycles-pp.tcp_sendmsg
      2.07 ±  1%      +7.2%       2.22 ±  3%  perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      3.43 ±  2%     -17.5%       2.83 ±  0%  perf-profile.children.cycles-pp.try_to_wake_up
      2.46 ±  3%     -11.6%       2.17 ±  1%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.34 ±  3%      +8.4%       1.45 ±  4%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.81 ±  6%     +18.8%       0.96 ±  3%  perf-profile.self.cycles-pp.mod_timer

                              perf-stat.context-switches

    6e+07 ++----------------------------------------------------------------+
          |   **.*.*.**.*.*.*.**.*     **.*.*    *.*.*.**.*.*.*.**.*.*.**.*.*
  5.5e+07 ++ +                    +   +      +  +                           |
          *.*                      *.*        **                            |
          |                                                                 |
    5e+07 ++                                                                |
          |                                                                 |
  4.5e+07 ++                                                                |
          |                                                                 |
    4e+07 ++                                                                |
          |                                                                 |
          |                                                                 |
  3.5e+07 ++                                                                |
          O O OO O O OO O O O OO O O O OO O O OO O O O                      |
    3e+07 ++----------------------------------------------------------------+

                              perf-stat.cpu-migrations

  11500 ++----O-------------------------------------------------------------+
        |   O                       OO                                      |
  11000 O+               O               O      O                           |
  10500 ++O      O O      O     O            O      O                       |
        |      O     O        O   O                                         |
  10000 ++                  O          O       O                            |
   9500 ++             O                   O      O                         |
        |                                                                   |
   9000 ++                                                                  |
   8500 ++                                                               .* |
        |       .*.          .*.          .*         .*.      .*. .*. *.*  +|
   8000 ++  *.**   *.*   **.*   *    *.*.*  :   *.*.*   *.**.*   *   *      *
   7500 ++ +          + +        +   :      :   :                           |
        *.*            *          *.*        *.*                            |
   7000 ++------------------------------------------------------------------+

                              netperf.Throughput_tps

  3050 ++------------------------------O-OO---O-O-O-------------------------+
       |        O O O O O O OO O O O O      O       O                       |
  3000 O+O O OO                                                             |
  2950 ++                                                                   |
       |                                                .*.   .*.        .*.|
  2900 ++  *.**.*. .*.*.*.*.**.*     *.*.**     *.*.*.**   *.*   *.*.**.*   *
       |   :      *            :     :    :     :                           |
  2850 ++  :                    :    :     :    :                           |
       |  :                     :   :      :   :                            |
  2800 ++ :                     :   :      :   :                            |
  2750 ++ :                     :   :      :   :                            |
       |  :                      :  :       :  :                            |
  2700 ++:                       : :        : :                             |
       *.*                       *.*        *.*                             |
  2650 ++-------------------------------------------------------------------+

                             netperf.time.system_time

  1800 ++-------------------------------------------------------------------+
       |                                                                    |
  1750 O+O O OO O O O O O O OO O O O O O OO O O O O O                       |
       |                                                                    |
       |                                                                    |
  1700 ++                                                                   |
       |                                                                    |
  1650 ++                                                                   |
       |                                                                    |
  1600 ++                                                                   |
       |                                                                    |
       |                                                                    |
  1550 *+*.*.**.*.*.*.*.*.*.**.*.*.*.*.*.**.*.*.*.*.*.**.*.*.*.*.*.*.**.*.*.*
       |                                                                    |
  1500 ++-------------------------------------------------------------------+

                     netperf.time.percent_of_cpu_this_job_got

  590 O+O-O-O-OO-O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-OO------------------------+
      |                                                                     |
  580 ++                                                                    |
  570 ++                                                                    |
      |                                                                     |
  560 ++                                                                    |
      |                                                                     |
  550 ++                                                                    |
      |                                                                     |
  540 ++                                                                    |
  530 ++                                                                    |
      |                                                                     |
  520 ++        .*.*. .*.*.                                                 |
      *.*.*.*.**     *     *.*.**.*.*.*.*.*.*.*.*.**.*.*.*.*.*.*.*.**.*.*.*.*
  510 ++--------------------------------------------------------------------+

                        netperf.time.voluntary_context_switches

  1.5e+07 ++----------------------------------------------------------------+
          O O OO O O OO O O O OO O O O OO O O OO O O O                      |
  1.4e+07 ++                                                                |
  1.3e+07 ++                                                                |
          |                                                                 |
  1.2e+07 ++                                                                |
  1.1e+07 ++                                                                |
          |                                                                 |
    1e+07 ++                                                                |
    9e+06 ++                                                                |
          |                                                                 |
    8e+06 ++                                                                |
    7e+06 ++ .**.*.*.**.*.*.*.**.*.   .**.*.*.  .*.*.*.**.*.*.*.**.*.*.**.*.*
          *.*                      *.*        **                            |
    6e+06 ++----------------------------------------------------------------+

                       netperf.time.involuntary_context_switches

  2.5e+07 ++----------------------------------------------------------------+
          |                                                                 |
          |  .**.*.*.**.*.*.*.**.*.   .**.*.*.  .*.*.*.**.*.*.*.**.*.*.**.*.*
    2e+07 *+*                      *.*        **                            |
          |                                                                 |
          |                                                                 |
  1.5e+07 ++                                                                |
          |                                                                 |
    1e+07 ++                                                                |
          |                                                                 |
          |                                                                 |
    5e+06 ++                                                                |
          |                                                                 |
          |                                                                 |
        0 O+O-OO-O-O-OO-O-O-O-OO-O-O-O-OO-O-O-OO-O-O-O----------------------+

	[*] bisect-good sample
	[O] bisect-bad  sample

Thanks,
Xiaolong