Subject: [cpufreq] d83f959b5e: aim7.jobs-per-min -30.2% regression

From: kernel test robot <[email-masked]>
Date: Sat, 20 Jun 2020 22:36:24 +0800

Greeting,

FYI, we noticed a -30.2% regression of aim7.jobs-per-min due to commit:

commit: d83f959b5e7a6378a4afbff23de2a2d064d95749 ("[PATCH 2/2] cpufreq: Specify default governor on command line")
url: https://github.com/0day-ci/linux/commits/Quentin-Perret/cpufreq-Specify-the-default-governor-on-command-line/20200616-005920
base: https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git linux-next

in testcase: aim7
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	disk: 1BRD_48G
	fs: xfs
	test: disk_src
	load: 3000
	cpufreq_governor: performance
	ucode: 0x2006906

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email-masked]>

Details are as below:
-------------------------------------------------------------------------------------------------->

To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/3000/debian-x86_64-20191114.cgz/lkp-skl-2sp7/disk_src/aim7/0x2006906

commit: 
  21bf3e69e5 ("cpufreq: Register governors at core_initcall")
  d83f959b5e ("cpufreq: Specify default governor on command line")

21bf3e69e51e91cd d83f959b5e7a6378a4afbff23de 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4          100%           4:4     kmsg.cpufreq:cpufreq_online:Failed_to_initialize_policy_for_cpu:#(-#)
         %stddev     %change         %stddev
             \          |                \  
     37531           -30.2%      26214        aim7.jobs-per-min
    479.85           +43.2%     686.95        aim7.time.elapsed_time
    479.85           +43.2%     686.95        aim7.time.elapsed_time.max
      1489 ±  2%     +38.4%       2061        aim7.time.involuntary_context_switches
      1126           +44.4%       1626        aim7.time.system_time
     49.29           +49.9%      73.87        aim7.time.user_time
      3.01            +1.7%       3.06        iostat.cpu.system
      4535           -32.3%       3070 ±  2%  vmstat.io.bo
    133979           -29.0%      95145        vmstat.system.cs
     24.32            +1.6%      24.71        boot-time.boot
     15.32            +3.8%      15.90        boot-time.dhcp
      1446            +1.6%       1468        boot-time.idle
 5.532e+08 ±  5%     -36.1%  3.538e+08 ±  2%  cpuidle.C1.time
   5739465 ±  2%     -31.5%    3933307        cpuidle.C1.usage
  78066984 ± 13%     +24.5%   97217135 ±  9%  cpuidle.C1E.usage
   7739904 ±  5%     +15.4%    8929792 ±  4%  meminfo.DirectMap2M
    590700 ±  5%     -23.9%     449388 ±  7%  meminfo.DirectMap4k
      5963           -30.2%       4160        meminfo.max_used_kB
    113.75 ± 57%    +295.4%     449.75 ± 79%  numa-meminfo.node0.Inactive(file)
   1040851 ±  6%     +17.5%    1222596 ± 14%  numa-meminfo.node0.MemUsed
      1316 ± 50%   +2193.8%      30192 ±143%  numa-meminfo.node0.PageTables
    129108 ±  6%     +33.7%     172589 ± 23%  numa-meminfo.node0.Slab
    100458 ± 37%     +73.4%     174147 ± 21%  numa-meminfo.node1.AnonHugePages
     81000 ±  5%     -16.3%      67824 ± 10%  numa-meminfo.node1.KReclaimable
     81000 ±  5%     -16.3%      67824 ± 10%  numa-meminfo.node1.SReclaimable
     13.50 ±133%    +596.3%      94.00 ±102%  numa-vmstat.node0.nr_dirtied
     28.25 ± 57%    +296.5%     112.00 ± 79%  numa-vmstat.node0.nr_inactive_file
    328.50 ± 50%   +2200.5%       7557 ±143%  numa-vmstat.node0.nr_page_table_pages
     13.50 ±133%    +592.6%      93.50 ±103%  numa-vmstat.node0.nr_written
     28.25 ± 57%    +296.5%     112.00 ± 79%  numa-vmstat.node0.nr_zone_inactive_file
     48.50 ± 37%     +74.2%      84.50 ± 21%  numa-vmstat.node1.nr_anon_transparent_hugepages
     20249 ±  5%     -16.3%      16952 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
      4779 ±  6%     +36.1%       6506 ±  2%  slabinfo.Acpi-State.active_objs
      4779 ±  6%     +36.1%       6506 ±  2%  slabinfo.Acpi-State.num_objs
    536.25 ±  9%     +13.8%     610.25 ±  7%  slabinfo.UDP.active_objs
    536.25 ±  9%     +13.8%     610.25 ±  7%  slabinfo.UDP.num_objs
      2894 ± 10%     +18.2%       3419 ±  6%  slabinfo.eventpoll_pwq.active_objs
      2894 ± 10%     +18.2%       3419 ±  6%  slabinfo.eventpoll_pwq.num_objs
    648.00 ± 21%     +34.6%     872.00 ±  4%  slabinfo.kmalloc-rcl-128.active_objs
    648.00 ± 21%     +34.6%     872.00 ±  4%  slabinfo.kmalloc-rcl-128.num_objs
    268.25 ± 21%     +57.9%     423.50 ±  7%  slabinfo.nfs_read_data.active_objs
    268.25 ± 21%     +57.9%     423.50 ±  7%  slabinfo.nfs_read_data.num_objs
    685.00 ± 10%     +16.8%     800.25 ±  5%  slabinfo.skbuff_fclone_cache.active_objs
    685.00 ± 10%     +16.8%     800.25 ±  5%  slabinfo.skbuff_fclone_cache.num_objs
     94.50            +6.3%     100.50        proc-vmstat.nr_anon_transparent_hugepages
    322.50            +2.8%     331.50        proc-vmstat.nr_dirtied
      4433            +2.1%       4526        proc-vmstat.nr_inactive_anon
    337.25            +2.6%     346.00        proc-vmstat.nr_inactive_file
    322.00            +2.7%     330.75        proc-vmstat.nr_written
      4433            +2.1%       4526        proc-vmstat.nr_zone_inactive_anon
    337.25            +2.6%     346.00        proc-vmstat.nr_zone_inactive_file
   1279746           +32.0%    1688708        proc-vmstat.numa_hit
   1256250           +32.6%    1665258        proc-vmstat.numa_local
      1568 ±  4%     -25.6%       1166 ±  2%  proc-vmstat.pgactivate
   1474183           +28.3%    1890851        proc-vmstat.pgalloc_normal
   1400035           +37.3%    1922052        proc-vmstat.pgfault
   1425068           +29.5%    1845014        proc-vmstat.pgfree
    857.00 ±  9%     +42.1%       1218 ±  5%  proc-vmstat.pgpgin
   2188373            -3.2%    2118251 ±  2%  proc-vmstat.pgpgout
      8038 ±  6%     +44.2%      11589        sched_debug.cfs_rq:/.exec_clock.avg
     11048 ±  2%     +38.4%      15291 ±  2%  sched_debug.cfs_rq:/.exec_clock.max
      6322 ±  7%     +42.8%       9029 ±  9%  sched_debug.cfs_rq:/.exec_clock.min
    750.65 ±  6%     +22.8%     922.15 ± 13%  sched_debug.cfs_rq:/.exec_clock.stddev
     18.09 ± 10%     -21.3%      14.24        sched_debug.cfs_rq:/.load_avg.avg
    394.62 ±  2%      +9.3%     431.44 ±  3%  sched_debug.cfs_rq:/.load_avg.max
      0.32 ± 13%    +121.3%       0.71 ± 11%  sched_debug.cfs_rq:/.nr_spread_over.avg
      3.22 ± 58%     -54.1%       1.48 ± 36%  sched_debug.cfs_rq:/.removed.load_avg.avg
    116.86 ±  5%     -28.2%      83.94        sched_debug.cfs_rq:/.removed.load_avg.max
     18.23 ± 26%     -40.3%      10.88 ± 17%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      3.22 ± 58%     -54.1%       1.48 ± 36%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    116.86 ±  5%     -28.2%      83.94        sched_debug.cfs_rq:/.removed.runnable_avg.max
     18.23 ± 26%     -40.3%      10.88 ± 17%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
    149.95 ±  4%     -22.3%     116.58 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
      1407 ±  6%      -8.3%       1291 ±  3%  sched_debug.cfs_rq:/.runnable_avg.max
      2.05 ± 20%     -70.5%       0.60 ± 35%  sched_debug.cfs_rq:/.runnable_avg.min
    239.05 ±  8%     -17.2%     197.94        sched_debug.cfs_rq:/.runnable_avg.stddev
    102746 ± 14%     -22.7%      79373 ± 22%  sched_debug.cfs_rq:/.spread0.max
     74.08 ±  2%     -20.9%      58.61 ±  2%  sched_debug.cfs_rq:/.util_avg.avg
      1.82 ± 25%     -73.7%       0.48 ± 46%  sched_debug.cfs_rq:/.util_avg.min
    123.17 ±  4%     -12.9%     107.28 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
     60901 ±  7%     +15.6%      70419 ±  6%  sched_debug.cpu.avg_idle.min
    260021 ±  5%     +37.8%     358434        sched_debug.cpu.clock.avg
    260042 ±  5%     +37.8%     358462        sched_debug.cpu.clock.max
    260000 ±  5%     +37.8%     358404        sched_debug.cpu.clock.min
     12.18 ±  2%     +36.2%      16.59 ± 10%  sched_debug.cpu.clock.stddev
    260021 ±  5%     +37.8%     358434        sched_debug.cpu.clock_task.avg
    260042 ±  5%     +37.8%     358462        sched_debug.cpu.clock_task.max
    260000 ±  5%     +37.8%     358404        sched_debug.cpu.clock_task.min
     12.18 ±  2%     +36.2%      16.59 ± 10%  sched_debug.cpu.clock_task.stddev
     10341 ±  3%     +24.0%      12822        sched_debug.cpu.curr->pid.max
      1421 ±  5%     +17.8%       1673 ±  2%  sched_debug.cpu.curr->pid.stddev
      0.00           +11.2%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
      2036 ±  5%     +26.6%       2578        sched_debug.cpu.ttwu_local.avg
      4220 ±  5%      +7.6%       4539 ±  3%  sched_debug.cpu.ttwu_local.max
      1074 ±  8%     +39.8%       1502 ±  4%  sched_debug.cpu.ttwu_local.min
    260000 ±  5%     +37.8%     358404        sched_debug.cpu_clk
    259499 ±  5%     +37.9%     357901        sched_debug.ktime
    260390 ±  5%     +37.8%     358806        sched_debug.sched_clk
 1.659e+09           -27.6%  1.201e+09        perf-stat.i.branch-instructions
  23511802 ±  2%     -24.7%   17704718        perf-stat.i.branch-misses
  16325833 ±  2%     -29.2%   11553674        perf-stat.i.cache-misses
  51488691 ±  2%     -28.8%   36674264        perf-stat.i.cache-references
    134382           -29.1%      95334        perf-stat.i.context-switches
      1.14 ±  2%      -6.6%       1.06        perf-stat.i.cpi
 9.011e+09           -31.8%  6.145e+09        perf-stat.i.cpu-cycles
      1591           -37.6%     993.54        perf-stat.i.cpu-migrations
    597.52            -6.5%     558.71        perf-stat.i.cycles-between-cache-misses
   2117105 ±  2%     -29.2%    1499831        perf-stat.i.dTLB-load-misses
 1.925e+09           -27.5%  1.396e+09        perf-stat.i.dTLB-loads
     93028 ±  7%     -28.7%      66336 ±  2%  perf-stat.i.dTLB-store-misses
 8.944e+08           -26.9%  6.541e+08        perf-stat.i.dTLB-stores
     15.06 ±  2%      -1.3       13.73        perf-stat.i.iTLB-load-miss-rate%
   1042616           -30.9%     720885        perf-stat.i.iTLB-load-misses
   5925503           -23.2%    4551782        perf-stat.i.iTLB-loads
 8.199e+09           -27.7%  5.929e+09        perf-stat.i.instructions
     10027 ±  3%      -8.7%       9159 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.91            +6.4%       0.96        perf-stat.i.ipc
      0.13           -31.8%       0.09        perf-stat.i.metric.GHz
      0.12          +709.6%       0.96        perf-stat.i.metric.K/sec
     63.08           -27.4%      45.79        perf-stat.i.metric.M/sec
      2820            -3.9%       2710        perf-stat.i.minor-faults
   5163419           -29.7%    3630179        perf-stat.i.node-load-misses
    845228 ±  3%     -29.9%     592653 ±  2%  perf-stat.i.node-loads
   1288470           -30.3%     897482        perf-stat.i.node-store-misses
    125883           -28.0%      90575        perf-stat.i.node-stores
      2820            -3.9%       2710        perf-stat.i.page-faults
      1.42 ±  2%      +0.1        1.47        perf-stat.overall.branch-miss-rate%
      1.10            -5.7%       1.04        perf-stat.overall.cpi
     14.96            -1.3       13.68        perf-stat.overall.iTLB-load-miss-rate%
      7864            +4.6%       8224        perf-stat.overall.instructions-per-iTLB-miss
      0.91            +6.0%       0.96        perf-stat.overall.ipc
 1.655e+09           -27.5%    1.2e+09        perf-stat.ps.branch-instructions
  23464114 ±  2%     -24.6%   17681251        perf-stat.ps.branch-misses
  16290823 ±  2%     -29.2%   11536638        perf-stat.ps.cache-misses
  51379893 ±  2%     -28.7%   36622024        perf-stat.ps.cache-references
    134092           -29.0%      95192        perf-stat.ps.context-switches
 8.992e+09           -31.8%  6.136e+09        perf-stat.ps.cpu-cycles
      1587           -37.5%     992.51        perf-stat.ps.cpu-migrations
   2112603 ±  2%     -29.1%    1497800        perf-stat.ps.dTLB-load-misses
 1.921e+09           -27.4%  1.394e+09        perf-stat.ps.dTLB-loads
     92837 ±  7%     -28.6%      66254 ±  2%  perf-stat.ps.dTLB-store-misses
 8.925e+08           -26.8%  6.532e+08        perf-stat.ps.dTLB-stores
   1040393           -30.8%     719911        perf-stat.ps.iTLB-load-misses
   5912777           -23.1%    4544925        perf-stat.ps.iTLB-loads
 8.181e+09           -27.6%  5.921e+09        perf-stat.ps.instructions
      2816            -3.8%       2708        perf-stat.ps.minor-faults
   5152279           -29.6%    3624732        perf-stat.ps.node-load-misses
    843418 ±  3%     -29.8%     591785 ±  2%  perf-stat.ps.node-loads
   1285691           -30.3%     896142        perf-stat.ps.node-store-misses
    125624           -28.0%      90468        perf-stat.ps.node-stores
      2816            -3.8%       2708        perf-stat.ps.page-faults
 3.934e+12            +3.5%  4.072e+12        perf-stat.total.instructions
    148225 ±  3%     +34.9%     199978 ±  2%  softirqs.CPU0.RCU
     68852           +36.5%      94003        softirqs.CPU0.SCHED
    161692 ±  5%     +56.1%     252411 ±  3%  softirqs.CPU0.TIMER
    149090 ±  3%     +32.7%     197803 ±  3%  softirqs.CPU1.RCU
     64130           +38.9%      89072 ±  2%  softirqs.CPU1.SCHED
    156371 ±  6%     +59.5%     249452 ±  3%  softirqs.CPU1.TIMER
    149696 ±  3%     +35.0%     202059 ±  2%  softirqs.CPU10.RCU
     62512 ±  2%     +44.0%      89989        softirqs.CPU10.SCHED
    156527 ±  6%     +60.0%     250425 ±  3%  softirqs.CPU10.TIMER
    147497 ±  2%     +35.1%     199264        softirqs.CPU11.RCU
     62553           +41.7%      88638 ±  2%  softirqs.CPU11.SCHED
    155703 ±  4%     +60.2%     249398 ±  4%  softirqs.CPU11.TIMER
    148757 ±  3%     +35.2%     201173 ±  2%  softirqs.CPU12.RCU
     63210           +42.0%      89782        softirqs.CPU12.SCHED
    156322 ±  6%     +59.8%     249797 ±  4%  softirqs.CPU12.TIMER
    149867 ±  3%     +31.6%     197237 ±  5%  softirqs.CPU13.RCU
     63165           +44.3%      91168 ±  2%  softirqs.CPU13.SCHED
    157012 ±  5%     +77.0%     277943 ± 13%  softirqs.CPU13.TIMER
    145801 ±  3%     +36.7%     199320        softirqs.CPU14.RCU
     62885           +42.0%      89294        softirqs.CPU14.SCHED
    154607 ±  6%     +61.1%     249128 ±  4%  softirqs.CPU14.TIMER
    125830           +47.3%     185328        softirqs.CPU15.RCU
     63213           +39.1%      87898 ±  2%  softirqs.CPU15.SCHED
    155314 ±  5%     +60.2%     248821 ±  3%  softirqs.CPU15.TIMER
    127243           +47.1%     187237        softirqs.CPU16.RCU
     62817 ±  2%     +42.8%      89726        softirqs.CPU16.SCHED
    155933 ±  5%     +60.1%     249636 ±  3%  softirqs.CPU16.TIMER
    128980 ±  2%     +45.0%     187015        softirqs.CPU17.RCU
     63146           +38.4%      87373 ±  2%  softirqs.CPU17.SCHED
    159461 ±  5%     +56.9%     250262 ±  3%  softirqs.CPU17.TIMER
    149393           +35.6%     202597        softirqs.CPU18.RCU
     66228 ±  2%     +36.9%      90683        softirqs.CPU18.SCHED
    159695           +57.9%     252201        softirqs.CPU18.TIMER
    145542           +38.5%     201516        softirqs.CPU19.RCU
     64394 ±  2%     +39.9%      90093        softirqs.CPU19.SCHED
    148693 ±  5%     +64.3%     244278        softirqs.CPU19.TIMER
    146496 ±  2%     +36.2%     199533        softirqs.CPU2.RCU
     62601           +42.5%      89193 ±  2%  softirqs.CPU2.SCHED
    156672 ±  4%     +60.6%     251583 ±  3%  softirqs.CPU2.TIMER
    145548 ±  7%     +41.3%     205694        softirqs.CPU20.RCU
     64297 ±  2%     +38.9%      89319        softirqs.CPU20.SCHED
    160604 ±  8%     +54.4%     248021        softirqs.CPU20.TIMER
    150370           +36.5%     205196        softirqs.CPU21.RCU
     64605           +37.7%      88956 ±  2%  softirqs.CPU21.SCHED
    154841 ±  2%     +59.3%     246625 ±  2%  softirqs.CPU21.TIMER
    149245           +36.9%     204303        softirqs.CPU22.RCU
     63598           +39.5%      88735 ±  2%  softirqs.CPU22.SCHED
    152785 ±  2%     +60.4%     245140        softirqs.CPU22.TIMER
    145943           +37.6%     200890        softirqs.CPU23.RCU
     63605 ±  2%     +41.2%      89835        softirqs.CPU23.SCHED
    147754 ±  3%     +64.9%     243701        softirqs.CPU23.TIMER
    148007           +36.6%     202142        softirqs.CPU24.RCU
     64937 ±  2%     +38.3%      89833        softirqs.CPU24.SCHED
    152782 ±  3%     +59.2%     243256        softirqs.CPU24.TIMER
    149238           +34.4%     200565 ±  2%  softirqs.CPU25.RCU
     64300 ±  2%     +42.3%      91524 ±  3%  softirqs.CPU25.SCHED
    153715 ±  2%     +77.9%     273383 ± 17%  softirqs.CPU25.TIMER
    149265           +36.5%     203791        softirqs.CPU26.RCU
     64902 ±  2%     +39.6%      90608        softirqs.CPU26.SCHED
    155441           +57.7%     245113        softirqs.CPU26.TIMER
    146233           +37.8%     201483        softirqs.CPU27.RCU
     63266 ±  2%     +39.5%      88228 ±  3%  softirqs.CPU27.SCHED
    147013 ±  4%     +65.5%     243357        softirqs.CPU27.TIMER
    145540 ±  3%     +38.8%     201950        softirqs.CPU28.RCU
     65407 ±  3%     +36.7%      89434 ±  3%  softirqs.CPU28.SCHED
    148271           +36.3%     202125        softirqs.CPU29.RCU
     64378           +38.8%      89361        softirqs.CPU29.SCHED
    149114 ±  4%     +62.7%     242571        softirqs.CPU29.TIMER
    149246 ±  3%     +34.5%     200795        softirqs.CPU3.RCU
     63878           +37.7%      87943 ±  3%  softirqs.CPU3.SCHED
    157953 ±  6%     +58.6%     250442 ±  3%  softirqs.CPU3.TIMER
    138500           +35.4%     187554 ±  5%  softirqs.CPU30.RCU
     63410 ±  5%     +42.1%      90114 ±  2%  softirqs.CPU30.SCHED
    153064 ±  3%     +61.1%     246521 ±  3%  softirqs.CPU30.TIMER
    138816           +39.5%     193704 ±  2%  softirqs.CPU31.RCU
     64501 ±  2%     +40.2%      90406 ±  4%  softirqs.CPU31.SCHED
    152639 ±  4%     +78.3%     272192 ± 16%  softirqs.CPU31.TIMER
    136214           +42.4%     193926        softirqs.CPU32.RCU
     65755 ±  4%     +37.3%      90313        softirqs.CPU32.SCHED
    136873           +42.2%     194572        softirqs.CPU33.RCU
     64588 ±  2%     +37.1%      88570 ±  2%  softirqs.CPU33.SCHED
    149425 ±  3%     +63.1%     243776        softirqs.CPU33.TIMER
    137813           +41.7%     195263        softirqs.CPU34.RCU
     63090 ±  2%     +43.5%      90558        softirqs.CPU34.SCHED
    150682 ±  3%     +62.5%     244916        softirqs.CPU34.TIMER
    138930           +40.8%     195547        softirqs.CPU35.RCU
     64733 ±  2%     +32.7%      85915 ±  6%  softirqs.CPU35.SCHED
    154054 ±  2%     +58.8%     244669        softirqs.CPU35.TIMER
    146677 ±  2%     +38.8%     203648        softirqs.CPU36.RCU
     61997           +42.5%      88335        softirqs.CPU36.SCHED
    151515 ±  4%     +63.8%     248136 ±  3%  softirqs.CPU36.TIMER
    148944 ±  3%     +35.6%     201924 ±  2%  softirqs.CPU37.RCU
     62089           +40.3%      87128 ±  3%  softirqs.CPU37.SCHED
    152938 ±  5%     +62.0%     247793 ±  3%  softirqs.CPU37.TIMER
    147059           +39.2%     204753        softirqs.CPU38.RCU
     62171           +42.7%      88691        softirqs.CPU38.SCHED
    152184 ±  5%     +62.7%     247603 ±  3%  softirqs.CPU38.TIMER
    149805 ±  4%     +37.0%     205264        softirqs.CPU39.RCU
     61107 ±  2%     +45.3%      88786        softirqs.CPU39.SCHED
    152465 ±  4%     +62.1%     247221 ±  3%  softirqs.CPU39.TIMER
    150072 ±  3%     +34.4%     201733        softirqs.CPU4.RCU
     63819           +40.5%      89697        softirqs.CPU4.SCHED
    158029 ±  6%     +58.6%     250631 ±  3%  softirqs.CPU4.TIMER
    152095 ±  3%     +36.8%     208069        softirqs.CPU40.RCU
     62807           +41.3%      88778        softirqs.CPU40.SCHED
    152878 ±  5%     +62.5%     248464 ±  3%  softirqs.CPU40.TIMER
    147032 ±  2%     +39.3%     204813        softirqs.CPU41.RCU
     60510 ±  5%     +47.6%      89284        softirqs.CPU41.SCHED
    152687 ±  3%     +62.1%     247555 ±  3%  softirqs.CPU41.TIMER
    147261 ±  2%     +38.9%     204553        softirqs.CPU42.RCU
     62877           +40.8%      88551        softirqs.CPU42.SCHED
    153002 ±  4%     +62.1%     248053 ±  3%  softirqs.CPU42.TIMER
    149469 ±  2%     +37.7%     205807        softirqs.CPU43.RCU
     61857 ±  2%     +43.1%      88502        softirqs.CPU43.SCHED
    152893 ±  4%     +62.0%     247754 ±  3%  softirqs.CPU43.TIMER
    144650 ±  9%     +42.0%     205431        softirqs.CPU44.RCU
     63153           +36.4%      86145 ±  5%  softirqs.CPU44.SCHED
    152723 ±  5%     +62.2%     247666 ±  3%  softirqs.CPU44.TIMER
    142302 ±  2%     +42.7%     203007        softirqs.CPU45.RCU
     61870           +42.5%      88138        softirqs.CPU45.SCHED
    151680 ±  4%     +62.5%     246506 ±  3%  softirqs.CPU45.TIMER
    150475 ±  2%     +37.1%     206324        softirqs.CPU46.RCU
     61258 ±  2%     +45.8%      89321        softirqs.CPU46.SCHED
    152918 ±  5%     +62.0%     247671 ±  3%  softirqs.CPU46.TIMER
    149712 ±  2%     +37.7%     206114        softirqs.CPU47.RCU
     61526 ±  4%     +44.4%      88873        softirqs.CPU47.SCHED
    152360 ±  4%     +62.2%     247203 ±  3%  softirqs.CPU47.TIMER
    149904 ±  3%     +38.1%     206972        softirqs.CPU48.RCU
     62334           +42.1%      88572        softirqs.CPU48.SCHED
    152221 ±  4%     +61.9%     246446 ±  4%  softirqs.CPU48.TIMER
    152162 ±  2%     +32.4%     201405 ±  6%  softirqs.CPU49.RCU
     62446           +40.8%      87927 ±  2%  softirqs.CPU49.SCHED
    151903 ±  4%     +56.9%     238395 ± 10%  softirqs.CPU49.TIMER
    148700 ±  3%     +35.2%     200969        softirqs.CPU5.RCU
     63266           +40.3%      88735        softirqs.CPU5.SCHED
    155744 ±  5%     +60.9%     250589 ±  3%  softirqs.CPU5.TIMER
    144528 ±  5%     +41.8%     204869        softirqs.CPU50.RCU
     60457 ±  3%     +45.1%      87749 ±  2%  softirqs.CPU50.SCHED
    152440 ±  3%     +61.2%     245785 ±  4%  softirqs.CPU50.TIMER
    148237 ±  2%     +37.9%     204469        softirqs.CPU51.RCU
     60863 ±  4%     +44.6%      88006        softirqs.CPU51.SCHED
    152274 ±  4%     +61.9%     246595 ±  3%  softirqs.CPU51.TIMER
    149127 ±  2%     +38.4%     206376        softirqs.CPU52.RCU
     62360           +43.4%      89443        softirqs.CPU52.SCHED
    152512 ±  5%     +62.2%     247332 ±  3%  softirqs.CPU52.TIMER
    151816 ±  3%     +36.1%     206605        softirqs.CPU53.RCU
     61809 ±  4%     +44.6%      89378        softirqs.CPU53.SCHED
    153584 ±  4%     +61.4%     247836 ±  3%  softirqs.CPU53.TIMER
    138286           +41.3%     195411        softirqs.CPU54.RCU
     63203 ±  3%     +43.8%      90857        softirqs.CPU54.SCHED
    148682 ±  3%     +67.6%     249252        softirqs.CPU54.TIMER
    136334           +43.5%     195592        softirqs.CPU55.RCU
     63586 ±  2%     +40.2%      89146        softirqs.CPU55.SCHED
    145454 ±  5%     +66.1%     241627        softirqs.CPU55.TIMER
    137781 ±  2%     +42.9%     196874        softirqs.CPU56.RCU
     64602 ±  2%     +34.8%      87089 ±  4%  softirqs.CPU56.SCHED
    139229           +41.5%     197025        softirqs.CPU57.RCU
     59929 ±  8%     +49.5%      89584        softirqs.CPU57.SCHED
    145306 ±  4%     +66.2%     241553        softirqs.CPU57.TIMER
    138590           +42.8%     197852        softirqs.CPU58.RCU
     63888 ±  2%     +33.2%      85085 ±  8%  softirqs.CPU58.SCHED
    144762 ±  5%     +67.1%     241869        softirqs.CPU58.TIMER
    134272 ±  2%     +43.9%     193254        softirqs.CPU59.RCU
     62096 ±  2%     +42.7%      88626 ±  2%  softirqs.CPU59.SCHED
    142782 ±  5%     +68.7%     240865        softirqs.CPU59.TIMER
    147604 ±  3%     +35.4%     199910        softirqs.CPU6.RCU
     63325           +41.3%      89494        softirqs.CPU6.SCHED
    156352 ±  4%     +59.9%     250043 ±  3%  softirqs.CPU6.TIMER
    137781           +37.2%     189074        softirqs.CPU60.RCU
     61228 ±  6%     +46.7%      89834        softirqs.CPU60.SCHED
    147828 ±  4%     +62.9%     240826 ±  2%  softirqs.CPU60.TIMER
    139277           +32.6%     184693 ±  5%  softirqs.CPU61.RCU
     60631 ±  3%     +44.8%      87809 ±  3%  softirqs.CPU61.SCHED
    147103 ±  4%     +57.3%     231356 ±  8%  softirqs.CPU61.TIMER
    139039           +37.0%     190516        softirqs.CPU62.RCU
     63559           +41.4%      89882        softirqs.CPU62.SCHED
    145849 ±  4%     +65.1%     240854 ±  2%  softirqs.CPU62.TIMER
    134528 ±  2%     +40.6%     189101        softirqs.CPU63.RCU
     62093           +41.9%      88085 ±  2%  softirqs.CPU63.SCHED
    144435 ±  5%     +66.8%     240912 ±  2%  softirqs.CPU63.TIMER
    131970 ±  7%     +43.4%     189308        softirqs.CPU64.RCU
     61853 ±  4%     +43.5%      88780        softirqs.CPU64.SCHED
    145033 ±  5%     +66.0%     240731 ±  2%  softirqs.CPU64.TIMER
    139350           +36.7%     190526        softirqs.CPU65.RCU
     61564 ±  5%     +45.0%      89289 ±  2%  softirqs.CPU65.SCHED
    145387 ±  4%     +65.2%     240188 ±  2%  softirqs.CPU65.TIMER
    138915           +35.3%     187968 ±  2%  softirqs.CPU66.RCU
     62678 ±  2%     +43.3%      89802 ±  5%  softirqs.CPU66.SCHED
    145202 ±  5%     +86.0%     270051 ± 18%  softirqs.CPU66.TIMER
    138859 ±  2%     +33.2%     184905 ±  5%  softirqs.CPU67.RCU
     63267           +40.4%      88853        softirqs.CPU67.SCHED
    145227 ±  5%     +58.7%     230424 ±  8%  softirqs.CPU67.TIMER
    129176 ±  6%     +45.1%     187491        softirqs.CPU68.RCU
     60993 ±  3%     +46.1%      89128        softirqs.CPU68.SCHED
    144409 ±  6%     +65.7%     239313 ±  2%  softirqs.CPU68.TIMER
    135472 ±  2%     +39.2%     188515        softirqs.CPU69.RCU
     62141 ±  2%     +39.7%      86793 ±  4%  softirqs.CPU69.SCHED
    144417 ±  5%     +67.4%     241757 ±  2%  softirqs.CPU69.TIMER
    147138 ±  3%     +35.1%     198748        softirqs.CPU7.RCU
     63426           +40.9%      89340        softirqs.CPU7.SCHED
    156336 ±  5%     +60.3%     250589 ±  3%  softirqs.CPU7.TIMER
    137717           +37.8%     189723        softirqs.CPU70.RCU
     63375 ±  2%     +41.5%      89708        softirqs.CPU70.SCHED
    145921 ±  4%     +66.0%     242291        softirqs.CPU70.TIMER
    139796           +36.5%     190794        softirqs.CPU71.RCU
     62089 ±  4%     +44.3%      89621        softirqs.CPU71.SCHED
    145373 ±  4%     +66.5%     242008        softirqs.CPU71.TIMER
    145025 ±  8%     +38.2%     200408        softirqs.CPU8.RCU
     64759 ±  2%     +37.0%      88724        softirqs.CPU8.SCHED
    185013 ± 19%     +35.6%     250815 ±  3%  softirqs.CPU8.TIMER
    145962 ±  2%     +36.7%     199584        softirqs.CPU9.RCU
     62967           +41.8%      89283        softirqs.CPU9.SCHED
    153151 ±  5%     +62.8%     249403 ±  3%  softirqs.CPU9.TIMER
  10338644           +38.1%   14279978        softirqs.RCU
   4540831           +41.2%    6411461        softirqs.SCHED
  11032630           +61.3%   17798417        softirqs.TIMER
    410.75 ±  3%     +44.1%     591.75 ±  3%  interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
      1682 ±172%    -100.0%       0.75 ±173%  interrupts.87:PCI-MSI.31981620-edge.i40e-eth0-TxRx-51
    962.50           +43.1%       1377        interrupts.9:IO-APIC.9-fasteoi.acpi
    960488           +43.2%    1375718        interrupts.CPU0.LOC:Local_timer_interrupts
    962.50           +43.1%       1377        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    960657           +43.2%    1375739        interrupts.CPU1.LOC:Local_timer_interrupts
    581.50 ±  5%     -36.6%     368.50 ±  2%  interrupts.CPU1.NMI:Non-maskable_interrupts
    581.50 ±  5%     -36.6%     368.50 ±  2%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    960352           +43.3%    1375770        interrupts.CPU10.LOC:Local_timer_interrupts
    545.75 ± 12%     -42.8%     312.25 ±  8%  interrupts.CPU10.NMI:Non-maskable_interrupts
    545.75 ± 12%     -42.8%     312.25 ±  8%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      3029 ±  4%     +52.2%       4609 ±  4%  interrupts.CPU10.RES:Rescheduling_interrupts
    960563           +43.2%    1375749        interrupts.CPU11.LOC:Local_timer_interrupts
    580.75 ± 12%     -38.9%     355.00 ±  7%  interrupts.CPU11.NMI:Non-maskable_interrupts
    580.75 ± 12%     -38.9%     355.00 ±  7%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      3049 ±  5%     +58.8%       4841 ±  3%  interrupts.CPU11.RES:Rescheduling_interrupts
    960595           +43.2%    1375735        interrupts.CPU12.LOC:Local_timer_interrupts
    568.50 ±  7%     -33.6%     377.50 ±  2%  interrupts.CPU12.NMI:Non-maskable_interrupts
    568.50 ±  7%     -33.6%     377.50 ±  2%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      3001 ±  4%     +53.8%       4615 ±  4%  interrupts.CPU12.RES:Rescheduling_interrupts
    960551           +43.2%    1375659        interrupts.CPU13.LOC:Local_timer_interrupts
    571.50 ±  5%     -30.6%     396.50 ±  7%  interrupts.CPU13.NMI:Non-maskable_interrupts
    571.50 ±  5%     -30.6%     396.50 ±  7%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      3054 ±  3%     +53.2%       4678 ±  6%  interrupts.CPU13.RES:Rescheduling_interrupts
    960642           +43.2%    1375651        interrupts.CPU14.LOC:Local_timer_interrupts
    582.75 ±  9%     -42.6%     334.25 ±  4%  interrupts.CPU14.NMI:Non-maskable_interrupts
    582.75 ±  9%     -42.6%     334.25 ±  4%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      2990 ±  3%     +50.8%       4511 ±  8%  interrupts.CPU14.RES:Rescheduling_interrupts
    960608           +43.2%    1375557        interrupts.CPU15.LOC:Local_timer_interrupts
    560.75 ±  4%     -30.5%     389.75 ±  8%  interrupts.CPU15.NMI:Non-maskable_interrupts
    560.75 ±  4%     -30.5%     389.75 ±  8%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      3037 ±  2%     +49.8%       4551 ±  7%  interrupts.CPU15.RES:Rescheduling_interrupts
    960636           +43.2%    1375721        interrupts.CPU16.LOC:Local_timer_interrupts
    547.00 ±  8%     -29.3%     387.00 ± 13%  interrupts.CPU16.NMI:Non-maskable_interrupts
    547.00 ±  8%     -29.3%     387.00 ± 13%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      2956 ±  3%     +52.3%       4502 ±  2%  interrupts.CPU16.RES:Rescheduling_interrupts
    960558           +43.2%    1375685        interrupts.CPU17.LOC:Local_timer_interrupts
      2954 ±  6%     +47.1%       4345 ±  3%  interrupts.CPU17.RES:Rescheduling_interrupts
    960329           +43.2%    1375542        interrupts.CPU18.LOC:Local_timer_interrupts
    598.25 ± 10%     -35.2%     387.50 ±  3%  interrupts.CPU18.NMI:Non-maskable_interrupts
    598.25 ± 10%     -35.2%     387.50 ±  3%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
      3027 ±  5%     +56.4%       4735 ±  2%  interrupts.CPU18.RES:Rescheduling_interrupts
    960405           +43.3%    1375801        interrupts.CPU19.LOC:Local_timer_interrupts
    533.75 ± 11%     -41.1%     314.25 ± 16%  interrupts.CPU19.NMI:Non-maskable_interrupts
    533.75 ± 11%     -41.1%     314.25 ± 16%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      2878 ±  3%     +54.6%       4449 ±  2%  interrupts.CPU19.RES:Rescheduling_interrupts
    960642           +43.2%    1375585        interrupts.CPU2.LOC:Local_timer_interrupts
    574.25 ± 17%     -35.4%     371.25 ±  6%  interrupts.CPU2.NMI:Non-maskable_interrupts
    574.25 ± 17%     -35.4%     371.25 ±  6%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      3107 ±  4%     +58.3%       4921 ± 12%  interrupts.CPU2.RES:Rescheduling_interrupts
    960247           +43.3%    1375760        interrupts.CPU20.LOC:Local_timer_interrupts
    547.50 ±  8%     -48.4%     282.25 ± 24%  interrupts.CPU20.NMI:Non-maskable_interrupts
    547.50 ±  8%     -48.4%     282.25 ± 24%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      2940 ±  6%     +53.1%       4501 ±  5%  interrupts.CPU20.RES:Rescheduling_interrupts
    960353           +43.3%    1375737        interrupts.CPU21.LOC:Local_timer_interrupts
      2926 ±  4%     +54.6%       4523        interrupts.CPU21.RES:Rescheduling_interrupts
    960365           +43.3%    1375751        interrupts.CPU22.LOC:Local_timer_interrupts
    503.50 ± 13%     -32.5%     339.75 ±  8%  interrupts.CPU22.NMI:Non-maskable_interrupts
    503.50 ± 13%     -32.5%     339.75 ±  8%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      2936 ±  6%     +53.3%       4500        interrupts.CPU22.RES:Rescheduling_interrupts
    960336           +43.3%    1375691        interrupts.CPU23.LOC:Local_timer_interrupts
    479.75 ± 12%     -23.6%     366.75 ±  6%  interrupts.CPU23.NMI:Non-maskable_interrupts
    479.75 ± 12%     -23.6%     366.75 ±  6%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      2958 ±  6%     +56.2%       4619 ±  4%  interrupts.CPU23.RES:Rescheduling_interrupts
    960345           +43.2%    1375689        interrupts.CPU24.LOC:Local_timer_interrupts
    549.00 ±  8%     -33.3%     366.00 ±  8%  interrupts.CPU24.NMI:Non-maskable_interrupts
    549.00 ±  8%     -33.3%     366.00 ±  8%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      2976 ±  4%     +59.1%       4736 ±  4%  interrupts.CPU24.RES:Rescheduling_interrupts
    960335           +43.3%    1375732        interrupts.CPU25.LOC:Local_timer_interrupts
    541.50 ± 11%     -46.2%     291.50 ± 22%  interrupts.CPU25.NMI:Non-maskable_interrupts
    541.50 ± 11%     -46.2%     291.50 ± 22%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      2986 ±  4%     +59.7%       4770        interrupts.CPU25.RES:Rescheduling_interrupts
    960059           +43.3%    1375626        interrupts.CPU26.LOC:Local_timer_interrupts
    529.00 ±  9%     -42.8%     302.50 ± 11%  interrupts.CPU26.NMI:Non-maskable_interrupts
    529.00 ±  9%     -42.8%     302.50 ± 11%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      3041           +52.3%       4631        interrupts.CPU26.RES:Rescheduling_interrupts
    960367           +43.2%    1375528        interrupts.CPU27.LOC:Local_timer_interrupts
      2905 ±  4%     +70.9%       4963 ±  6%  interrupts.CPU27.RES:Rescheduling_interrupts
    960325           +43.2%    1375585        interrupts.CPU28.LOC:Local_timer_interrupts
      3009 ± 11%     +57.6%       4742 ±  2%  interrupts.CPU28.RES:Rescheduling_interrupts
    960290           +43.3%    1375648        interrupts.CPU29.LOC:Local_timer_interrupts
      2936 ± 14%     +58.6%       4658 ±  2%  interrupts.CPU29.RES:Rescheduling_interrupts
    960600           +43.2%    1375689        interrupts.CPU3.LOC:Local_timer_interrupts
    546.25 ± 11%     -34.9%     355.50 ± 12%  interrupts.CPU3.NMI:Non-maskable_interrupts
    546.25 ± 11%     -34.9%     355.50 ± 12%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    960286           +43.3%    1375748        interrupts.CPU30.LOC:Local_timer_interrupts
      2853 ±  9%     +61.4%       4604 ±  2%  interrupts.CPU30.RES:Rescheduling_interrupts
    960310           +43.3%    1375744        interrupts.CPU31.LOC:Local_timer_interrupts
      2919 ±  6%     +51.1%       4410 ±  7%  interrupts.CPU31.RES:Rescheduling_interrupts
    960330           +43.2%    1375667        interrupts.CPU32.LOC:Local_timer_interrupts
      2996 ±  2%     +46.1%       4375 ± 12%  interrupts.CPU32.RES:Rescheduling_interrupts
    960234           +43.2%    1375505        interrupts.CPU33.LOC:Local_timer_interrupts
    547.50 ± 13%     -30.2%     382.25 ±  6%  interrupts.CPU33.NMI:Non-maskable_interrupts
    547.50 ± 13%     -30.2%     382.25 ±  6%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
      3075 ±  6%     +45.8%       4482 ± 10%  interrupts.CPU33.RES:Rescheduling_interrupts
    960291           +43.3%    1375718        interrupts.CPU34.LOC:Local_timer_interrupts
      2883 ±  6%     +56.4%       4511 ±  8%  interrupts.CPU34.RES:Rescheduling_interrupts
    960236           +43.3%    1375694        interrupts.CPU35.LOC:Local_timer_interrupts
      2759 ±  6%     +71.9%       4745 ±  5%  interrupts.CPU35.RES:Rescheduling_interrupts
    960636           +43.2%    1375756        interrupts.CPU36.LOC:Local_timer_interrupts
      3057 ±  7%     +46.1%       4466 ±  6%  interrupts.CPU36.RES:Rescheduling_interrupts
    960637           +43.2%    1375794        interrupts.CPU37.LOC:Local_timer_interrupts
    584.75 ±  6%     -39.5%     353.75 ±  4%  interrupts.CPU37.NMI:Non-maskable_interrupts
    584.75 ±  6%     -39.5%     353.75 ±  4%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      3144 ±  6%     +38.6%       4357 ± 13%  interrupts.CPU37.RES:Rescheduling_interrupts
    960594           +43.2%    1375658        interrupts.CPU38.LOC:Local_timer_interrupts
    582.25 ± 16%     -37.9%     361.50 ±  6%  interrupts.CPU38.NMI:Non-maskable_interrupts
    582.25 ± 16%     -37.9%     361.50 ±  6%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      3104 ±  2%     +40.8%       4373 ±  9%  interrupts.CPU38.RES:Rescheduling_interrupts
    960595           +43.2%    1375668        interrupts.CPU39.LOC:Local_timer_interrupts
    575.25 ± 14%     -39.5%     347.75 ± 12%  interrupts.CPU39.NMI:Non-maskable_interrupts
    575.25 ± 14%     -39.5%     347.75 ± 12%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      3109 ±  5%     +44.9%       4505 ±  5%  interrupts.CPU39.RES:Rescheduling_interrupts
    960626           +43.2%    1375725        interrupts.CPU4.LOC:Local_timer_interrupts
    578.50 ±  8%     -40.5%     344.25 ± 10%  interrupts.CPU4.NMI:Non-maskable_interrupts
    578.50 ±  8%     -40.5%     344.25 ± 10%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
      3200 ±  4%     +43.1%       4579 ±  5%  interrupts.CPU4.RES:Rescheduling_interrupts
    960614           +43.2%    1375706        interrupts.CPU40.LOC:Local_timer_interrupts
    581.00 ± 14%     -40.1%     348.25 ±  9%  interrupts.CPU40.NMI:Non-maskable_interrupts
    581.00 ± 14%     -40.1%     348.25 ±  9%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      3067 ±  6%     +55.2%       4762 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
    960561           +43.2%    1375676        interrupts.CPU41.LOC:Local_timer_interrupts
    504.50 ± 21%     -35.0%     327.75 ± 21%  interrupts.CPU41.NMI:Non-maskable_interrupts
    504.50 ± 21%     -35.0%     327.75 ± 21%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      3106 ±  4%     +46.0%       4536 ±  4%  interrupts.CPU41.RES:Rescheduling_interrupts
    960651           +43.2%    1375751        interrupts.CPU42.LOC:Local_timer_interrupts
    503.25 ± 23%     -42.7%     288.50 ± 10%  interrupts.CPU42.NMI:Non-maskable_interrupts
    503.25 ± 23%     -42.7%     288.50 ± 10%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      3104 ±  2%     +46.1%       4536 ±  3%  interrupts.CPU42.RES:Rescheduling_interrupts
    960627           +43.2%    1375681        interrupts.CPU43.LOC:Local_timer_interrupts
    552.00 ± 11%     -40.7%     327.50 ± 15%  interrupts.CPU43.NMI:Non-maskable_interrupts
    552.00 ± 11%     -40.7%     327.50 ± 15%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      3032 ±  2%     +55.4%       4711 ±  2%  interrupts.CPU43.RES:Rescheduling_interrupts
    960513           +43.2%    1375747        interrupts.CPU44.LOC:Local_timer_interrupts
    588.25 ±  7%     -45.4%     321.00 ± 11%  interrupts.CPU44.NMI:Non-maskable_interrupts
    588.25 ±  7%     -45.4%     321.00 ± 11%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      3015 ±  3%     +57.3%       4744 ±  2%  interrupts.CPU44.RES:Rescheduling_interrupts
    960612           +43.2%    1375625        interrupts.CPU45.LOC:Local_timer_interrupts
      3155 ±  7%     +49.0%       4701 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
    960536           +43.2%    1375689        interrupts.CPU46.LOC:Local_timer_interrupts
      2994 ±  8%     +51.7%       4542        interrupts.CPU46.RES:Rescheduling_interrupts
    960600           +43.2%    1375692        interrupts.CPU47.LOC:Local_timer_interrupts
      3063 ±  8%     +53.6%       4704 ±  5%  interrupts.CPU47.RES:Rescheduling_interrupts
    960789           +43.2%    1375688        interrupts.CPU48.LOC:Local_timer_interrupts
      2935 ±  2%     +56.0%       4578 ±  5%  interrupts.CPU48.RES:Rescheduling_interrupts
    960531           +43.2%    1375777        interrupts.CPU49.LOC:Local_timer_interrupts
      3038 ±  6%     +50.8%       4581 ±  3%  interrupts.CPU49.RES:Rescheduling_interrupts
    960587           +43.2%    1375671        interrupts.CPU5.LOC:Local_timer_interrupts
      3227           +63.2%       5268 ± 27%  interrupts.CPU5.RES:Rescheduling_interrupts
    960627           +43.2%    1375680        interrupts.CPU50.LOC:Local_timer_interrupts
      3031 ±  6%     +56.4%       4742 ±  6%  interrupts.CPU50.RES:Rescheduling_interrupts
      1681 ±172%    -100.0%       0.50 ±173%  interrupts.CPU51.87:PCI-MSI.31981620-edge.i40e-eth0-TxRx-51
    960555           +43.2%    1375716        interrupts.CPU51.LOC:Local_timer_interrupts
      3062 ±  7%     +50.7%       4616 ±  6%  interrupts.CPU51.RES:Rescheduling_interrupts
    960450           +43.2%    1375720        interrupts.CPU52.LOC:Local_timer_interrupts
    551.25 ±  8%     -30.5%     383.00 ± 13%  interrupts.CPU52.NMI:Non-maskable_interrupts
    551.25 ±  8%     -30.5%     383.00 ± 13%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      3074 ±  5%     +51.0%       4641 ±  2%  interrupts.CPU52.RES:Rescheduling_interrupts
    960557           +43.2%    1375674        interrupts.CPU53.LOC:Local_timer_interrupts
    546.25 ±  8%     -32.3%     370.00 ± 29%  interrupts.CPU53.NMI:Non-maskable_interrupts
    546.25 ±  8%     -32.3%     370.00 ± 29%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      3061           +53.4%       4694 ±  4%  interrupts.CPU53.RES:Rescheduling_interrupts
    959955           +43.3%    1375697        interrupts.CPU54.LOC:Local_timer_interrupts
    585.75 ±  6%     -37.3%     367.25 ±  6%  interrupts.CPU54.NMI:Non-maskable_interrupts
    585.75 ±  6%     -37.3%     367.25 ±  6%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      2918 ±  4%     +61.9%       4726 ±  2%  interrupts.CPU54.RES:Rescheduling_interrupts
    960295           +43.3%    1375697        interrupts.CPU55.LOC:Local_timer_interrupts
    535.50 ± 18%     -38.7%     328.25 ± 15%  interrupts.CPU55.NMI:Non-maskable_interrupts
    535.50 ± 18%     -38.7%     328.25 ± 15%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      2970 ±  3%     +61.2%       4788 ±  3%  interrupts.CPU55.RES:Rescheduling_interrupts
    960252           +43.3%    1375697        interrupts.CPU56.LOC:Local_timer_interrupts
    538.75 ±  7%     -44.9%     296.75 ± 18%  interrupts.CPU56.NMI:Non-maskable_interrupts
    538.75 ±  7%     -44.9%     296.75 ± 18%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      3066 ±  6%     +54.6%       4741 ±  2%  interrupts.CPU56.RES:Rescheduling_interrupts
    960316           +43.3%    1375727        interrupts.CPU57.LOC:Local_timer_interrupts
    573.00 ± 15%     -43.8%     322.00 ± 19%  interrupts.CPU57.NMI:Non-maskable_interrupts
    573.00 ± 15%     -43.8%     322.00 ± 19%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      2914 ±  4%     +57.3%       4582 ±  7%  interrupts.CPU57.RES:Rescheduling_interrupts
    960271           +43.3%    1375634        interrupts.CPU58.LOC:Local_timer_interrupts
    525.00 ± 10%     -33.6%     348.50 ±  9%  interrupts.CPU58.NMI:Non-maskable_interrupts
    525.00 ± 10%     -33.6%     348.50 ±  9%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      2919 ±  3%     +51.3%       4415 ±  6%  interrupts.CPU58.RES:Rescheduling_interrupts
    960331           +43.2%    1375652        interrupts.CPU59.LOC:Local_timer_interrupts
    488.00 ± 13%     -26.1%     360.50 ±  3%  interrupts.CPU59.NMI:Non-maskable_interrupts
    488.00 ± 13%     -26.1%     360.50 ±  3%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      2942 ±  5%     +56.6%       4607 ±  6%  interrupts.CPU59.RES:Rescheduling_interrupts
    960564           +43.2%    1375751        interrupts.CPU6.LOC:Local_timer_interrupts
    490.50 ± 21%     -38.6%     301.00 ±  5%  interrupts.CPU6.NMI:Non-maskable_interrupts
    490.50 ± 21%     -38.6%     301.00 ±  5%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      3214 ±  3%     +44.9%       4658 ±  3%  interrupts.CPU6.RES:Rescheduling_interrupts
    960338           +43.2%    1375563        interrupts.CPU60.LOC:Local_timer_interrupts
    550.50 ± 10%     -33.3%     367.25 ± 11%  interrupts.CPU60.NMI:Non-maskable_interrupts
    550.50 ± 10%     -33.3%     367.25 ± 11%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      2956 ±  4%     +55.1%       4584 ±  6%  interrupts.CPU60.RES:Rescheduling_interrupts
    960364           +43.2%    1375687        interrupts.CPU61.LOC:Local_timer_interrupts
    523.75 ± 14%     -46.9%     278.00 ± 26%  interrupts.CPU61.NMI:Non-maskable_interrupts
    523.75 ± 14%     -46.9%     278.00 ± 26%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      3118 ±  4%     +44.4%       4502 ±  5%  interrupts.CPU61.RES:Rescheduling_interrupts
    960292           +43.3%    1375635        interrupts.CPU62.LOC:Local_timer_interrupts
      3097 ±  4%     +42.9%       4425 ±  4%  interrupts.CPU62.RES:Rescheduling_interrupts
    960341           +43.2%    1375680        interrupts.CPU63.LOC:Local_timer_interrupts
    510.50 ± 14%     -35.1%     331.25 ±  2%  interrupts.CPU63.NMI:Non-maskable_interrupts
    510.50 ± 14%     -35.1%     331.25 ±  2%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      3174 ±  3%     +42.0%       4508 ±  9%  interrupts.CPU63.RES:Rescheduling_interrupts
    960310           +43.3%    1375721        interrupts.CPU64.LOC:Local_timer_interrupts
    529.50 ± 14%     -32.3%     358.50 ±  8%  interrupts.CPU64.NMI:Non-maskable_interrupts
    529.50 ± 14%     -32.3%     358.50 ±  8%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      3028 ±  7%     +45.7%       4410 ±  3%  interrupts.CPU64.RES:Rescheduling_interrupts
    960268           +43.3%    1375675        interrupts.CPU65.LOC:Local_timer_interrupts
    534.00 ± 12%     -30.1%     373.25 ±  7%  interrupts.CPU65.NMI:Non-maskable_interrupts
    534.00 ± 12%     -30.1%     373.25 ±  7%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      3059 ±  6%     +49.7%       4578 ±  5%  interrupts.CPU65.RES:Rescheduling_interrupts
    960252           +43.3%    1375680        interrupts.CPU66.LOC:Local_timer_interrupts
    547.25 ±  9%     -33.9%     362.00 ±  8%  interrupts.CPU66.NMI:Non-maskable_interrupts
    547.25 ±  9%     -33.9%     362.00 ±  8%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      3050 ±  6%     +50.5%       4589 ±  5%  interrupts.CPU66.RES:Rescheduling_interrupts
    960281           +43.3%    1375628        interrupts.CPU67.LOC:Local_timer_interrupts
    491.00 ±  6%     -21.9%     383.50 ±  7%  interrupts.CPU67.NMI:Non-maskable_interrupts
    491.00 ±  6%     -21.9%     383.50 ±  7%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      3090 ±  4%     +43.6%       4439        interrupts.CPU67.RES:Rescheduling_interrupts
    960327           +43.2%    1375585        interrupts.CPU68.LOC:Local_timer_interrupts
    556.75 ± 13%     -30.0%     390.00 ±  6%  interrupts.CPU68.NMI:Non-maskable_interrupts
    556.75 ± 13%     -30.0%     390.00 ±  6%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      3148           +43.8%       4526        interrupts.CPU68.RES:Rescheduling_interrupts
    960260           +43.3%    1375666        interrupts.CPU69.LOC:Local_timer_interrupts
    550.00 ± 12%     -31.4%     377.25 ±  7%  interrupts.CPU69.NMI:Non-maskable_interrupts
    550.00 ± 12%     -31.4%     377.25 ±  7%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
      3114 ±  2%     +46.6%       4565 ±  2%  interrupts.CPU69.RES:Rescheduling_interrupts
    410.75 ±  3%     +44.1%     591.75 ±  3%  interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
    960569           +43.2%    1375605        interrupts.CPU7.LOC:Local_timer_interrupts
    546.75 ±  9%     -43.7%     307.75 ± 14%  interrupts.CPU7.NMI:Non-maskable_interrupts
    546.75 ±  9%     -43.7%     307.75 ± 14%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      2889 ±  4%     +53.5%       4435 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
    960484           +43.2%    1375854        interrupts.CPU70.LOC:Local_timer_interrupts
      3239 ±  5%     +42.2%       4605        interrupts.CPU70.RES:Rescheduling_interrupts
    960766           +43.3%    1376330        interrupts.CPU71.LOC:Local_timer_interrupts
    568.25 ±  8%     -32.4%     384.00 ±  2%  interrupts.CPU71.NMI:Non-maskable_interrupts
    568.25 ±  8%     -32.4%     384.00 ±  2%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      3017 ±  6%     +56.4%       4719        interrupts.CPU71.RES:Rescheduling_interrupts
    960486           +43.2%    1375677        interrupts.CPU8.LOC:Local_timer_interrupts
    578.25 ±  8%     -44.5%     320.75 ± 17%  interrupts.CPU8.NMI:Non-maskable_interrupts
    578.25 ±  8%     -44.5%     320.75 ± 17%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      3050 ±  7%     +56.8%       4783 ± 11%  interrupts.CPU8.RES:Rescheduling_interrupts
    960636           +43.2%    1375675        interrupts.CPU9.LOC:Local_timer_interrupts
    561.75 ±  7%     -41.3%     329.75 ± 14%  interrupts.CPU9.NMI:Non-maskable_interrupts
    561.75 ±  7%     -41.3%     329.75 ± 14%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      3299 ± 19%     +40.4%       4631 ±  5%  interrupts.CPU9.RES:Rescheduling_interrupts
  69152219           +43.2%   99050063        interrupts.LOC:Local_timer_interrupts
     72.00          +100.0%     144.00        interrupts.MCP:Machine_check_polls
     37793 ±  4%     -33.4%      25156        interrupts.NMI:Non-maskable_interrupts
     37793 ±  4%     -33.4%      25156        interrupts.PMI:Performance_monitoring_interrupts
    222056           +50.1%     333203        interrupts.RES:Rescheduling_interrupts
     20.17            -1.6       18.57        perf-profile.calltrace.cycles-pp.unlink
     19.91            -1.6       18.34        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     19.91            -1.6       18.34        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
     19.65            -1.5       18.13        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
     12.92            -1.1       11.87        perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
     13.06            -1.0       12.02        perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
     17.17 ±  2%      -1.0       16.18        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     17.18 ±  2%      -1.0       16.18        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
     17.38 ±  2%      -1.0       16.39        perf-profile.calltrace.cycles-pp.creat
     17.03 ±  2%      -1.0       16.05        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     17.02 ±  2%      -1.0       16.05        perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     16.78 ±  2%      -0.9       15.85        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.76 ±  2%      -0.9       15.84        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      7.43 ±  3%      -0.9        6.57        perf-profile.calltrace.cycles-pp.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      7.22 ±  3%      -0.8        6.38        perf-profile.calltrace.cycles-pp.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ±  3%      -0.8        6.34        perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64
      7.16 ±  3%      -0.8        6.31        perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat
      9.69            -0.6        9.12        perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64
      9.71            -0.6        9.15        perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.84            -0.5        9.29        perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      4.00            -0.5        3.46 ±  4%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_create.xfs_generic_create.path_openat.do_filp_open
      3.93            -0.5        3.41 ±  4%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_create.xfs_generic_create.path_openat
      2.57 ±  3%      -0.4        2.13 ±  3%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
      2.51 ±  3%      -0.4        2.08 ±  3%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      4.33 ±  3%      -0.4        3.97        perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
      2.52 ±  2%      -0.3        2.19 ±  4%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink
      2.60 ±  2%      -0.3        2.27 ±  4%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      1.55            -0.3        1.24 ±  9%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_create.xfs_generic_create
      5.00 ±  2%      -0.3        4.72        perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      5.00 ±  2%      -0.3        4.73        perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      3.09 ±  4%      -0.2        2.89 ±  2%  perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      2.31            -0.2        2.10 ±  5%  perf-profile.calltrace.cycles-pp.up.xfs_buf_unlock.xfs_buf_item_release.xfs_log_commit_cil.__xfs_trans_commit
      4.41 ±  2%      -0.2        4.22        perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create
      0.84 ±  5%      -0.2        0.68 ±  8%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start
      0.94 ±  2%      -0.1        0.79 ±  6%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink
      0.84 ±  2%      -0.1        0.70 ±  7%  perf-profile.calltrace.cycles-pp.xfs_buf_item_format.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_create
      1.86 ±  3%      -0.1        1.73 ±  2%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree
      0.89 ±  4%      -0.1        0.76 ±  9%  perf-profile.calltrace.cycles-pp.xfs_buf_item_release.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive
      0.84 ±  4%      -0.1        0.71 ±  7%  perf-profile.calltrace.cycles-pp.xfs_buf_unlock.xfs_buf_item_release.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree
      1.16 ±  4%      -0.1        1.04 ±  3%  perf-profile.calltrace.cycles-pp.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      0.87 ±  4%      -0.1        0.75 ±  8%  perf-profile.calltrace.cycles-pp.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      1.89 ±  2%      -0.1        1.77 ±  2%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.70 ±  4%      -0.1        0.59 ±  7%  perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink
      0.73 ±  4%      -0.1        0.63 ±  6%  perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink
      0.67 ±  8%      -0.1        0.57 ±  5%  perf-profile.calltrace.cycles-pp.xfs_btree_lookup_get_block.xfs_btree_lookup.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc
      0.88 ±  6%      -0.1        0.79 ±  6%  perf-profile.calltrace.cycles-pp.xfs_btree_lookup.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc
      1.02 ±  4%      -0.1        0.94 ±  5%  perf-profile.calltrace.cycles-pp.xfs_dir2_node_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink
      0.77 ±  3%      -0.1        0.69 ±  5%  perf-profile.calltrace.cycles-pp.xfs_buf_unlock.xfs_buf_item_release.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      1.16 ±  6%      +0.2        1.37 ±  5%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      1.66 ±  7%      +0.3        1.99 ±  3%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter
      1.74 ±  4%      +0.3        2.08 ±  3%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state
      3.27 ±  2%      +0.5        3.80 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter
      0.00            +0.6        0.61 ±  5%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.13 ±173%      +0.6        0.74 ±  4%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      1.41 ±  2%      +0.6        2.05 ±  2%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.38 ±  2%      +1.1        6.43 ±  3%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter.do_idle
     53.08            +1.3       54.35        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     53.58            +1.5       55.05        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.86            +1.9        7.78 ±  3%  perf-profile.calltrace.cycles-pp.__irqentry_text_start.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     58.56            +2.1       60.70        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     58.58            +2.2       60.73        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     58.58            +2.2       60.73        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     59.33            +2.2       61.52        perf-profile.calltrace.cycles-pp.secondary_startup_64
     38.81            -2.4       36.36        perf-profile.children.cycles-pp.do_syscall_64
     38.83            -2.4       36.39        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     20.21            -1.6       18.61        perf-profile.children.cycles-pp.unlink
     19.65            -1.5       18.14        perf-profile.children.cycles-pp.do_unlinkat
      9.19            -1.3        7.89 ±  2%  perf-profile.children.cycles-pp.__xfs_trans_commit
      8.99            -1.3        7.70 ±  2%  perf-profile.children.cycles-pp.xfs_log_commit_cil
     12.92            -1.1       11.87        perf-profile.children.cycles-pp.xfs_create
     13.06            -1.0       12.02        perf-profile.children.cycles-pp.xfs_generic_create
     17.40 ±  2%      -1.0       16.41        perf-profile.children.cycles-pp.creat
     17.07 ±  2%      -1.0       16.09        perf-profile.children.cycles-pp.do_sys_openat2
     17.08 ±  2%      -1.0       16.11        perf-profile.children.cycles-pp.do_sys_open
     16.82 ±  2%      -0.9       15.89        perf-profile.children.cycles-pp.do_filp_open
     16.81 ±  2%      -0.9       15.88        perf-profile.children.cycles-pp.path_openat
      7.43 ±  3%      -0.9        6.57        perf-profile.children.cycles-pp.destroy_inode
      7.22 ±  3%      -0.8        6.38        perf-profile.children.cycles-pp.xfs_fs_destroy_inode
      7.16 ±  3%      -0.8        6.31        perf-profile.children.cycles-pp.xfs_inactive_ifree
      7.18 ±  3%      -0.8        6.34        perf-profile.children.cycles-pp.xfs_inactive
      9.70            -0.6        9.13        perf-profile.children.cycles-pp.xfs_remove
      3.01            -0.6        2.45 ±  5%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      9.71            -0.6        9.15        perf-profile.children.cycles-pp.xfs_vn_unlink
      9.84            -0.5        9.29        perf-profile.children.cycles-pp.vfs_unlink
      4.00            -0.5        3.50        perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      4.33 ±  3%      -0.4        3.97        perf-profile.children.cycles-pp.xfs_ifree
      2.92 ±  2%      -0.3        2.58 ±  3%  perf-profile.children.cycles-pp.xfs_buf_read_map
      2.76 ±  2%      -0.3        2.44 ±  3%  perf-profile.children.cycles-pp.xfs_buf_get_map
      5.00 ±  2%      -0.3        4.72        perf-profile.children.cycles-pp.xfs_ialloc
      5.00 ±  2%      -0.3        4.73        perf-profile.children.cycles-pp.xfs_dir_ialloc
      2.90            -0.2        2.65 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      2.48 ±  3%      -0.2        2.23 ±  3%  perf-profile.children.cycles-pp.xfs_buf_find
      2.59            -0.2        2.37 ±  6%  perf-profile.children.cycles-pp.xfs_buf_item_release
      2.44            -0.2        2.22 ±  5%  perf-profile.children.cycles-pp.xfs_buf_unlock
      3.10 ±  4%      -0.2        2.89 ±  2%  perf-profile.children.cycles-pp.xfs_difree
      1.16 ±  4%      -0.2        0.95 ±  4%  perf-profile.children.cycles-pp.xfs_trans_unreserve_and_mod_sb
      2.32            -0.2        2.12 ±  5%  perf-profile.children.cycles-pp.up
      2.15 ±  2%      -0.2        1.95 ±  4%  perf-profile.children.cycles-pp.xfs_read_agi
      1.28 ±  3%      -0.2        1.09 ±  3%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      1.52 ±  3%      -0.2        1.32 ±  9%  perf-profile.children.cycles-pp.xfs_buf_item_format_segment
      1.58 ±  3%      -0.2        1.39 ±  8%  perf-profile.children.cycles-pp.xfs_buf_item_format
      4.41 ±  2%      -0.2        4.22        perf-profile.children.cycles-pp.xfs_dialloc
      1.04 ±  5%      -0.2        0.85 ±  5%  perf-profile.children.cycles-pp.__percpu_counter_compare
      1.02 ±  5%      -0.2        0.83 ±  6%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.58 ±  4%      -0.2        0.40 ± 11%  perf-profile.children.cycles-pp.xfs_inode_item_format
      1.06 ±  6%      -0.2        0.88 ±  5%  perf-profile.children.cycles-pp.xfs_mod_ifree
      1.92 ±  2%      -0.2        1.75        perf-profile.children.cycles-pp.xfs_btree_lookup
      0.99 ±  3%      -0.2        0.84 ±  6%  perf-profile.children.cycles-pp.xfs_btree_read_buf_block
      0.92 ±  6%      -0.1        0.77 ±  3%  perf-profile.children.cycles-pp.rwsem_wake
      1.02 ±  5%      -0.1        0.88 ±  5%  perf-profile.children.cycles-pp.xfs_da_read_buf
      1.17 ±  4%      -0.1        1.04 ±  3%  perf-profile.children.cycles-pp.xfs_iunlink_remove
      0.87 ±  4%      -0.1        0.75 ±  8%  perf-profile.children.cycles-pp.xfs_iunlink
      0.85 ±  2%      -0.1        0.73 ±  6%  perf-profile.children.cycles-pp._xfs_trans_bjoin
      0.25 ± 13%      -0.1        0.14 ± 18%  perf-profile.children.cycles-pp.xfs_inode_item_format_data_fork
      0.79 ±  6%      -0.1        0.69 ±  2%  perf-profile.children.cycles-pp.wake_up_q
      2.30            -0.1        2.20        perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.22 ± 14%      -0.1        0.12 ± 16%  perf-profile.children.cycles-pp.xfs_iextents_copy
      1.93 ±  2%      -0.1        1.83 ±  2%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.56 ±  4%      -0.1        0.47 ±  9%  perf-profile.children.cycles-pp.xfs_verify_agino
      0.30 ±  7%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.xfs_inobt_btrec_to_irec
      0.58 ±  4%      -0.1        0.50 ±  5%  perf-profile.children.cycles-pp.xfs_dir3_block_read
      1.02 ±  4%      -0.1        0.94 ±  5%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.21 ±  9%      -0.1        0.14 ± 19%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.47 ±  7%      -0.1        0.40 ±  4%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.34 ±  5%      -0.1        0.27 ±  2%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.40 ±  2%      -0.1        0.34 ± 11%  perf-profile.children.cycles-pp.xfs_iget
      0.42 ±  3%      -0.1        0.36 ±  9%  perf-profile.children.cycles-pp.xfs_iunlink_update_inode
      0.48 ±  3%      -0.1        0.42 ±  9%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.25 ±  5%      -0.1        0.19 ± 12%  perf-profile.children.cycles-pp._xfs_buf_obj_cmp
      0.27 ± 12%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.14 ± 15%      -0.1        0.08 ±  8%  perf-profile.children.cycles-pp.xfs_bmap_validate_extent
      0.23 ± 11%      -0.0        0.18 ±  6%  perf-profile.children.cycles-pp.down_write
      0.13 ± 16%      -0.0        0.08 ± 19%  perf-profile.children.cycles-pp.complete_walk
      0.07 ±  5%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.xfs_droplink
      0.16 ± 10%      -0.0        0.11 ± 14%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.25 ±  7%      -0.0        0.21 ±  7%  perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
      0.16 ± 12%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.08 ±  8%      -0.0        0.05 ± 58%  perf-profile.children.cycles-pp.xlog_cil_push_work
      0.19 ±  2%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.inode_permission
      0.12 ±  7%      -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.12 ±  4%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.__list_add_valid
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.07 ± 15%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.__might_sleep
      0.04 ± 58%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.xfs_btree_delrec
      0.07 ± 20%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.06 ± 11%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.rcu_nmi_enter
      0.23 ±  6%      +0.0        0.26 ±  5%  perf-profile.children.cycles-pp.process_one_work
      0.08 ±  5%      +0.0        0.11 ± 18%  perf-profile.children.cycles-pp.menu_reflect
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.interrupt_entry
      0.04 ± 58%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.xfs_btree_delete
      0.05 ±  9%      +0.0        0.09 ± 14%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.24 ±  5%      +0.0        0.27 ±  3%  perf-profile.children.cycles-pp.worker_thread
      0.04 ± 57%      +0.0        0.07 ± 15%  perf-profile.children.cycles-pp.timerqueue_add
      0.05 ± 62%      +0.0        0.09 ± 14%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.21 ±  9%      +0.0        0.25 ±  5%  perf-profile.children.cycles-pp.native_sched_clock
      0.30 ±  6%      +0.0        0.34 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
      0.29 ±  7%      +0.0        0.34 ±  8%  perf-profile.children.cycles-pp.kthread
      0.22 ±  6%      +0.0        0.26 ±  6%  perf-profile.children.cycles-pp.sched_clock
      0.08 ± 15%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.vfs_read
      0.08 ± 15%      +0.0        0.13 ± 10%  perf-profile.children.cycles-pp.ksys_read
      0.10 ± 18%      +0.0        0.14 ± 12%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.06 ±  6%      +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.run_timer_softirq
      0.05 ± 67%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.new_sync_read
      0.23 ±  8%      +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.24 ± 14%      +0.1        0.29 ±  7%  perf-profile.children.cycles-pp.load_balance
      0.09 ±  8%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.read
      0.08 ±  5%      +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.01 ±173%      +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.22 ± 13%      +0.1        0.27 ±  3%  perf-profile.children.cycles-pp.update_blocked_averages
      0.12 ± 21%      +0.1        0.18 ± 11%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.07 ±  5%      +0.1        0.13 ± 11%  perf-profile.children.cycles-pp.__run_timers
      0.23 ± 16%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.18 ± 13%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.04 ± 58%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.vfprintf
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.fprintf
      0.01 ±173%      +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.05 ±  8%      +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      1.04 ±  5%      +0.1        1.12 ±  3%  perf-profile.children.cycles-pp.update_process_times
      0.04 ± 58%      +0.1        0.13 ± 10%  perf-profile.children.cycles-pp.__handle_mm_fault
      1.07 ±  4%      +0.1        1.16 ±  3%  perf-profile.children.cycles-pp.tick_sched_handle
      0.57 ±  6%      +0.1        0.66 ±  4%  perf-profile.children.cycles-pp.__orc_find
      0.06 ± 13%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.02 ±173%      +0.1        0.11 ± 10%  perf-profile.children.cycles-pp.delay_tsc
      0.04 ± 57%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.handle_mm_fault
      0.12 ± 14%      +0.1        0.22 ± 17%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.07 ± 16%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.page_fault
      0.70 ±  8%      +0.1        0.80 ±  6%  perf-profile.children.cycles-pp.orc_find
      0.24 ± 13%      +0.1        0.37 ± 12%  perf-profile.children.cycles-pp.tick_irq_enter
      0.35 ±  8%      +0.1        0.48 ±  4%  perf-profile.children.cycles-pp.read_tsc
      0.29 ±  7%      +0.1        0.43 ±  9%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.17 ± 12%      +0.1        0.31 ± 15%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.19 ± 15%      +0.1        0.33 ± 13%  perf-profile.children.cycles-pp.io_serial_in
      0.25 ±  4%      +0.2        0.41 ± 11%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.28 ±  8%      +0.2        0.45 ±  6%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.32 ± 11%      +0.2        0.48 ±  7%  perf-profile.children.cycles-pp.irq_enter
      0.38 ±  5%      +0.2        0.56 ±  5%  perf-profile.children.cycles-pp.native_write_msr
      0.23 ±  5%      +0.2        0.42 ±  7%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.23 ±  4%      +0.2        0.42 ±  6%  perf-profile.children.cycles-pp.uart_console_write
      0.23 ±  4%      +0.2        0.43 ±  7%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.23 ±  4%      +0.2        0.44 ±  6%  perf-profile.children.cycles-pp.serial8250_console_write
      0.25 ±  4%      +0.2        0.46 ±  5%  perf-profile.children.cycles-pp.irq_work_run
      0.25 ±  4%      +0.2        0.46 ±  5%  perf-profile.children.cycles-pp.printk
      0.25 ±  4%      +0.2        0.46 ±  5%  perf-profile.children.cycles-pp.vprintk_emit
      0.25 ±  4%      +0.2        0.46 ±  5%  perf-profile.children.cycles-pp.console_unlock
      0.26 ±  3%      +0.2        0.47 ±  6%  perf-profile.children.cycles-pp.irq_work_run_list
      0.25 ±  4%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.25 ±  4%      +0.2        0.46 ±  6%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      1.22 ±  6%      +0.2        1.45 ±  6%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.38 ±  8%      +0.2        0.62 ±  5%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.2        0.24 ± 14%  perf-profile.children.cycles-pp.xfs_dir2_block_lookup
      2.30 ±  2%      +0.3        2.58 ±  3%  perf-profile.children.cycles-pp.xfs_dir2_block_lookup_int
      0.47 ±  8%      +0.3        0.76 ±  4%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.3        0.29 ±  9%  perf-profile.children.cycles-pp.xfs_dir_lookup
      0.00            +0.3        0.30 ±  9%  perf-profile.children.cycles-pp.xfs_lookup
      0.00            +0.3        0.30 ±  9%  perf-profile.children.cycles-pp.xfs_vn_lookup
      1.74 ±  7%      +0.3        2.09 ±  4%  perf-profile.children.cycles-pp.irq_exit
      1.88 ±  4%      +0.4        2.26 ±  2%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      3.47            +0.6        4.08 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.44 ±  2%      +0.7        2.10 ±  3%  perf-profile.children.cycles-pp.menu_select
      5.66            +1.2        6.83 ±  3%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      6.18            +1.4        7.55 ±  3%  perf-profile.children.cycles-pp.__irqentry_text_start
     54.26            +1.5       55.73        perf-profile.children.cycles-pp.cpuidle_enter_state
     54.27            +1.5       55.75        perf-profile.children.cycles-pp.cpuidle_enter
     58.58            +2.2       60.73        perf-profile.children.cycles-pp.start_secondary
     59.33            +2.2       61.52        perf-profile.children.cycles-pp.secondary_startup_64
     59.33            +2.2       61.52        perf-profile.children.cycles-pp.cpu_startup_entry
     59.34            +2.2       61.55        perf-profile.children.cycles-pp.do_idle
      0.58 ±  3%      -0.1        0.45 ±  9%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.95 ±  2%      -0.1        0.83 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.63 ±  2%      -0.1        0.53 ±  3%  perf-profile.self.cycles-pp.unwind_next_frame
      1.06 ±  4%      -0.1        0.96 ±  3%  perf-profile.self.cycles-pp.memcpy_erms
      0.46 ±  3%      -0.1        0.36 ±  7%  perf-profile.self.cycles-pp.xlog_cil_insert_items
      0.58 ±  7%      -0.1        0.48 ± 10%  perf-profile.self.cycles-pp.__percpu_counter_sum
      0.96 ±  5%      -0.1        0.88 ±  7%  perf-profile.self.cycles-pp.xfs_agino_range
      0.21 ±  9%      -0.1        0.13 ± 19%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.34 ±  4%      -0.1        0.27 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.30 ±  9%      -0.1        0.23 ± 10%  perf-profile.self.cycles-pp.xfs_inode_item_format
      0.28 ±  6%      -0.1        0.21 ±  5%  perf-profile.self.cycles-pp.xfs_inobt_btrec_to_irec
      0.38 ±  6%      -0.1        0.31 ±  7%  perf-profile.self.cycles-pp.xfs_buf_item_init
      0.27 ±  9%      -0.1        0.20 ±  8%  perf-profile.self.cycles-pp.xfs_buf_get_map
      0.37 ±  2%      -0.1        0.31 ±  7%  perf-profile.self.cycles-pp._xfs_trans_bjoin
      0.27 ± 12%      -0.1        0.21 ±  6%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.24 ±  6%      -0.1        0.19 ± 12%  perf-profile.self.cycles-pp._xfs_buf_obj_cmp
      0.24 ±  4%      -0.0        0.20 ±  6%  perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
      0.45            -0.0        0.41 ±  5%  perf-profile.self.cycles-pp.do_syscall_64
      0.15 ±  8%      -0.0        0.11 ± 14%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.12 ± 12%      -0.0        0.09 ± 13%  perf-profile.self.cycles-pp.note_gp_changes
      0.16 ±  4%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.inode_permission
      0.11 ± 10%      -0.0        0.07 ± 26%  perf-profile.self.cycles-pp.xfs_iget
      0.14 ±  3%      -0.0        0.10 ± 18%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.18 ±  8%      -0.0        0.15 ± 10%  perf-profile.self.cycles-pp.down_write
      0.17 ±  8%      -0.0        0.14 ± 10%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.15 ±  6%      -0.0        0.12 ± 12%  perf-profile.self.cycles-pp.xfs_verify_agbno
      0.18 ±  6%      -0.0        0.15 ±  8%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.14 ± 15%      -0.0        0.11 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.08 ± 15%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.poll_idle
      0.10 ± 10%      -0.0        0.08 ± 13%  perf-profile.self.cycles-pp.xfs_create
      0.07            -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.xlog_space_left
      0.23 ±  3%      +0.0        0.26 ±  6%  perf-profile.self.cycles-pp.xfs_btree_get_rec
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.interrupt_entry
      0.12 ±  7%      +0.0        0.15 ± 10%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.10 ± 24%      +0.0        0.13 ±  9%  perf-profile.self.cycles-pp.native_read_msr
      0.05 ± 58%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.11 ± 10%      +0.0        0.15 ±  7%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.32 ±  8%      +0.1        0.37 ±  8%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.09 ± 11%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.04 ±107%      +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.56 ±  6%      +0.1        0.65 ±  3%  perf-profile.self.cycles-pp.__orc_find
      0.02 ±173%      +0.1        0.11 ± 10%  perf-profile.self.cycles-pp.delay_tsc
      0.32 ± 10%      +0.1        0.45 ±  5%  perf-profile.self.cycles-pp.read_tsc
      0.19 ± 15%      +0.1        0.33 ± 13%  perf-profile.self.cycles-pp.io_serial_in
      0.25 ±  4%      +0.2        0.41 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.38 ±  5%      +0.2        0.56 ±  5%  perf-profile.self.cycles-pp.native_write_msr
      0.60 ±  2%      +0.2        0.80 ±  3%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.83 ±  4%      +0.3        1.16 ±  4%  perf-profile.self.cycles-pp.menu_select

                                  aim7.jobs-per-min                             
                                                                                
  45000 +-------------------------------------------------------------------+   
        |         .+.                                           .+.         |   
  40000 |.+. +.+.+   ++.+   ++.+.+.+.+ .+.+.+.+ .+.+. .++.+.+.++   +.+.++.+.|   
  35000 |-+ +           :   :         +        +     +                      |   
        |               :   :                                               |   
  30000 |-O    O O       :  :O                                              |   
  25000 |-+ OO     O OO O:O:O  O O O OO O O O OO                            |   
        |                : :                                                |   
  20000 |-+              : :                                                |   
  15000 |-+              : :                                                |   
        |                : :                                                |   
  10000 |-+               ::                                                |   
   5000 |-+               :                                                 |   
        |                 :                                                 |   
      0 +-------------------------------------------------------------------+   

                               aim7.time.user_time                              
                                                                                
  80 +----------------------------------------------------------------------+   
     | O O OO O O O O O O OO O O O O O O OO O O                             |   
  70 |-+                                                                    |   
  60 |-+                                                                    |   
     |                                                                      |   
  50 |.+.+.++.+.+.+.+.+   ++.+.+.+.+.+.+.++.+.+.+.+.+.++.+.+.+.   .+.++.+.+.|   
     |                :   :                                    +.+          |   
  40 |-+              :   :                                                 |   
     |                 : :                                                  |   
  30 |-+               : :                                                  |   
  20 |-+               : :                                                  |   
     |                 : :                                                  |   
  10 |-+                :                                                   |   
     |                  :                                                   |   
   0 +----------------------------------------------------------------------+   

                               aim7.time.system_time                            
                                                                                
  1800 +--------------------------------------------------------------------+   
       |                            O O O  O O O                            |   
  1600 |-O OO O O O O OO O O O OO O       O                                 |   
  1400 |-+                                                                  |   
       |                                                                    |   
  1200 |.+.++.+.                 .+.+.+.+.++.+.+.+.++.+.+.+.++.   .+.+.++.+.|   
  1000 |-+      +.+.+.++   +.+.++                              +.+          |   
       |               :   :                                                |   
   800 |-+              :  :                                                |   
   600 |-+              : :                                                 |   
       |                : :                                                 |   
   400 |-+              : :                                                 |   
   200 |-+               ::                                                 |   
       |                 :                                                  |   
     0 +--------------------------------------------------------------------+   

                              aim7.time.elapsed_time                            
                                                                                
  700 +---------------------------------------------------------------------+   
      | O O OO O O O O OO O O                                               |   
  600 |-+                                                                   |   
      |                                                                     |   
  500 |.+.+.++.                  .+ .+.+.+.+. .++.+.+.+.+.++.+.   .+.++.+.+.|   
      | +      +.+.+.+.+  +.+.+.+  +         +                 +.+          |   
  400 |-+              :  :                                                 |   
      |                :  :                                                 |   
  300 |-+              : :                                                  |   
      |                : :                                                  |   
  200 |-+               ::                                                  |   
      |                 ::                                                  |   
  100 |-+               :                                                   |   
      |                 :                                                   |   
    0 +---------------------------------------------------------------------+   

                            aim7.time.elapsed_time.max                          
                                                                                
  700 +---------------------------------------------------------------------+   
      | O O OO O O O O OO O O                                               |   
  600 |-+                                                                   |   
      |                                                                     |   
  500 |.+.+.++.                  .+ .+.+.+.+. .++.+.+.+.+.++.+.   .+.++.+.+.|   
      | +      +.+.+.+.+  +.+.+.+  +         +                 +.+          |   
  400 |-+              :  :                                                 |   
      |                :  :                                                 |   
  300 |-+              : :                                                  |   
      |                : :                                                  |   
  200 |-+               ::                                                  |   
      |                 ::                                                  |   
  100 |-+               :                                                   |   
      |                 :                                                   |   
    0 +---------------------------------------------------------------------+   

[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-kblr-hp1: 8 threads Kaby Lake-R with 8G memory
=========================================================================================
compiler/iterations/kconfig/mode/rootfs/tbox_group/testcase:
  gcc-9/10/x86_64-rhel-7.6/freeze/debian-x86_64-20180403.cgz/lkp-kblr-hp1/suspend-stress

commit: 
  21bf3e69e5 ("cpufreq: Register governors at core_initcall")
  d83f959b5e ("cpufreq: Specify default governor on command line")

21bf3e69e51e91cd d83f959b5e7a6378a4afbff23de 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4          100%           4:4     kmsg.cpufreq:cpufreq_online:Failed_to_initialize_policy_for_cpu:#(-#)
         %stddev     %change         %stddev
             \          |                \  
      2514 ±  2%      +7.9%       2713        suspend-stress.time.involuntary_context_switches

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Thanks,
Rong Chen