Subject: Re: Serious performance degradation in Linux 4.15

From: Matt Fleming <[email-masked]>
Date: Wed, 14 Feb 2018 22:46:20 +0000

On Mon, 12 Feb, at 04:16:42PM, Peter Zijlstra wrote:
> On Fri, Feb 09, 2018 at 05:59:12PM +0000, Jon Maloy wrote:
> > Command for TCP:
> > "netperf TCP_STREAM  (netperf -n 4 -f m -c 4 -C 4 -P 1 -H 10.0.0.1 -t TCP_STREAM -l 10 -- -O THROUGHPUT)"
> > Command for TIPC:
> > "netperf TIPC_STREAM (netperf -n 4 -f m -c 4 -C 4 -P 1 -H 10.0.0.1 -t TCP_STREAM -l 10 -- -O THROUGHPUT)"
> 
> That looks like identical tests to me. And my netperf (debian testing)
> doesn't appear to have -t TIPC_STREAM.
> 
> Please try a coherent report and I'll have another look. Don't (again)
> forget to mention what kind of setup you're running this on.
> 
> 
> On my IVB-EP (2 sockets, 10 cores, 2 threads), performance cpufreq,
> PTI=n RETPOLINE=n, I get:

Here's some more numbers. This is with RETPOLINE=y but you'll see it
doesn't make much of a difference. Oh, this is also with powersave
cpufreq governor.

The 'tip+' column is tip/master, commit ca96ad6978c3 ("Merge branch 'x86/mm'")

The 'tip-plus-patch+' column is tip/master plus Peter's patch from
[email-masked]

netperf-tcp
                            4.15.0-rc1                 4.15.0             4.16.0-rc1             4.16.0-rc1
                               vanilla                vanilla                   tip+        tip-plus-patch+
Min       64        1804.73 (   0.00%)      951.28 ( -47.29%)      956.77 ( -46.99%)      936.19 ( -48.13%)
Min       128       3352.00 (   0.00%)     1847.80 ( -44.87%)     1831.41 ( -45.36%)     1808.88 ( -46.04%)
Min       256       5619.02 (   0.00%)     3327.27 ( -40.79%)     3287.00 ( -41.50%)     3311.33 ( -41.07%)
Min       1024     17325.58 (   0.00%)    11053.24 ( -36.20%)    11098.91 ( -35.94%)    10892.59 ( -37.13%)
Min       2048     27564.59 (   0.00%)    18311.31 ( -33.57%)    18649.89 ( -32.34%)    18327.69 ( -33.51%)
Min       3312     33677.30 (   0.00%)    25254.43 ( -25.01%)    24897.65 ( -26.07%)    25464.71 ( -24.39%)
Min       4096     35624.64 (   0.00%)    28186.09 ( -20.88%)    27317.58 ( -23.32%)    27046.46 ( -24.08%)
Min       8192     42950.87 (   0.00%)    33407.18 ( -22.22%)    34133.19 ( -20.53%)    33429.82 ( -22.17%)
Min       16384    46798.74 (   0.00%)    40020.99 ( -14.48%)    40761.81 ( -12.90%)    40370.88 ( -13.74%)
Hmean     64        1818.68 (   0.00%)      959.16 ( -47.26%)      962.40 ( -47.08%)      954.96 ( -47.49%)
Hmean     128       3405.06 (   0.00%)     1860.21 ( -45.37%)     1844.12 ( -45.84%)     1849.44 ( -45.69%)
Hmean     256       5777.53 (   0.00%)     3371.67 ( -41.64%)     3341.43 ( -42.17%)     3360.35 ( -41.84%)
Hmean     1024     17679.46 (   0.00%)    11326.96 ( -35.93%)    11192.24 ( -36.69%)    11219.22 ( -36.54%)
Hmean     2048     27764.04 (   0.00%)    18864.94 ( -32.05%)    18833.51 ( -32.17%)    18740.31 ( -32.50%)
Hmean     3312     35253.65 (   0.00%)    25444.33 ( -27.82%)    25700.57 ( -27.10%)    25610.63 ( -27.35%)
Hmean     4096     36479.20 (   0.00%)    28636.63 ( -21.50%)    28073.90 ( -23.04%)    27856.51 ( -23.64%)
Hmean     8192     43386.27 (   0.00%)    34771.52 ( -19.86%)    35213.44 ( -18.84%)    34603.90 ( -20.24%)
Hmean     16384    47487.74 (   0.00%)    41329.50 ( -12.97%)    41096.73 ( -13.46%)    40787.33 ( -14.11%)
Stddev    64          12.42 (   0.00%)        6.35 (  48.87%)        5.77 (  53.54%)       12.21 (   1.73%)
Stddev    128         45.84 (   0.00%)        9.25 (  79.82%)       13.49 (  70.57%)       23.86 (  47.95%)
Stddev    256         90.59 (   0.00%)       30.55 (  66.28%)       37.07 (  59.08%)       28.66 (  68.36%)
Stddev    1024       322.33 (   0.00%)      164.75 (  48.89%)      119.05 (  63.07%)      265.42 (  17.65%)
Stddev    2048       153.04 (   0.00%)      424.98 (-177.70%)      176.40 ( -15.26%)      242.90 ( -58.72%)
Stddev    3312      1024.93 (   0.00%)      182.58 (  82.19%)      585.07 (  42.92%)      108.93 (  89.37%)
Stddev    4096       696.34 (   0.00%)      433.20 (  37.79%)      626.42 (  10.04%)      712.05 (  -2.26%)
Stddev    8192       478.31 (   0.00%)      808.23 ( -68.98%)      794.39 ( -66.08%)      698.27 ( -45.99%)
Stddev    16384      720.05 (   0.00%)      816.70 ( -13.42%)      412.26 (  42.75%)      325.43 (  54.81%)
CoeffVar  64           0.68 (   0.00%)        0.66 (   3.05%)        0.60 (  12.20%)        1.28 ( -87.13%)
CoeffVar  128          1.35 (   0.00%)        0.50 (  63.06%)        0.73 (  45.66%)        1.29 (   4.17%)
CoeffVar  256          1.57 (   0.00%)        0.91 (  42.21%)        1.11 (  29.24%)        0.85 (  45.59%)
CoeffVar  1024         1.82 (   0.00%)        1.45 (  20.22%)        1.06 (  41.65%)        2.36 ( -29.74%)
CoeffVar  2048         0.55 (   0.00%)        2.25 (-308.53%)        0.94 ( -69.91%)        1.30 (-135.12%)
CoeffVar  3312         2.91 (   0.00%)        0.72 (  75.30%)        2.28 (  21.68%)        0.43 (  85.36%)
CoeffVar  4096         1.91 (   0.00%)        1.51 (  20.74%)        2.23 ( -16.88%)        2.55 ( -33.88%)
CoeffVar  8192         1.10 (   0.00%)        2.32 (-110.77%)        2.25 (-104.56%)        2.02 ( -82.99%)
CoeffVar  16384        1.52 (   0.00%)        1.98 ( -30.31%)        1.00 (  33.83%)        0.80 (  47.37%)
Max       64        1832.51 (   0.00%)      966.09 ( -47.28%)      970.35 ( -47.05%)      967.15 ( -47.22%)
Max       128       3476.62 (   0.00%)     1873.20 ( -46.12%)     1865.28 ( -46.35%)     1869.10 ( -46.24%)
Max       256       5839.83 (   0.00%)     3402.61 ( -41.73%)     3379.67 ( -42.13%)     3383.69 ( -42.06%)
Max       1024     18031.63 (   0.00%)    11482.14 ( -36.32%)    11396.22 ( -36.80%)    11463.71 ( -36.42%)
Max       2048     27912.65 (   0.00%)    19343.06 ( -30.70%)    19095.51 ( -31.59%)    18969.02 ( -32.04%)
Max       3312     36142.68 (   0.00%)    25749.54 ( -28.76%)    26503.65 ( -26.67%)    25767.14 ( -28.71%)
Max       4096     37481.84 (   0.00%)    29189.76 ( -22.12%)    28875.41 ( -22.96%)    28973.52 ( -22.70%)
Max       8192     44101.03 (   0.00%)    35471.04 ( -19.57%)    35890.95 ( -18.62%)    35178.96 ( -20.23%)
Max       16384    48321.50 (   0.00%)    42086.21 ( -12.90%)    41793.29 ( -13.51%)    41152.43 ( -14.84%)

Peter, if you want to run this test yourself you can do:

 1. git clone https://github.com/gorman/mmmtests.git
 2. cd mmtests
 3. ./run-mmtests.sh --config=configs/config-global-dhp__network-netperf-unbound `uname -r`