Subject: next/master baseline: 710 runs, 37 regressions (next-20230116)

From: "kernelci.org bot" <[email-masked]>
Date: Mon, 16 Jan 2023 00:46:08 -0800 (PST)

next/master baseline: 710 runs, 37 regressions (next-20230116)

Regressions Summary
-------------------

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15          | arm   | lab-linaro-lkft | gcc-10   | multi_v7_defconfig+debug     | 1          
fsl-ls1088a-rdb            | arm64 | lab-nxp         | clang-16 | defconfig+CON..._64K_PAGES=y | 1          
fsl-ls1088a-rdb            | arm64 | lab-nxp         | clang-16 | defconfig                    | 1          
imx6dl-riotboard           | arm   | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
imx6dl-udoo                | arm   | lab-broonie     | gcc-10   | imx_v6_v7_defconfig          | 1          
imx6q-udoo                 | arm   | lab-broonie     | gcc-10   | imx_v6_v7_defconfig          | 1          
imx6qp-wandboard-revd1     | arm   | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
imx8mp-evk                 | arm64 | lab-nxp         | clang-16 | defconfig+CON..._64K_PAGES=y | 1          
imx8mp-evk                 | arm64 | lab-nxp         | clang-16 | defconfig                    | 1          
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+CON...OMIZE_BASE=y | 1          
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+ima                | 1          
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+crypto             | 1          
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig                    | 1          
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+videodec           | 1          
juno-uboot                 | arm64 | lab-broonie     | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          
kontron-pitx-imx8m         | arm64 | lab-kontron     | clang-16 | defconfig+CON..._64K_PAGES=y | 1          
kontron-pitx-imx8m         | arm64 | lab-kontron     | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          
mt8192-asurada-spherion-r0 | arm64 | lab-collabora   | clang-13 | cros://chrome...4-chromebook | 1          
mt8192-asurada-spherion-r0 | arm64 | lab-collabora   | clang-13 | cros://chrome...4-chromebook | 1          
qemu_arm-virt-gicv2        | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2        | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi   | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi   | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3        | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3        | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3        | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi   | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi   | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie     | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie     | gcc-10   | defconfig+debug              | 1          
rk3399-roc-pc              | arm64 | lab-clabbe      | gcc-10   | defconfig+debug              | 1          
rk3399-rock-pi-4b          | arm64 | lab-collabora   | clang-16 | defconfig                    | 1          
rk3399-rock-pi-4b          | arm64 | lab-collabora   | gcc-10   | defconfig+videodec           | 1          
zynqmp-zcu102              | arm64 | lab-cip         | gcc-10   | defconfig+debug              | 1          

  Details:  https://kernelci.org/test/job/next/branch/master/kernel/next-20230116/plan/baseline/

  Test:     baseline
  Tree:     next
  Branch:   master
  Describe: next-20230116
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
  SHA:      c12e2e5b76b2e739ccdf196bee960412b45d5f85 

Test Regressions
---------------- 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15          | arm   | lab-linaro-lkft | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4f7dff209f252561d39e3

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4f7dff209f252561d39e4
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
fsl-ls1088a-rdb            | arm64 | lab-nxp         | clang-16 | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e4346c38b519871d39c2

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    clang-16 (Debian clang version 16.0.0 (++20230114071858+79858d1908b5-1~exp1~20230114072010.530))
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-nxp/baseline-fsl-ls1088a-rdb.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-nxp/baseline-fsl-ls1088a-rdb.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63c4e4346c38b519871d39c5
        failing since 10 days (last pass: next-20221125, first fail: next-20230106)

    2023-01-16T05:43:53.879416  [   12.536976] <LAVA_SIGNAL_ENDRUN 0_dmesg 1137491_1.5.2.4.1>
    2023-01-16T05:43:53.985248  / # #
    2023-01-16T05:43:54.087146  export SHELL=/bin/sh
    2023-01-16T05:43:54.087778  #
    2023-01-16T05:43:54.189182  / # export SHELL=/bin/sh. /lava-1137491/environment
    2023-01-16T05:43:54.189683  
    2023-01-16T05:43:54.291074  / # . /lava-1137491/environment/lava-1137491/bin/lava-test-runner /lava-1137491/1
    2023-01-16T05:43:54.291866  
    2023-01-16T05:43:54.293786  / # /lava-1137491/bin/lava-test-runner /lava-1137491/1
    2023-01-16T05:43:54.316935  + export 'TESTRUN_ID=1_bootrr' 
    ... (11 line(s) more)  

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
fsl-ls1088a-rdb            | arm64 | lab-nxp         | clang-16 | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e4ed5e299d196b1d3a50

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: defconfig
  Compiler:    clang-16 (Debian clang version 16.0.0 (++20230114071858+79858d1908b5-1~exp1~20230114072010.530))
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/clang-16/lab-nxp/baseline-fsl-ls1088a-rdb.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/clang-16/lab-nxp/baseline-fsl-ls1088a-rdb.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63c4e4ed5e299d196b1d3a53
        failing since 10 days (last pass: next-20221124, first fail: next-20230106)

    2023-01-16T05:47:11.679205  + [   12.807540] <LAVA_SIGNAL_ENDRUN 0_dmesg 1137492_1.5.2.4.1>
    2023-01-16T05:47:11.679515  set +x
    2023-01-16T05:47:11.785165  / # #
    2023-01-16T05:47:11.887124  export SHELL=/bin/sh
    2023-01-16T05:47:11.887588  #
    2023-01-16T05:47:11.988942  / # export SHELL=/bin/sh. /lava-1137492/environment
    2023-01-16T05:47:11.989389  
    2023-01-16T05:47:12.090771  / # . /lava-1137492/environment/lava-1137492/bin/lava-test-runner /lava-1137492/1
    2023-01-16T05:47:12.091535  
    2023-01-16T05:47:12.093584  / # /lava-1137492/bin/lava-test-runner /lava-1137492/1 
    ... (12 line(s) more)  

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx6dl-riotboard           | arm   | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e1f8e87efb65f61d39cd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e1f8e87efb65f61d39ce
        failing since 159 days (last pass: next-20220808, first fail: next-20220810) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx6dl-udoo                | arm   | lab-broonie     | gcc-10   | imx_v6_v7_defconfig          | 1          

  Details:     https://kernelci.org/test/plan/id/63c4debd2e308e754c1d39ca

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: imx_v6_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/imx_v6_v7_defconfig/gcc-10/lab-broonie/baseline-imx6dl-udoo.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/imx_v6_v7_defconfig/gcc-10/lab-broonie/baseline-imx6dl-udoo.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4debd2e308e754c1d39cb
        failing since 219 days (last pass: next-20220607, first fail: next-20220610) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx6q-udoo                 | arm   | lab-broonie     | gcc-10   | imx_v6_v7_defconfig          | 1          

  Details:     https://kernelci.org/test/plan/id/63c4df35157b2a75551d39f1

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: imx_v6_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/imx_v6_v7_defconfig/gcc-10/lab-broonie/baseline-imx6q-udoo.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/imx_v6_v7_defconfig/gcc-10/lab-broonie/baseline-imx6q-udoo.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4df35157b2a75551d39f2
        failing since 130 days (last pass: next-20220831, first fail: next-20220907) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx6qp-wandboard-revd1     | arm   | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4ded663006924811d39d5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4ded663006924811d39d6
        failing since 146 days (last pass: next-20220810, first fail: next-20220822) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | clang-16 | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e7c93d7dad48301d3a4d

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    clang-16 (Debian clang version 16.0.0 (++20230114071858+79858d1908b5-1~exp1~20230114072010.530))
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e7c93d7dad48301d3a4e
        failing since 10 days (last pass: next-20221125, first fail: next-20230106) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | clang-16 | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e962a25cebd5771d39c2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    clang-16 (Debian clang version 16.0.0 (++20230114071858+79858d1908b5-1~exp1~20230114072010.530))
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/clang-16/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/clang-16/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e962a25cebd5771d39c3
        failing since 11 days (last pass: next-20221124, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+CON...OMIZE_BASE=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4dd1561f28209361d39cf

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_RANDOMIZE_BASE=y
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_RANDOMIZE_BASE=y/gcc-10/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_RANDOMIZE_BASE=y/gcc-10/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4dd1561f28209361d39d0
        failing since 10 days (last pass: next-20221226, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+ima                | 1          

  Details:     https://kernelci.org/test/plan/id/63c4de9db51f8b669d1d39d1

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+ima
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+ima/gcc-10/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+ima/gcc-10/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4de9db51f8b669d1d39d2
        failing since 10 days (last pass: next-20221226, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+crypto             | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e088edf16778c71d39d2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+crypto
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+crypto/gcc-10/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+crypto/gcc-10/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e088edf16778c71d39d3
        failing since 11 days (last pass: next-20221226, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e5fcf2185fc1921d39c2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e5fcf2185fc1921d39c3
        failing since 11 days (last pass: next-20221226, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/63c4eae5df66288cce1d39c2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/gcc-10/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/gcc-10/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4eae5df66288cce1d39c3
        failing since 11 days (last pass: next-20221125, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
imx8mp-evk                 | arm64 | lab-nxp         | gcc-10   | defconfig+videodec           | 1          

  Details:     https://kernelci.org/test/plan/id/63c4eb896924b834891d39d2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+videodec
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+videodec/gcc-10/lab-nxp/baseline-imx8mp-evk.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+videodec/gcc-10/lab-nxp/baseline-imx8mp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4eb896924b834891d39d3
        failing since 11 days (last pass: next-20221226, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
juno-uboot                 | arm64 | lab-broonie     | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e419dd9a44497f1d3a3a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-broonie/baseline-juno-uboot.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-broonie/baseline-juno-uboot.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e419dd9a44497f1d3a3b
        new failure (last pass: next-20230113) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
kontron-pitx-imx8m         | arm64 | lab-kontron     | clang-16 | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e8d7873b7902e81d3a42

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    clang-16 (Debian clang version 16.0.0 (++20230114071858+79858d1908b5-1~exp1~20230114072010.530))
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-kontron/baseline-kontron-pitx-imx8m.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-kontron/baseline-kontron-pitx-imx8m.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e8d7873b7902e81d3a43
        failing since 75 days (last pass: next-20221028, first fail: next-20221101) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
kontron-pitx-imx8m         | arm64 | lab-kontron     | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e7963d7dad48301d39d4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-kontron/baseline-kontron-pitx-imx8m.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-kontron/baseline-kontron-pitx-imx8m.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e7963d7dad48301d39d5
        failing since 67 days (last pass: next-20221107, first fail: next-20221109) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
mt8192-asurada-spherion-r0 | arm64 | lab-collabora   | clang-13 | cros://chrome...4-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63c4de3647789ca16e1d39fc

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: cros://chromeos-5.10/arm64/chromiumos-arm64.flavour.config+arm64-chromebook
  Compiler:    clang-13 (Debian clang version 13.0.1-6~deb11u1)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/cros---chromeos-5.10-arm64-chromiumos-arm64.flavour.config+arm64-chromebook/clang-13/lab-collabora/baseline-mt8192-asurada-spherion-r0.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/cros---chromeos-5.10-arm64-chromiumos-arm64.flavour.config+arm64-chromebook/clang-13/lab-collabora/baseline-mt8192-asurada-spherion-r0.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4de3647789ca16e1d39fd
        failing since 5 days (last pass: next-20230109, first fail: next-20230110) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
mt8192-asurada-spherion-r0 | arm64 | lab-collabora   | clang-13 | cros://chrome...4-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/63c4dd1e8c0aab57141d39de

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: cros://chromeos-5.10/arm64/chromiumos-mediatek.flavour.config+arm64-chromebook
  Compiler:    clang-13 (Debian clang version 13.0.1-6~deb11u1)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/cros---chromeos-5.10-arm64-chromiumos-mediatek.flavour.config+arm64-chromebook/clang-13/lab-collabora/baseline-mt8192-asurada-spherion-r0.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/cros---chromeos-5.10-arm64-chromiumos-mediatek.flavour.config+arm64-chromebook/clang-13/lab-collabora/baseline-mt8192-asurada-spherion-r0.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4dd1e8c0aab57141d39df
        failing since 2 days (last pass: next-20230112, first fail: next-20230113) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2        | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4deafcaed93b1681d39c5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4deafcaed93b1681d39c6
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2        | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e1b0e008d5ac5e1d3a28

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e1b0e008d5ac5e1d3a29
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi   | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4dec0caed93b1681d39e2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4dec0caed93b1681d39e3
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi   | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e1b165b557729c1d39c4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e1b165b557729c1d39c5
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3        | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4deacb51f8b669d1d3a09

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4deacb51f8b669d1d3a0a
        failing since 122 days (last pass: next-20220907, first fail: next-20220915) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3        | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e0d27d3c9e49f11d3a06

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e0d27d3c9e49f11d3a07
        failing since 122 days (last pass: next-20220907, first fail: next-20220915) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3        | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e06eedf16778c71d39ca

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e06eedf16778c71d39cb
        failing since 122 days (last pass: next-20220907, first fail: next-20220915) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi   | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4deaeb51f8b669d1d3a12

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4deaeb51f8b669d1d3a13
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi   | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e1af86b27e06021d39d5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e1af86b27e06021d39d6
        failing since 124 days (last pass: next-20220907, first fail: next-20220913) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e0a56f090cadd61d39cb

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e0a56f090cadd61d39cc
        failing since 10 days (last pass: next-20221207, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv2-uefi | arm64 | lab-broonie     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e37b7019baee141d39c4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e37b7019baee141d39c5
        failing since 10 days (last pass: next-20221207, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e0a26f090cadd61d39c6

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e0a26f090cadd61d39c7
        failing since 10 days (last pass: next-20221207, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv3-uefi | arm64 | lab-broonie     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e367602a22d2d21d39de

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e367602a22d2d21d39df
        failing since 10 days (last pass: next-20221207, first fail: next-20230105) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
rk3399-roc-pc              | arm64 | lab-clabbe      | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e0eb4ddb0894e31d39da

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-clabbe/baseline-rk3399-roc-pc.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-clabbe/baseline-rk3399-roc-pc.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63c4e0eb4ddb0894e31d39dd
        failing since 10 days (last pass: next-20221130, first fail: next-20230105)

    2023-01-16T05:29:53.628379  + set +x
    2023-01-16T05:29:53.629613  <8>[   66.429366][  T194] <LAVA_SIGNAL_ENDRUN 0_dmesg 379842_1.5.2.4.1>
    2023-01-16T05:29:53.766769  #
    2023-01-16T05:29:53.869292  / # #export SHELL=/bin/sh
    2023-01-16T05:29:53.869897  
    2023-01-16T05:29:53.971488  / # export SHELL=/bin/sh. /lava-379842/environment
    2023-01-16T05:29:53.972070  
    2023-01-16T05:29:54.073633  / # . /lava-379842/environment/lava-379842/bin/lava-test-runner /lava-379842/1
    2023-01-16T05:29:54.074570  
    2023-01-16T05:29:54.083339  / # /lava-379842/bin/lava-test-runner /lava-379842/1 
    ... (43 line(s) more)  

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
rk3399-rock-pi-4b          | arm64 | lab-collabora   | clang-16 | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e32a4ac159aa041d39c2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    clang-16 (Debian clang version 16.0.0 (++20230114071858+79858d1908b5-1~exp1~20230114072010.530))
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/clang-16/lab-collabora/baseline-rk3399-rock-pi-4b.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig/clang-16/lab-collabora/baseline-rk3399-rock-pi-4b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e32a4ac159aa041d39c3
        new failure (last pass: next-20230112) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
rk3399-rock-pi-4b          | arm64 | lab-collabora   | gcc-10   | defconfig+videodec           | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e4a6cf55d7f6441d3a41

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+videodec
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+videodec/gcc-10/lab-collabora/baseline-rk3399-rock-pi-4b.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+videodec/gcc-10/lab-collabora/baseline-rk3399-rock-pi-4b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/63c4e4a6cf55d7f6441d3a42
        failing since 2 days (last pass: next-20230112, first fail: next-20230113) 

platform                   | arch  | lab             | compiler | defconfig                    | regressions
---------------------------+-------+-----------------+----------+------------------------------+------------
zynqmp-zcu102              | arm64 | lab-cip         | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/63c4e137e3d2fdfd211d39cb

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-cip/baseline-zynqmp-zcu102.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20230116/arm64/defconfig+debug/gcc-10/lab-cip/baseline-zynqmp-zcu102.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230109.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/63c4e137e3d2fdfd211d39ce
        failing since 10 days (last pass: next-20221207, first fail: next-20230105)

    2023-01-16T05:31:08.822309  + set +x
    2023-01-16T05:31:08.823468  <8>[   44.237225][  T172] <LAVA_SIGNAL_ENDRUN 0_dmesg 826074_1.5.2.4.1>
    2023-01-16T05:31:08.953219  #
    2023-01-16T05:31:09.055478  / # #export SHELL=/bin/sh
    2023-01-16T05:31:09.055932  
    2023-01-16T05:31:09.157347  / # export SHELL=/bin/sh. /lava-826074/environment
    2023-01-16T05:31:09.157802  
    2023-01-16T05:31:09.258998  / # . /lava-826074/environment/lava-826074/bin/lava-test-runner /lava-826074/1
    2023-01-16T05:31:09.259933  
    2023-01-16T05:31:09.264100  / # /lava-826074/bin/lava-test-runner /lava-826074/1 
    ... (18 line(s) more)