Subject: next/pending-fixes baseline: 420 runs, 25 regressions (v5.19-rc2-392-gc16436fe44ee8)

From: "kernelci.org bot" <[email-masked]>
Date: Wed, 15 Jun 2022 22:17:14 -0700 (PDT)

next/pending-fixes baseline: 420 runs, 25 regressions (v5.19-rc2-392-gc16436fe44ee8)

Regressions Summary
-------------------

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
hp-x360-14-G1-sona           | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...6-chromebook | 1          
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | imx_v6_v7_defconfig          | 1          
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig           | 1          
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+ima       | 1          
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+ima       | 1          
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | tegra_defconfig              | 1          
kontron-kbox-a-230-ls        | arm64  | lab-kontron     | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv2-uefi   | arm64  | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv2-uefi   | arm64  | lab-broonie     | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv3-uefi   | arm64  | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv3-uefi   | arm64  | lab-broonie     | gcc-10   | defconfig+debug              | 1          
sun50i-a64-bananapi-m64      | arm64  | lab-clabbe      | gcc-10   | defconfig                    | 1          
sun50i-a64-bananapi-m64      | arm64  | lab-clabbe      | gcc-10   | defconfig+crypto             | 1          
sun50i-a64-bananapi-m64      | arm64  | lab-clabbe      | gcc-10   | defconfig+debug              | 1          
sun50i-a64-pine64-plus       | arm64  | lab-broonie     | gcc-10   | defconfig+crypto             | 1          
sun50i-h5-nanopi-neo-plus2   | arm64  | lab-clabbe      | gcc-10   | defconfig                    | 1          
sun50i-h6-orangepi-one-plus  | arm64  | lab-clabbe      | gcc-10   | defconfig                    | 1          
sun50i-h6-pine-h64           | arm64  | lab-collabora   | gcc-10   | defconfig+crypto             | 1          
sun8i-a83t-bananapi-m3       | arm    | lab-clabbe      | gcc-10   | multi_v7_defconfig+ima       | 1          
sun8i-h2-plus...ch-all-h3-cc | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
sun8i-h2-plus-orangepi-zero  | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
sun8i-h3-bananapi-m2-plus    | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
sun8i-r40-bananapi-m2-ultra  | arm    | lab-clabbe      | gcc-10   | multi_v7_defconfig           | 1          

  Details:  https://kernelci.org/test/job/next/branch/pending-fixes/kernel/v5.19-rc2-392-gc16436fe44ee8/plan/baseline/

  Test:     baseline
  Tree:     next
  Branch:   pending-fixes
  Describe: v5.19-rc2-392-gc16436fe44ee8
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
  SHA:      c16436fe44ee8984a006aa3779f7f399fc8cb1c3 

Test Regressions
---------------- 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
hp-x360-14-G1-sona           | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...6-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/62aa83d77664166657a39c25

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-chromebook
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/x86_64/x86_64_defconfig+x86-chromebook/gcc-10/lab-collabora/baseline-hp-x360-14-G1-sona.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/x86_64/x86_64_defconfig+x86-chromebook/gcc-10/lab-collabora/baseline-hp-x360-14-G1-sona.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa83d77664166657a39c26
        new failure (last pass: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | imx_v6_v7_defconfig          | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8809bed1b741c0a39be5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: imx_v6_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/imx_v6_v7_defconfig/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/imx_v6_v7_defconfig/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8809bed1b741c0a39be6
        failing since 16 days (last pass: v5.16-rc4-241-g3e464b455df7, first fail: v5.18-11575-gceccc06b624ed) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/62aa89fd3321881e12a39bdd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa89fd3321881e12a39bde
        failing since 16 days (last pass: v5.16-rc4-241-g3e464b455df7, first fail: v5.18-11575-gceccc06b624ed) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+ima       | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8bf1eec921ec59a39be4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+ima
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+ima/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+ima/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8bf1eec921ec59a39be5
        failing since 16 days (last pass: v5.16-rc4-356-ga119cd39dcbf, first fail: v5.18-11575-gceccc06b624ed) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6ul-pico-hobbit           | arm    | lab-pengutronix | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8de59ed7038244a39bfd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_SMP=n
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-pengutronix/baseline-imx6ul-pico-hobbit.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8de59ed7038244a39bfe
        failing since 16 days (last pass: v5.16-rc4-356-ga119cd39dcbf, first fail: v5.18-11575-gceccc06b624ed) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/62aa92256cd88f0c94a39bd2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa92256cd88f0c94a39bd3
        failing since 26 days (last pass: v5.18-rc7-165-g2424086909d29, first fail: v5.18-rc7-251-g195d7f7a62c6) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+ima       | 1          

  Details:     https://kernelci.org/test/plan/id/62aa979ca5073c5c6ca39bd3

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+ima
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+ima/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+ima/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa979ca5073c5c6ca39bd4
        failing since 27 days (last pass: v5.18-rc7-165-g2424086909d29, first fail: v5.18-rc7-251-g195d7f7a62c6) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | tegra_defconfig              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa94d7515c5c50d6a39bf8

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: tegra_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/tegra_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/tegra_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa94d7515c5c50d6a39bf9
        failing since 10 days (last pass: v5.18-12139-gb5db4eaa8649, first fail: v5.18-14068-g9e633753407a5) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
kontron-kbox-a-230-ls        | arm64  | lab-kontron     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa891dc621c25f9aa39be3

  Results:     94 PASS, 1 FAIL, 1 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-kontron/baseline-kontron-kbox-a-230-ls.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-kontron/baseline-kontron-kbox-a-230-ls.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.bootrr.mscc_felix-probed: https://kernelci.org/test/case/id/62aa891dc621c25f9aa39bec
        failing since 16 days (last pass: v5.18-5613-ge079cf8f6817, first fail: v5.18-11575-gceccc06b624ed)

    2022-06-16T01:36:00.643966  /lava-128214/1/../bin/lava-test-case
    2022-06-16T01:36:00.745109  <8>[   41.561207][  T317] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=mscc_felix-probed RESULT=fail>   

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv2-uefi   | arm64  | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8bf4091efc17f1a39bee

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8bf4091efc17f1a39bef
        failing since 36 days (last pass: v5.18-rc5-178-g63184bc90d435, first fail: v5.18-rc6-217-gd854612e01df) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv2-uefi   | arm64  | lab-broonie     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8d52a5b4028d5da39be2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8d52a5b4028d5da39be3
        failing since 36 days (last pass: v5.18-rc5-178-g63184bc90d435, first fail: v5.18-rc6-217-gd854612e01df) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv3-uefi   | arm64  | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8ce544973b7c4ba39bcd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8ce544973b7c4ba39bce
        failing since 36 days (last pass: v5.18-rc5-178-g63184bc90d435, first fail: v5.18-rc6-217-gd854612e01df) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv3-uefi   | arm64  | lab-broonie     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8d6785b666d6afa39be8

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8d6785b666d6afa39be9
        failing since 36 days (last pass: v5.18-rc5-178-g63184bc90d435, first fail: v5.18-rc6-217-gd854612e01df) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-a64-bananapi-m64      | arm64  | lab-clabbe      | gcc-10   | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/62aa86be4931282db2a39bda

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig/gcc-10/lab-clabbe/baseline-sun50i-a64-bananapi-m64.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig/gcc-10/lab-clabbe/baseline-sun50i-a64-bananapi-m64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa86be4931282db2a39bdb
        failing since 9 days (last pass: v5.18-14068-g9e633753407a5, first fail: v5.19-rc1-102-g5a749a6e62f23) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-a64-bananapi-m64      | arm64  | lab-clabbe      | gcc-10   | defconfig+crypto             | 1          

  Details:     https://kernelci.org/test/plan/id/62aa893fc621c25f9aa39cee

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+crypto
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+crypto/gcc-10/lab-clabbe/baseline-sun50i-a64-bananapi-m64.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+crypto/gcc-10/lab-clabbe/baseline-sun50i-a64-bananapi-m64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa893fc621c25f9aa39cef
        failing since 8 days (last pass: v5.19-rc1-102-g5a749a6e62f23, first fail: v5.19-rc1-144-g9b6181f62e678) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-a64-bananapi-m64      | arm64  | lab-clabbe      | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8aa61a04c23c74a39bdc

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-clabbe/baseline-sun50i-a64-bananapi-m64.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+debug/gcc-10/lab-clabbe/baseline-sun50i-a64-bananapi-m64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8aa61a04c23c74a39bdd
        new failure (last pass: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-a64-pine64-plus       | arm64  | lab-broonie     | gcc-10   | defconfig+crypto             | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8942330f1230a9a39bfd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+crypto
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+crypto/gcc-10/lab-broonie/baseline-sun50i-a64-pine64-plus.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+crypto/gcc-10/lab-broonie/baseline-sun50i-a64-pine64-plus.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8942330f1230a9a39bfe
        failing since 21 days (last pass: v5.18-1222-g391cf5a1244f, first fail: v5.18-5613-ge079cf8f6817) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-h5-nanopi-neo-plus2   | arm64  | lab-clabbe      | gcc-10   | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/62aa86c54931282db2a39bf4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig/gcc-10/lab-clabbe/baseline-sun50i-h5-nanopi-neo-plus2.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig/gcc-10/lab-clabbe/baseline-sun50i-h5-nanopi-neo-plus2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa86c54931282db2a39bf5
        failing since 5 days (last pass: v5.19-rc1-102-g5a749a6e62f23, first fail: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-h6-orangepi-one-plus  | arm64  | lab-clabbe      | gcc-10   | defconfig                    | 1          

  Details:     https://kernelci.org/test/plan/id/62aa87d8b21a8bc56ca39c43

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig/gcc-10/lab-clabbe/baseline-sun50i-h6-orangepi-one-plus.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig/gcc-10/lab-clabbe/baseline-sun50i-h6-orangepi-one-plus.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa87d8b21a8bc56ca39c44
        new failure (last pass: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun50i-h6-pine-h64           | arm64  | lab-collabora   | gcc-10   | defconfig+crypto             | 1          

  Details:     https://kernelci.org/test/plan/id/62aa88a38d82019f4ba39bf9

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+crypto
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+crypto/gcc-10/lab-collabora/baseline-sun50i-h6-pine-h64.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm64/defconfig+crypto/gcc-10/lab-collabora/baseline-sun50i-h6-pine-h64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa88a38d82019f4ba39bfa
        failing since 8 days (last pass: v5.19-rc1-102-g5a749a6e62f23, first fail: v5.19-rc1-144-g9b6181f62e678) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun8i-a83t-bananapi-m3       | arm    | lab-clabbe      | gcc-10   | multi_v7_defconfig+ima       | 1          

  Details:     https://kernelci.org/test/plan/id/62aa88994d1db30ee9a39c14

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+ima
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+ima/gcc-10/lab-clabbe/baseline-sun8i-a83t-bananapi-m3.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig+ima/gcc-10/lab-clabbe/baseline-sun8i-a83t-bananapi-m3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa88994d1db30ee9a39c15
        new failure (last pass: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun8i-h2-plus...ch-all-h3-cc | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/62aa871aa9068df41ea39c42

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun8i-h2-plus-libretech-all-h3-cc.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun8i-h2-plus-libretech-all-h3-cc.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa871aa9068df41ea39c43
        new failure (last pass: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun8i-h2-plus-orangepi-zero  | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/62aa8725c9de0742dda39bd0

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun8i-h2-plus-orangepi-zero.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun8i-h2-plus-orangepi-zero.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa8725c9de0742dda39bd1
        failing since 5 days (last pass: v5.19-rc1-102-g5a749a6e62f23, first fail: v5.19-rc1-331-g13bdc020d5006) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun8i-h3-bananapi-m2-plus    | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/62aa86e1f79b6ea78fa39bd9

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun8i-h3-bananapi-m2-plus.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun8i-h3-bananapi-m2-plus.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa86e1f79b6ea78fa39bda
        failing since 16 days (last pass: v5.18-1222-g391cf5a1244f, first fail: v5.18-11575-gceccc06b624ed) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sun8i-r40-bananapi-m2-ultra  | arm    | lab-clabbe      | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/62aa906f2e9e67ff9ea39c44

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-clabbe/baseline-sun8i-r40-bananapi-m2-ultra.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v5.19-rc2-392-gc16436fe44ee8/arm/multi_v7_defconfig/gcc-10/lab-clabbe/baseline-sun8i-r40-bananapi-m2-ultra.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220610.1/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/62aa906f2e9e67ff9ea39c45
        failing since 9 days (last pass: v5.18-12248-g882f1d1979b5, first fail: v5.19-rc1-102-g5a749a6e62f23)