Subject: next/master baseline: 404 runs, 33 regressions (next-20221011)

From: "kernelci.org bot" <[email-masked]>
Date: Tue, 11 Oct 2022 01:25:05 -0700 (PDT)

next/master baseline: 404 runs, 33 regressions (next-20221011)

Regressions Summary
-------------------

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15            | arm   | lab-linaro-lkft | gcc-10   | multi_v7_defconfig+debug     | 1          
jetson-tk1                   | arm   | lab-baylibre    | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
jetson-tk1                   | arm   | lab-baylibre    | gcc-10   | tegra_defconfig              | 2          
kontron-kbox-a-230-ls        | arm64 | lab-kontron     | gcc-10   | defconfig+debug              | 1          
kontron-pitx-imx8m           | arm64 | lab-kontron     | clang-16 | defconfig+CON..._64K_PAGES=y | 1          
kontron-sl28-var3-ads2       | arm64 | lab-kontron     | gcc-10   | defconfig+debug              | 1          
meson-g12a-sei510            | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
meson-g12a-u200              | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
meson-g12b-odroid-n2         | arm64 | lab-baylibre    | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          
meson-g12b-odroid-n2         | arm64 | lab-baylibre    | gcc-10   | defconfig+ima                | 1          
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre    | gcc-10   | defconfig+videodec           | 1          
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre    | gcc-10   | defconfig+crypto             | 1          
mt8183-kukui-...uniper-sku16 | arm64 | lab-collabora   | clang-13 | cros://chrome...4-chromebook | 1          
ox820-clouden...lug-series-3 | arm   | lab-baylibre    | gcc-10   | oxnas_v6_defconfig           | 1          
qemu_arm-virt-gicv2          | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2          | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2          | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi     | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi     | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi     | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi     | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi     | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi     | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm64-virt-gicv2        | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv2-uefi   | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv3        | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          
qemu_arm64-virt-gicv3-uefi   | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          
rk3328-rock64                | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:  https://kernelci.org/test/job/next/branch/master/kernel/next-20221011/plan/baseline/

  Test:     baseline
  Tree:     next
  Branch:   master
  Describe: next-20221011
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
  SHA:      b9f85101cad3397ef1e509909602a90e257ab9d8 

Test Regressions
---------------- 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15            | arm   | lab-linaro-lkft | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f66841717cf22dcab626

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f66841717cf22dcab627
        failing since 26 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
jetson-tk1                   | arm   | lab-baylibre    | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/6344f8b4edeb2312a9cab5ed

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f8b4edeb2312a9cab5ee
        failing since 182 days (last pass: next-20220401, first fail: next-20220411) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
jetson-tk1                   | arm   | lab-baylibre    | gcc-10   | tegra_defconfig              | 2          

  Details:     https://kernelci.org/test/plan/id/6344f446dcb429c27ccab607

  Results:     3 PASS, 2 FAIL, 2 SKIP
  Full config: tegra_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/tegra_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/tegra_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.dmesg.emerg: https://kernelci.org/test/case/id/6344f446dcb429c27ccab60e
        failing since 12 days (last pass: next-20220921, first fail: next-20220928)
        45 lines

    2022-10-11T04:41:33.746953  kern  :alert : Register r1 information: 2-page vmalloc region starting at 0xf0ccc000 allocated at kernel_clone+0x8c/0x398
    2022-10-11T04:41:33.786619  kern  :alert : Register r2 information: NULL pointer
    2022-10-11T04:41:33.787102  kern  :alert : Register r3 information: NULL pointer
    2022-10-11T04:41:33.787302  kern  :alert : Register r4 information: slab kmalloc-1k start c16ea400 pointer offset 16 size 1024
    2022-10-11T04:41:33.787454  kern  :alert : Register r5 information: 6-page vmalloc region starting at 0xbf000000 allocated at load_module+0x99c/0x1e28
    2022-10-11T04:41:33.787597  kern  :alert : Register r6 information: slab kmalloc-1k start c2a3b400 pointer offset 0 size 1024
    2022-10-11T04:41:33.829586  kern  :alert : Register r7 information: slab kmalloc-512 start c2cfa000 pointer offset 64 size 512
    2022-10-11T04:41:33.830061  kern  :alert : Register r8 information: slab task_struct start c17f40c0 pointer offset 0
    2022-10-11T04:41:33.830219  kern  :alert : Register r9 information: slab kmalloc-512 start c2cfa000 pointer offset 64 size 512
    2022-10-11T04:41:33.830370  kern  :alert : Register r10 information: NULL pointer 
    ... (44 line(s) more)  

  * baseline.dmesg.alert: https://kernelci.org/test/case/id/6344f446dcb429c27ccab60f
        failing since 12 days (last pass: next-20220921, first fail: next-20220928)
        16 lines

    2022-10-11T04:41:33.743509  kern  :alert : 8<--- cut here ---
    2022-10-11T04:41:33.744046  kern  :alert : Unable to handle kernel NULL pointer dereference at virtual address 00000000
    2022-10-11T04:41:33.744304  kern  :a<8>[   43.013600] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=alert RESULT=fail UNITS=lines MEASUREMENT=16>
    2022-10-11T04:41:33.744491  lert : [00000000] *pgd=f9e09835
    2022-10-11T04:41:33.744687  kern  :alert : Register r0 information: slab kmalloc-1k start c2a3b400 pointer offset 0 size 1024   

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
kontron-kbox-a-230-ls        | arm64 | lab-kontron     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f57654ba049ee5cab620

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-kontron/baseline-kontron-kbox-a-230-ls.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-kontron/baseline-kontron-kbox-a-230-ls.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f57654ba049ee5cab621
        failing since 6 days (last pass: next-20220923, first fail: next-20221004) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
kontron-pitx-imx8m           | arm64 | lab-kontron     | clang-16 | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/6344f3d5b8ba8579aacab66f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    clang-16 (Debian clang version 16.0.0-++20221003071821+2ddbe56b34ab-1~exp1~20221003071926.408)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-kontron/baseline-kontron-pitx-imx8m.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/clang-16/lab-kontron/baseline-kontron-pitx-imx8m.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f3d5b8ba8579aacab670
        failing since 5 days (last pass: next-20220923, first fail: next-20221005) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
kontron-sl28-var3-ads2       | arm64 | lab-kontron     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f742d3a0bb2252cab604

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-kontron/baseline-kontron-sl28-var3-ads2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-kontron/baseline-kontron-sl28-var3-ads2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f742d3a0bb2252cab605
        failing since 6 days (last pass: next-20220923, first fail: next-20221004) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-g12a-sei510            | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f4d43bd5414b38cab5ed

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-meson-g12a-sei510.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-meson-g12a-sei510.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f4d43bd5414b38cab5ee
        failing since 17 days (last pass: next-20220916, first fail: next-20220923) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-g12a-u200              | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f564c25aa35174cab614

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-meson-g12a-u200.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-meson-g12a-u200.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f564c25aa35174cab615
        failing since 17 days (last pass: next-20220912, first fail: next-20220923) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-g12b-odroid-n2         | arm64 | lab-baylibre    | gcc-10   | defconfig+CON..._64K_PAGES=y | 1          

  Details:     https://kernelci.org/test/plan/id/6344f2bf06d0906306cab607

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+CONFIG_ARM64_64K_PAGES=y
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-baylibre/baseline-meson-g12b-odroid-n2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+CONFIG_ARM64_64K_PAGES=y/gcc-10/lab-baylibre/baseline-meson-g12b-odroid-n2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f2bf06d0906306cab608
        failing since 0 day (last pass: next-20221006, first fail: next-20221010) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-g12b-odroid-n2         | arm64 | lab-baylibre    | gcc-10   | defconfig+ima                | 1          

  Details:     https://kernelci.org/test/plan/id/6344f9340b3a6ffddfcab5f0

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+ima
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+ima/gcc-10/lab-baylibre/baseline-meson-g12b-odroid-n2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+ima/gcc-10/lab-baylibre/baseline-meson-g12b-odroid-n2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f9340b3a6ffddfcab5f1
        failing since 0 day (last pass: next-20221006, first fail: next-20221010) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f54ffba69a150bcab65e

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f54ffba69a150bcab65f
        failing since 17 days (last pass: next-20220912, first fail: next-20220923) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre    | gcc-10   | defconfig+videodec           | 1          

  Details:     https://kernelci.org/test/plan/id/6344f79b38f0a5e8aecab60a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+videodec
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+videodec/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+videodec/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f79b38f0a5e8aecab60b
        failing since 0 day (last pass: next-20221006, first fail: next-20221010) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre    | gcc-10   | defconfig+crypto             | 1          

  Details:     https://kernelci.org/test/plan/id/6344f91660952b7ffdcab5f7

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+crypto
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+crypto/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+crypto/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f91660952b7ffdcab5f8
        new failure (last pass: next-20221010) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
mt8183-kukui-...uniper-sku16 | arm64 | lab-collabora   | clang-13 | cros://chrome...4-chromebook | 1          

  Details:     https://kernelci.org/test/plan/id/6345000b31ec2620f1cab607

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: cros://chromeos-5.10/arm64/chromiumos-arm64.flavour.config+arm64-chromebook
  Compiler:    clang-13 (Debian clang version 13.0.1-6~deb11u1)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/cros---chromeos-5.10-arm64-chromiumos-arm64.flavour.config+arm64-chromebook/clang-13/lab-collabora/baseline-mt8183-kukui-jacuzzi-juniper-sku16.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/cros---chromeos-5.10-arm64-chromiumos-arm64.flavour.config+arm64-chromebook/clang-13/lab-collabora/baseline-mt8183-kukui-jacuzzi-juniper-sku16.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6345000b31ec2620f1cab608
        failing since 88 days (last pass: next-20220705, first fail: next-20220714) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
ox820-clouden...lug-series-3 | arm   | lab-baylibre    | gcc-10   | oxnas_v6_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/6344f1b3bdd55e3c92cab5fc

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: oxnas_v6_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/oxnas_v6_defconfig/gcc-10/lab-baylibre/baseline-ox820-cloudengines-pogoplug-series-3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/oxnas_v6_defconfig/gcc-10/lab-baylibre/baseline-ox820-cloudengines-pogoplug-series-3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f1b3bdd55e3c92cab5fd
        failing since 48 days (last pass: next-20220822, first fail: next-20220823) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f6909f7fcf3362cab5f2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f6909f7fcf3362cab5f3
        failing since 27 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f9df370238b11ccab607

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f9df370238b11ccab608
        failing since 27 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f7989f1a32c62fcab5ed

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f7989f1a32c62fcab5ee
        failing since 27 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi     | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f6939f7fcf3362cab5f5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f6939f7fcf3362cab5f6
        failing since 25 days (last pass: next-20220907, first fail: next-20220915) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi     | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344fb5b87f203a213cab5fe

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344fb5b87f203a213cab5ff
        failing since 25 days (last pass: next-20220907, first fail: next-20220915) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi     | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f7b99f1a32c62fcab608

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f7b99f1a32c62fcab609
        failing since 25 days (last pass: next-20220907, first fail: next-20220915) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f69266752df6f8cab5f4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f69266752df6f8cab5f5
        failing since 25 days (last pass: next-20220907, first fail: next-20220915) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344fa43931371e5f7cab608

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344fa43931371e5f7cab609
        failing since 25 days (last pass: next-20220907, first fail: next-20220915) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f79934c4d575c4cab5f5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f79934c4d575c4cab5f6
        failing since 25 days (last pass: next-20220907, first fail: next-20220915) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi     | arm   | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f694a1fef06dfacab5ff

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f694a1fef06dfacab600
        failing since 27 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi     | arm   | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344fbbf20187e2bafcab5f2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344fbbf20187e2bafcab5f3
        failing since 27 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi     | arm   | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6344f7c09f1a32c62fcab60b

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f7c09f1a32c62fcab60c
        failing since 27 days (last pass: next-20220907, first fail: next-20220913) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv2        | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f53f2ecd047258cab614

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f53f2ecd047258cab615
        failing since 17 days (last pass: next-20220916, first fail: next-20220923) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv2-uefi   | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f5402ecd047258cab617

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f5402ecd047258cab618
        failing since 28 days (last pass: next-20220908, first fail: next-20220912) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv3        | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f56854ba049ee5cab5f7

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f56854ba049ee5cab5f8
        failing since 0 day (last pass: next-20221005, first fail: next-20221010) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
qemu_arm64-virt-gicv3-uefi   | arm64 | lab-collabora   | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f525dd30e5206ccab607

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm64-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f525dd30e5206ccab608
        failing since 28 days (last pass: next-20220908, first fail: next-20220912) 

platform                     | arch  | lab             | compiler | defconfig                    | regressions
-----------------------------+-------+-----------------+----------+------------------------------+------------
rk3328-rock64                | arm64 | lab-baylibre    | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/6344f63b4a8fb9e369cab62a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-rk3328-rock64.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20221011/arm64/defconfig+debug/gcc-10/lab-baylibre/baseline-rk3328-rock64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20220919.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6344f63b4a8fb9e369cab62b
        failing since 6 days (last pass: next-20220912, first fail: next-20221004)