Subject: next/master baseline: 443 runs, 32 regressions (next-20240408)

From: "kernelci.org bot" <[email-masked]>
Date: Mon, 08 Apr 2024 02:43:05 -0700 (PDT)

next/master baseline: 443 runs, 32 regressions (next-20240408)

Regressions Summary
-------------------

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
acer-R721T-grunt             | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...rd+kselftest | 1          
acer-R721T-grunt             | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          
at91-sama5d4_xplained        | arm    | lab-baylibre    | gcc-10   | sama5_defconfig              | 1          
bcm2835-rpi-b-rev2           | arm    | lab-broonie     | gcc-10   | bcm2835_defconfig            | 2          
bcm2836-rpi-2-b              | arm    | lab-collabora   | gcc-10   | bcm2835_defconfig            | 1          
bcm2836-rpi-2-b              | arm    | lab-collabora   | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          
beagle-xm                    | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          
beagle-xm                    | arm    | lab-baylibre    | gcc-10   | omap2plus_defconfig          | 1          
hp-11A-G6-EE-grunt           | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...rd+kselftest | 1          
hp-11A-G6-EE-grunt           | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          
hp-14-db0003na-grunt         | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...rd+kselftest | 1          
hp-14-db0003na-grunt         | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          
imx6dl-riotboard             | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
imx6qp-wandboard-revd1       | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
kontron-kswit...0-mmt-6g-2gs | arm    | lab-kontron     | gcc-10   | multi_v7_defconfig+kselftest | 1          
kontron-kswitch-d10-mmt-8g   | arm    | lab-kontron     | gcc-10   | multi_v7_defconfig+kselftest | 1          
mt8173-elm-hana              | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          
mt8192-asurada-spherion-r0   | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          
qemu_arm-virt-gicv2          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_riscv64                 | riscv  | lab-broonie     | gcc-10   | defconfig+debug              | 1          
qemu_riscv64                 | riscv  | lab-collabora   | gcc-10   | defconfig+debug              | 1          
qemu_smp8_riscv64            | riscv  | lab-broonie     | gcc-10   | defconfig+debug              | 1          
qemu_smp8_riscv64            | riscv  | lab-collabora   | gcc-10   | defconfig+debug              | 1          
rk3288-rock2-square          | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
rk3288-veyron-jaq            | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
sc7180-trogdor-kingoftown    | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          
sc7180-trogdo...zor-limozeen | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          
zynqmp-zcu102                | arm64  | lab-cip         | gcc-10   | defconfig+arm...ok+kselftest | 1          

  Details:  https://kernelci.org/test/job/next/branch/master/kernel/next-20240408/plan/baseline/

  Test:     baseline
  Tree:     next
  Branch:   master
  Describe: next-20240408
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
  SHA:      11cb68ad52ac78c81e33b806b531f097e68edfa2 

Test Regressions
---------------- 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
acer-R721T-grunt             | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...rd+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/661381d81577fafd504c42db

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board+kselftest
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board+kselftest/gcc-10/lab-collabora/baseline-acer-R721T-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board+kselftest/gcc-10/lab-collabora/baseline-acer-R721T-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381d81577fafd504c42dc
        failing since 33 days (last pass: next-20240226, first fail: next-20240305) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
acer-R721T-grunt             | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          

  Details:     https://kernelci.org/test/plan/id/661385a0736e80b6ce4c42e3

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-acer-R721T-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-acer-R721T-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661385a0736e80b6ce4c42e4
        failing since 39 days (last pass: next-20240226, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
at91-sama5d4_xplained        | arm    | lab-baylibre    | gcc-10   | sama5_defconfig              | 1          

  Details:     https://kernelci.org/test/plan/id/661381d8f2ed37ceb74c42da

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: sama5_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/sama5_defconfig/gcc-10/lab-baylibre/baseline-at91-sama5d4_xplained.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/sama5_defconfig/gcc-10/lab-baylibre/baseline-at91-sama5d4_xplained.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381d8f2ed37ceb74c42db
        failing since 432 days (last pass: next-20230125, first fail: next-20230131) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
bcm2835-rpi-b-rev2           | arm    | lab-broonie     | gcc-10   | bcm2835_defconfig            | 2          

  Details:     https://kernelci.org/test/plan/id/66137bbd6962e586a94c42ef

  Results:     2 PASS, 2 FAIL, 0 SKIP
  Full config: bcm2835_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/bcm2835_defconfig/gcc-10/lab-broonie/baseline-bcm2835-rpi-b-rev2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/bcm2835_defconfig/gcc-10/lab-broonie/baseline-bcm2835-rpi-b-rev2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.dmesg.emerg: https://kernelci.org/test/case/id/66137bbd6962e586a94c42f2
        failing since 4 days (last pass: next-20240325, first fail: next-20240403)
        24 lines

    2024-04-08T05:07:58.014409  k_struct start c23c6d00 pointer offset 0 size 1920
    2024-04-08T05:07:58.019717  kern  :alert : Register r5 information: NULL pointer
    2024-04-08T05:07:58.025178  kern  :alert : Register r6 information: slab kmalloc-4k start c320b000 pointer offset 0 size 4096
    2024-04-08T05:07:58.036358  kern  :alert : Register r7 information: slab task_struct start c323ed00 pointer offset 0 size 1920
    2024-04-08T05:07:58.047711  kern  :alert : Register r8 information: slab task_struct start c323ed00 pointer offset 0 size 1920
    2024-04-08T05:07:58.047933  kern  :alert : Register r9 information: non-paged memory
    2024-04-08T05:07:58.058333  kern  :alert : Register r10 information: 4-page vmalloc region starting at 0xbf02f000 allocated at load_module+0x730/0x16d8
    2024-04-08T05:07:58.064079  kern  :alert : Register r11 information: NULL pointer
    2024-04-08T05:07:58.073045  kern  :alert : Register r12 information: non-paged memory
    2024-04-08T05:07:58.097219  kern  :emerg : Internal error: Oops: 17 [#1] ARM 
    ... (6 line(s) more)  

  * baseline.dmesg.alert: https://kernelci.org/test/case/id/66137bbd6962e586a94c42f3
        failing since 4 days (last pass: next-20240325, first fail: next-20240403)
        16 lines

    2024-04-08T05:07:57.958600  kern  :alert : 8<--- cut here ---
    2024-04-08T05:07:57.969478  kern  :alert : Unable to handle kernel NULL pointer dereference at virtual address 00000000 when read
    2024-04-08T05:07:57.969725  kern  :alert : [00000000] *pgd=00000000
    2024-04-08T05:07:57.975215  kern  :alert : Register r0 information: non-paged memory
    2024-04-08T05:07:57.980693  kern  :alert : Register r1 information: NULL pointer
    2024-04-08T05:07:57.986126  kern  :alert : Register r2 information: NULL pointer
    2024-04-08T05:07:57.991852  kern  :alert : Register r3 information: non-paged memory
    2024-04-08T05:07:58.007204  kern  :alert : Register r4 information: slab tas<8>[   20.766687] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=alert RESULT=fail UNITS=lines MEASUREMENT=16>   

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
bcm2836-rpi-2-b              | arm    | lab-collabora   | gcc-10   | bcm2835_defconfig            | 1          

  Details:     https://kernelci.org/test/plan/id/66137c7eb15743fdc54c42db

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: bcm2835_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/bcm2835_defconfig/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/bcm2835_defconfig/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/66137c7eb15743fdc54c42dc
        failing since 307 days (last pass: next-20230601, first fail: next-20230605) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
bcm2836-rpi-2-b              | arm    | lab-collabora   | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          

  Details:     https://kernelci.org/test/plan/id/6613839bb84b9576f04c42e0

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613839bb84b9576f04c42e1
        failing since 376 days (last pass: next-20230327, first fail: next-20230328) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
beagle-xm                    | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          

  Details:     https://kernelci.org/test/plan/id/6613832db0aed162664c42fd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-baylibre/baseline-beagle-xm.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-baylibre/baseline-beagle-xm.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613832db0aed162664c42fe
        failing since 373 days (last pass: next-20230330, first fail: next-20230331) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
beagle-xm                    | arm    | lab-baylibre    | gcc-10   | omap2plus_defconfig          | 1          

  Details:     https://kernelci.org/test/plan/id/66137c9cb15743fdc54c42f1

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: omap2plus_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-beagle-xm.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-beagle-xm.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/66137c9cb15743fdc54c42f2
        failing since 3 days (last pass: next-20240403, first fail: next-20240404) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
hp-11A-G6-EE-grunt           | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...rd+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/661381df1577fafd504c42e7

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board+kselftest
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board+kselftest/gcc-10/lab-collabora/baseline-hp-11A-G6-EE-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board+kselftest/gcc-10/lab-collabora/baseline-hp-11A-G6-EE-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381df1577fafd504c42e8
        failing since 33 days (last pass: next-20240226, first fail: next-20240305) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
hp-11A-G6-EE-grunt           | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          

  Details:     https://kernelci.org/test/plan/id/6613859bb59b2c45354c42da

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-11A-G6-EE-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-11A-G6-EE-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613859bb59b2c45354c42db
        failing since 39 days (last pass: next-20240226, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
hp-14-db0003na-grunt         | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...rd+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/6613821ce98675f13b4c42f4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board+kselftest
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board+kselftest/gcc-10/lab-collabora/baseline-hp-14-db0003na-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board+kselftest/gcc-10/lab-collabora/baseline-hp-14-db0003na-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613821ce98675f13b4c42f5
        failing since 33 days (last pass: next-20240226, first fail: next-20240305) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
hp-14-db0003na-grunt         | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          

  Details:     https://kernelci.org/test/plan/id/661385ee4d9b7e2aff4c42da

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-14-db0003na-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-14-db0003na-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661385ee4d9b7e2aff4c42db
        failing since 39 days (last pass: next-20240226, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6dl-riotboard             | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6613831c0deb1f22784c42f2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613831c0deb1f22784c42f3
        failing since 607 days (last pass: next-20220808, first fail: next-20220810) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6qp-wandboard-revd1       | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6613831d0deb1f22784c42f5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613831d0deb1f22784c42f6
        failing since 594 days (last pass: next-20220810, first fail: next-20220822) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
kontron-kswit...0-mmt-6g-2gs | arm    | lab-kontron     | gcc-10   | multi_v7_defconfig+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/66137e144fc68af7ab4c4374

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+kselftest
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+kselftest/gcc-10/lab-kontron/baseline-kontron-kswitch-d10-mmt-6g-2gs.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+kselftest/gcc-10/lab-kontron/baseline-kontron-kswitch-d10-mmt-6g-2gs.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/66137e144fc68af7ab4c4375
        new failure (last pass: next-20240325) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
kontron-kswitch-d10-mmt-8g   | arm    | lab-kontron     | gcc-10   | multi_v7_defconfig+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/66137e1514a1a149c74c42e4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+kselftest
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+kselftest/gcc-10/lab-kontron/baseline-kontron-kswitch-d10-mmt-8g.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+kselftest/gcc-10/lab-kontron/baseline-kontron-kswitch-d10-mmt-8g.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/66137e1514a1a149c74c42e5
        new failure (last pass: next-20240325) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
mt8173-elm-hana              | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/661381a7bf69b879564c42db

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook+kselftest
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-mt8173-elm-hana.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-mt8173-elm-hana.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381a7bf69b879564c42dc
        failing since 14 days (last pass: next-20240320, first fail: next-20240325) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
mt8192-asurada-spherion-r0   | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/661381da1577fafd504c42e1

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook+kselftest
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-mt8192-asurada-spherion-r0.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-mt8192-asurada-spherion-r0.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381da1577fafd504c42e2
        failing since 14 days (last pass: next-20240320, first fail: next-20240325) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/66138266f554cabb204c42e5

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/66138266f554cabb204c42e6
        failing since 39 days (last pass: next-20240122, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/661381eb1577fafd504c42ed

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381eb1577fafd504c42ee
        failing since 39 days (last pass: next-20240122, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/661382ca3143a04eb54c42f1

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661382ca3143a04eb54c42f2
        failing since 39 days (last pass: next-20240122, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/661381ffd1540068184c42e0

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381ffd1540068184c42e1
        failing since 39 days (last pass: next-20240122, first fail: next-20240228) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_riscv64                 | riscv  | lab-broonie     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/661385ae736e80b6ce4c42f1

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_riscv64.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_riscv64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/riscv/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661385ae736e80b6ce4c42f2
        new failure (last pass: next-20240404) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_riscv64                 | riscv  | lab-collabora   | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/661385476fe78a55564c42e6

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_riscv64.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_riscv64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/riscv/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661385476fe78a55564c42e7
        new failure (last pass: next-20240404) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_smp8_riscv64            | riscv  | lab-broonie     | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/661385724bbb54b9cb4c42de

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_smp8_riscv64.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-broonie/baseline-qemu_smp8_riscv64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/riscv/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661385724bbb54b9cb4c42df
        failing since 3 days (last pass: next-20240403, first fail: next-20240404) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_smp8_riscv64            | riscv  | lab-collabora   | gcc-10   | defconfig+debug              | 1          

  Details:     https://kernelci.org/test/plan/id/661385471540961b314c430f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+debug
  Compiler:    gcc-10 (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_smp8_riscv64.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/riscv/defconfig+debug/gcc-10/lab-collabora/baseline-qemu_smp8_riscv64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/riscv/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661385471540961b314c4310
        failing since 3 days (last pass: next-20240403, first fail: next-20240404) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
rk3288-rock2-square          | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/6613855306e1cde73c4c4347

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-rock2-square.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-rock2-square.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613855306e1cde73c4c4348
        failing since 502 days (last pass: next-20221121, first fail: next-20221122) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
rk3288-veyron-jaq            | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/6613844f207d3585d64c42ff

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-veyron-jaq.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-veyron-jaq.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613844f207d3585d64c4300
        failing since 502 days (last pass: next-20221121, first fail: next-20221122) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sc7180-trogdor-kingoftown    | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/661381db1577fafd504c42e4

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook+kselftest
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-sc7180-trogdor-kingoftown.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-sc7180-trogdor-kingoftown.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/661381db1577fafd504c42e5
        failing since 14 days (last pass: next-20240320, first fail: next-20240325) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
sc7180-trogdo...zor-limozeen | arm64  | lab-collabora   | gcc-10   | defconfig+arm...ok+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/66138299f9225752664c42ea

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook+kselftest
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-sc7180-trogdor-lazor-limozeen.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-collabora/baseline-sc7180-trogdor-lazor-limozeen.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/66138299f9225752664c42eb
        failing since 14 days (last pass: next-20240320, first fail: next-20240325) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
zynqmp-zcu102                | arm64  | lab-cip         | gcc-10   | defconfig+arm...ok+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/6613816535589995264c4317

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+arm64-chromebook+kselftest
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-cip/baseline-zynqmp-zcu102.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240408/arm64/defconfig+arm64-chromebook+kselftest/gcc-10/lab-cip/baseline-zynqmp-zcu102.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6613816535589995264c4318
        failing since 14 days (last pass: next-20240320, first fail: next-20240325)