Subject: next/master baseline: 276 runs, 32 regressions (next-20240322)

From: "kernelci.org bot" <[email-masked]>
Date: Fri, 22 Mar 2024 03:09:14 -0700 (PDT)

next/master baseline: 276 runs, 32 regressions (next-20240322)

Regressions Summary
-------------------

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
acer-R721T-grunt       | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          
at91-sama5d4_xplained  | arm    | lab-baylibre    | gcc-10   | sama5_defconfig              | 1          
bcm2836-rpi-2-b        | arm    | lab-collabora   | gcc-10   | bcm2835_defconfig            | 1          
bcm2836-rpi-2-b        | arm    | lab-collabora   | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          
beagle-xm              | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          
beagle-xm              | arm    | lab-baylibre    | gcc-10   | omap2plus_defconfig          | 1          
hp-11A-G6-EE-grunt     | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          
hp-14-db0003na-grunt   | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          
imx6dl-riotboard       | arm    | lab-pengutronix | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          
imx6dl-riotboard       | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
imx6qp-wandboard-revd1 | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
jetson-tk1             | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
jetson-tk1             | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
kontron-kbox-a-230-ls  | arm64  | lab-kontron     | gcc-10   | defconfig+videodec           | 5          
kontron-sl28-var3-ads2 | arm64  | lab-kontron     | gcc-10   | defconfig+videodec           | 2          
mt8173-elm-hana        | arm64  | lab-collabora   | gcc-10   | defconfig+arm64-chromebook   | 2          
qemu_arm-virt-gicv2    | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2    | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3    | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3    | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
rk3288-rock2-square    | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
rk3288-veyron-jaq      | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
sun7i-a20-cubieboard2  | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          
sun7i-a20-cubieboard2  | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
sun7i-a20-cubieboard2  | arm    | lab-clabbe      | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          
sun7i-a20-cubieboard2  | arm    | lab-clabbe      | gcc-10   | multi_v7_defconfig           | 1          

  Details:  https://kernelci.org/test/job/next/branch/master/kernel/next-20240322/plan/baseline/

  Test:     baseline
  Tree:     next
  Branch:   master
  Describe: next-20240322
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
  SHA:      13ee4a7161b6fd938aef6688ff43b163f6d83e37 

Test Regressions
---------------- 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
acer-R721T-grunt       | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1e0b01ce9240b54c42df

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-acer-R721T-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-acer-R721T-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1e0b01ce9240b54c42e0
        failing since 22 days (last pass: next-20240226, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
at91-sama5d4_xplained  | arm    | lab-baylibre    | gcc-10   | sama5_defconfig              | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1fe8fe67b284c44c43a9

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: sama5_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/sama5_defconfig/gcc-10/lab-baylibre/baseline-at91-sama5d4_xplained.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/sama5_defconfig/gcc-10/lab-baylibre/baseline-at91-sama5d4_xplained.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1fe8fe67b284c44c43aa
        failing since 415 days (last pass: next-20230125, first fail: next-20230131) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
bcm2836-rpi-2-b        | arm    | lab-collabora   | gcc-10   | bcm2835_defconfig            | 1          

  Details:     https://kernelci.org/test/plan/id/65fd20809f157fe78f4c42db

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: bcm2835_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/bcm2835_defconfig/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/bcm2835_defconfig/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd20809f157fe78f4c42dc
        failing since 291 days (last pass: next-20230601, first fail: next-20230605) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
bcm2836-rpi-2-b        | arm    | lab-collabora   | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          

  Details:     https://kernelci.org/test/plan/id/65fd189ac6124385194c431d

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd189ac6124385194c431e
        failing since 359 days (last pass: next-20230327, first fail: next-20230328) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
beagle-xm              | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          

  Details:     https://kernelci.org/test/plan/id/65fd18846a579f18f14c433a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-baylibre/baseline-beagle-xm.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-baylibre/baseline-beagle-xm.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd18846a579f18f14c433b
        failing since 356 days (last pass: next-20230330, first fail: next-20230331) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
beagle-xm              | arm    | lab-baylibre    | gcc-10   | omap2plus_defconfig          | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1b2731800f22144c42e3

  Results:     3 PASS, 1 FAIL, 1 SKIP
  Full config: omap2plus_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-beagle-xm.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-beagle-xm.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1b2731800f22144c42ea
        new failure (last pass: next-20240321)

    2024-03-22T05:45:46.178816  Segmentation fault
    2024-03-22T05:45:46.281486  / # #
    2024-03-22T05:45:46.382511  export SHELL=/bin/sh
    2024-03-22T05:45:46.382931  #
    2024-03-22T05:45:46.483756  / # export SHELL=/bin/sh. /lava-3960835/environment
    2024-03-22T05:45:46.485125  
    2024-03-22T05:45:46.586817  / # . /lava-3960835/environment/lava-3960835/bin/lava-test-runner /lava-3960835/1
    2024-03-22T05:45:46.587525  
    2024-03-22T05:45:46.593234  / # /lava-3960835/bin/lava-test-runner /lava-3960835/1
    2024-03-22T05:45:46.635896  date: symbol lookup error: /lib/libc.so.6: undefined symbol: __pointer_chk_guard, version  
    ... (12 line(s) more)  

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
hp-11A-G6-EE-grunt     | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1e0464821781dd4c42fd

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-11A-G6-EE-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-11A-G6-EE-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1e0464821781dd4c42fe
        failing since 22 days (last pass: next-20240226, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
hp-14-db0003na-grunt   | x86_64 | lab-collabora   | gcc-10   | x86_64_defconfig+x86-board   | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1dfca749c136aa4c42df

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-board
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-14-db0003na-grunt.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/x86_64/x86_64_defconfig+x86-board/gcc-10/lab-collabora/baseline-hp-14-db0003na-grunt.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/x86/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1dfca749c136aa4c42e0
        failing since 22 days (last pass: next-20240226, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
imx6dl-riotboard       | arm    | lab-pengutronix | gcc-10   | multi_v7_defc...MB2_KERNEL=y | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1859f8cd4a00d74c439b

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1859f8cd4a00d74c43a4
        new failure (last pass: next-20240321)

    2024-03-22T05:33:54.146859  + set[   15.974705] <LAVA_SIGNAL_ENDRUN 0_dmesg 1035982_1.5.2.3.1>
    2024-03-22T05:33:54.147006   +x
    2024-03-22T05:33:54.252151  / # #
    2024-03-22T05:33:54.353371  export SHELL=/bin/sh
    2024-03-22T05:33:54.353991  #
    2024-03-22T05:33:54.454994  / # export SHELL=/bin/sh. /lava-1035982/environment
    2024-03-22T05:33:54.455442  
    2024-03-22T05:33:54.556207  / # . /lava-1035982/environment/lava-1035982/bin/lava-test-runner /lava-1035982/1
    2024-03-22T05:33:54.556745  
    2024-03-22T05:33:54.559768  / # /lava-1035982/bin/lava-test-runner /lava-1035982/1 
    ... (12 line(s) more)  

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
imx6dl-riotboard       | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1a75301c78fdde4c433f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6dl-riotboard.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1a75301c78fdde4c4340
        failing since 590 days (last pass: next-20220808, first fail: next-20220810) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
imx6qp-wandboard-revd1 | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1a74301c78fdde4c433a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1a74301c78fdde4c433b
        failing since 577 days (last pass: next-20220810, first fail: next-20220822) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
jetson-tk1             | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/65fd209c9f157fe78f4c42ef

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd209c9f157fe78f4c42f0
        failing since 1 day (last pass: next-20240315, first fail: next-20240320) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
jetson-tk1             | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/65fd2256363f3b59bc4c42ea

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd2256363f3b59bc4c42eb
        failing since 710 days (last pass: next-20220401, first fail: next-20220411) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
kontron-kbox-a-230-ls  | arm64  | lab-kontron     | gcc-10   | defconfig+videodec           | 5          

  Details:     https://kernelci.org/test/plan/id/65fd1787e86ac77e0b4c42de

  Results:     90 PASS, 5 FAIL, 1 SKIP
  Full config: defconfig+videodec
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm64/defconfig+videodec/gcc-10/lab-kontron/baseline-kontron-kbox-a-230-ls.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm64/defconfig+videodec/gcc-10/lab-kontron/baseline-kontron-kbox-a-230-ls.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1787e86ac77e0b4c42e5
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:22.425758  / ##
    2024-03-22T05:30:22.526802  export SHELL=/bin/sh
    2024-03-22T05:30:22.527106   #
    2024-03-22T05:30:22.627768  / # export SHELL=/bin/sh. /lava-446019/environment
    2024-03-22T05:30:22.628163  
    2024-03-22T05:30:22.729127  / # . /lava-446019/environment/lava-446019/bin/lava-test-runner /lava-446019/1
    2024-03-22T05:30:22.730110  
    2024-03-22T05:30:22.738249  / # /lava-446019/bin/lava-test-runner /lava-446019/1
    2024-03-22T05:30:22.794582  + export 'TESTRUN_ID=1_bootrr'
    2024-03-22T05:30:22.795007  + <8>[   20.933481] <LAVA_SIGNAL_STARTRUN 1_bootrr 446019_1.5.2.4.5> 
    ... (14 line(s) more)  

  * baseline.bootrr.fsl_enetc-enetc2-probed: https://kernelci.org/test/case/id/65fd1787e86ac77e0b4c42e9
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:24.903899  /lava-446019/1/../bin/lava-test-case
    2024-03-22T05:30:24.904320  <8>[   23.044533] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=fsl_enetc-enetc2-probed RESULT=fail>
    2024-03-22T05:30:24.904637  /lava-446019/1/../bin/lava-test-case   

  * baseline.bootrr.mscc_felix-probed: https://kernelci.org/test/case/id/65fd1787e86ac77e0b4c42eb
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:25.962314  /lava-446019/1/../bin/lava-test-case
    2024-03-22T05:30:25.962821  <8>[   24.083610] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=mscc_felix-probed RESULT=fail>   

  * baseline.bootrr.fsl_enetc-enetc0-probed: https://kernelci.org/test/case/id/65fd1787e86ac77e0b4c42f0
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:27.037620  /lava-446019/1/../bin/lava-test-case   

  * baseline.bootrr.fsl_enetc-enetc1-probed: https://kernelci.org/test/case/id/65fd1787e86ac77e0b4c42f1
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:27.040830  <8>[   25.196447] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=fsl_enetc-enetc0-probed RESULT=fail>
    2024-03-22T05:30:28.078005  /lava-446019/1/../bin/lava-test-case   

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
kontron-sl28-var3-ads2 | arm64  | lab-kontron     | gcc-10   | defconfig+videodec           | 2          

  Details:     https://kernelci.org/test/plan/id/65fd1785520ba76e5b4c42db

  Results:     101 PASS, 2 FAIL, 1 SKIP
  Full config: defconfig+videodec
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm64/defconfig+videodec/gcc-10/lab-kontron/baseline-kontron-sl28-var3-ads2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm64/defconfig+videodec/gcc-10/lab-kontron/baseline-kontron-sl28-var3-ads2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1785520ba76e5b4c42e2
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:23.660117  / # #
    2024-03-22T05:30:23.762140  export SHELL=/bin/sh
    2024-03-22T05:30:23.762873  #
    2024-03-22T05:30:23.863954  / # export SHELL=/bin/sh. /lava-446017/environment
    2024-03-22T05:30:23.864472  
    2024-03-22T05:30:23.965315  / # . /lava-446017/environment/lava-446017/bin/lava-test-runner /lava-446017/1
    2024-03-22T05:30:23.966270  
    2024-03-22T05:30:24.006119  / # /lava-446017/bin/lava-test-runner /lava-446017/1
    2024-03-22T05:30:24.039065  + export 'TESTRUN_ID=1_bootrr'
    2024-03-22T05:30:24.039270  + <8>[   21.128423] <LAVA_SIGNAL_STARTRUN 1_bootrr 446017_1.5.2.4.5> 
    ... (11 line(s) more)  

  * baseline.bootrr.fsl_enetc-enetc0-probed: https://kernelci.org/test/case/id/65fd1785520ba76e5b4c42f5
        failing since 100 days (last pass: next-20231207, first fail: next-20231212)

    2024-03-22T05:30:26.481042  /lava-446017/1/../bin/lava-test-case
    2024-03-22T05:30:26.481474  <8>[   23.556963] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=fsl_enetc-enetc0-probed RESULT=fail>
    2024-03-22T05:30:26.481747  /lava-446017/1/../bin/lava-test-case   

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
mt8173-elm-hana        | arm64  | lab-collabora   | gcc-10   | defconfig+arm64-chromebook   | 2          

  Details:     https://kernelci.org/test/plan/id/65fd18e25fbad14eea4c43c6

  Results:     27 PASS, 2 FAIL, 1 SKIP
  Full config: defconfig+arm64-chromebook
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-mt8173-elm-hana.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm64/defconfig+arm64-chromebook/gcc-10/lab-collabora/baseline-mt8173-elm-hana.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/arm64/rootfs.cpio.gz 

  * baseline.bootrr.armv8-pmu-a72-probed: https://kernelci.org/test/case/id/65fd18e25fbad14eea4c43e3
        failing since 35 days (last pass: next-20240214, first fail: next-20240215)

    2024-03-22T05:36:12.146114  <8>[   31.118954] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=armv8-pmu-a53-probed RESULT=fail>

    2024-03-22T05:36:13.168762  /lava-13118091/1/../bin/lava-test-case

    2024-03-22T05:36:13.175854  <8>[   32.149716] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=armv8-pmu-a72-probed RESULT=fail>

  * baseline.bootrr.armv8-pmu-a53-probed: https://kernelci.org/test/case/id/65fd18e25fbad14eea4c43e4
        failing since 35 days (last pass: next-20240214, first fail: next-20240215)

    2024-03-22T05:36:12.134326  /lava-13118091/1/../bin/lava-test-case

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2    | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/65fd196f3274a22fd84c435a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd196f3274a22fd84c435b
        failing since 22 days (last pass: next-20240122, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2    | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/65fd193c43c088540d4c439e

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd193c43c088540d4c439f
        failing since 22 days (last pass: next-20240122, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3    | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/65fd196d3274a22fd84c431f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd196d3274a22fd84c4320
        failing since 22 days (last pass: next-20240122, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3    | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1965c3f21a692d4c42dc

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd1965c3f21a692d4c42dd
        failing since 22 days (last pass: next-20240122, first fail: next-20240228) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
rk3288-rock2-square    | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/65fd237553af261c604c42e9

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-rock2-square.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-rock2-square.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd237553af261c604c42ea
        failing since 485 days (last pass: next-20221121, first fail: next-20221122) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
rk3288-veyron-jaq      | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/65fd2239bd49e99c6a4c42e3

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-veyron-jaq.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-veyron-jaq.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/65fd2239bd49e99c6a4c42e4
        failing since 485 days (last pass: next-20221121, first fail: next-20221122) 

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
sun7i-a20-cubieboard2  | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1e954580cc2c514c4309

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+CONFIG_SMP=n
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-baylibre/baseline-sun7i-a20-cubieboard2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-baylibre/baseline-sun7i-a20-cubieboard2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1e954580cc2c514c4312
        failing since 295 days (last pass: next-20230530, first fail: next-20230601)

    2024-03-22T06:00:28.168421  / # #
    2024-03-22T06:00:28.269734  export SHELL=/bin/sh
    2024-03-22T06:00:28.270516  #
    2024-03-22T06:00:28.371750  / # export SHELL=/bin/sh. /lava-3960871/environment
    2024-03-22T06:00:28.372129  
    2024-03-22T06:00:28.472825  / # . /lava-3960871/environment/lava-3960871/bin/lava-test-runner /lava-3960871/1
    2024-03-22T06:00:28.473363  
    2024-03-22T06:00:28.492227  / # /lava-3960871/bin/lava-test-runner /lava-3960871/1
    2024-03-22T06:00:28.606150  + export 'TESTRUN_ID=1_bootrr'
    2024-03-22T06:00:28.607105  + cd /lava-3960871/1/tests/1_bootrr 
    ... (10 line(s) more)  

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
sun7i-a20-cubieboard2  | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1fc036810311494c431a

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun7i-a20-cubieboard2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-sun7i-a20-cubieboard2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1fc036810311494c4323
        failing since 295 days (last pass: next-20230530, first fail: next-20230601)

    2024-03-22T06:05:25.896065  <8>[   13.780106] <LAVA_SIGNAL_ENDRUN 0_dmesg 3960896_1.5.2.4.1>
    2024-03-22T06:05:26.005171  / # #
    2024-03-22T06:05:26.108189  export SHELL=/bin/sh
    2024-03-22T06:05:26.109413  #
    2024-03-22T06:05:26.211198  / # export SHELL=/bin/sh. /lava-3960896/environment
    2024-03-22T06:05:26.212290  
    2024-03-22T06:05:26.313911  / # . /lava-3960896/environment/lava-3960896/bin/lava-test-runner /lava-3960896/1
    2024-03-22T06:05:26.315369  
    2024-03-22T06:05:26.327687  / # /lava-3960896/bin/lava-test-runner /lava-3960896/1
    2024-03-22T06:05:26.407401  + export 'TESTRUN_ID=1_bootrr' 
    ... (11 line(s) more)  

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
sun7i-a20-cubieboard2  | arm    | lab-clabbe      | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1e794580cc2c514c42eb

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+CONFIG_SMP=n
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-clabbe/baseline-sun7i-a20-cubieboard2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-clabbe/baseline-sun7i-a20-cubieboard2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1e794580cc2c514c42f4
        failing since 295 days (last pass: next-20230530, first fail: next-20230601)

    2024-03-22T06:00:09.751606  + set +x
    2024-03-22T06:00:09.753376  [   24.096537] <LAVA_SIGNAL_ENDRUN 0_dmesg 463086_1.5.2.4.1>
    2024-03-22T06:00:09.860429  / # #
    2024-03-22T06:00:09.962229  export SHELL=/bin/sh
    2024-03-22T06:00:09.963289  #
    2024-03-22T06:00:10.064376  / # export SHELL=/bin/sh. /lava-463086/environment
    2024-03-22T06:00:10.064825  
    2024-03-22T06:00:10.165715  / # . /lava-463086/environment/lava-463086/bin/lava-test-runner /lava-463086/1
    2024-03-22T06:00:10.166479  
    2024-03-22T06:00:10.169013  / # /lava-463086/bin/lava-test-runner /lava-463086/1 
    ... (12 line(s) more)  

platform               | arch   | lab             | compiler | defconfig                    | regressions
-----------------------+--------+-----------------+----------+------------------------------+------------
sun7i-a20-cubieboard2  | arm    | lab-clabbe      | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/65fd1fa48f4b5904164c4322

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig/gcc-10/lab-clabbe/baseline-sun7i-a20-cubieboard2.txt
  HTML log:    https://storage.kernelci.org//next/master/next-20240322/arm/multi_v7_defconfig/gcc-10/lab-clabbe/baseline-sun7i-a20-cubieboard2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230703.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/65fd1fa48f4b5904164c432b
        failing since 295 days (last pass: next-20230530, first fail: next-20230601)

    2024-03-22T06:05:03.885748  + set +x[   18.736123] <LAVA_SIGNAL_ENDRUN 0_dmesg 463090_1.5.2.4.1>
    2024-03-22T06:05:03.886079  
    2024-03-22T06:05:03.995889  / # #
    2024-03-22T06:05:04.097695  export SHELL=/bin/sh
    2024-03-22T06:05:04.098268  #
    2024-03-22T06:05:04.199242  / # export SHELL=/bin/sh. /lava-463090/environment
    2024-03-22T06:05:04.199757  
    2024-03-22T06:05:04.300669  / # . /lava-463090/environment/lava-463090/bin/lava-test-runner /lava-463090/1
    2024-03-22T06:05:04.301569  
    2024-03-22T06:05:04.304294  / # /lava-463090/bin/lava-test-runner /lava-463090/1 
    ... (12 line(s) more)