Subject: next/pending-fixes baseline: 365 runs, 33 regressions (v6.3-rc3-550-g902f54bbd6c7)

From: "kernelci.org bot" <[email-masked]>
Date: Sun, 26 Mar 2023 19:22:16 -0700 (PDT)

next/pending-fixes baseline: 365 runs, 33 regressions (v6.3-rc3-550-g902f54bbd6c7)

Regressions Summary
-------------------

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15            | arm    | lab-linaro-lkft | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          
am57xx-beagle-x15            | arm    | lab-linaro-lkft | gcc-10   | multi_v7_defconfig+debug     | 1          
asus-CM1400CXA-dalboz        | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...ok+kselftest | 1          
bcm2835-rpi-b-rev2           | arm    | lab-broonie     | gcc-10   | bcm2835_defconfig            | 1          
bcm2836-rpi-2-b              | arm    | lab-collabora   | gcc-10   | bcm2835_defconfig            | 1          
beagle-xm                    | arm    | lab-baylibre    | gcc-10   | omap2plus_defconfig          | 1          
cubietruck                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          
cubietruck                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+crypto    | 1          
cubietruck                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          
da850-lcdk                   | arm    | lab-baylibre    | gcc-10   | davinci_all_defconfig        | 2          
imx6qp-wandboard-revd1       | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          
imx6ul-14x14-evk             | arm    | lab-nxp         | gcc-10   | multi_v7_defconfig+crypto    | 1          
imx7d-sdb                    | arm    | lab-nxp         | gcc-10   | multi_v7_defconfig+crypto    | 1          
imx7ulp-evk                  | arm    | lab-nxp         | gcc-10   | multi_v7_defconfig+crypto    | 1          
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
jh7100-starfi...isionfive-v1 | riscv  | lab-clabbe      | gcc-10   | defconfig+kselftest          | 1          
kontron-kswit...0-mmt-6g-2gs | arm    | lab-kontron     | gcc-10   | multi_v7_defconfig+crypto    | 1          
meson-sm1-khadas-vim3l       | arm64  | lab-baylibre    | gcc-10   | defconfig+CON...BIG_ENDIAN=y | 1          
qemu_arm-virt-gicv2          | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi     | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi     | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv2-uefi     | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi     | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi     | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          
qemu_arm-virt-gicv3-uefi     | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          
rk3288-rock2-square          | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          
rk3288-veyron-jaq            | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:  https://kernelci.org/test/job/next/branch/pending-fixes/kernel/v6.3-rc3-550-g902f54bbd6c7/plan/baseline/

  Test:     baseline
  Tree:     next
  Branch:   pending-fixes
  Describe: v6.3-rc3-550-g902f54bbd6c7
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
  SHA:      902f54bbd6c704147f2fc71dcb4f1c64a05d5201 

Test Regressions
---------------- 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15            | arm    | lab-linaro-lkft | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          

  Details:     https://kernelci.org/test/plan/id/6420c33711ccc84f009c951f

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_SMP=n
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c33711ccc84f009c9520
        failing since 167 days (last pass: v6.0-6424-g1af0c6a154d32, first fail: v6.0-9666-g02c05e0b8d5c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
am57xx-beagle-x15            | arm    | lab-linaro-lkft | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc3488c10c73cd9c9511

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-linaro-lkft/baseline-am57xx-beagle-x15.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc3488c10c73cd9c9512
        failing since 165 days (last pass: v6.0-9666-g02c05e0b8d5c, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
asus-CM1400CXA-dalboz        | x86_64 | lab-collabora   | gcc-10   | x86_64_defcon...ok+kselftest | 1          

  Details:     https://kernelci.org/test/plan/id/6420c1fc88afd2c72f9c952b

  Results:     6 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig+x86-chromebook+kselftest
  Compiler:    gcc-10 (gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-10/lab-collabora/baseline-asus-CM1400CXA-dalboz.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-10/lab-collabora/baseline-asus-CM1400CXA-dalboz.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/x86/rootfs.cpio.gz 

  * baseline.dmesg.emerg: https://kernelci.org/test/case/id/6420c1fc88afd2c72f9c952e
        failing since 11 days (last pass: v6.3-rc2-254-g7f35d1f08bb0, first fail: v6.3-rc2-307-g900e881df1ea3)
        3 lines

    2023-03-26T22:06:29.675210  kern  :emerg : __common_interrupt: 1.55 No irq handler for vector

    2023-03-26T22:06:29.682054  kern  :emerg : __common_interrupt: 2.55 No irq handler for vector

    2023-03-26T22:06:29.687899  kern  :emerg : __common_interrupt: 3.55 No irq handler for vector

    2023-03-26T22:06:29.694378  <8>[   14.248372] <LAVA_SIGNAL_TESTCASE TEST_CASE_ID=emerg RESULT=fail UNITS=lines MEASUREMENT=3>

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
bcm2835-rpi-b-rev2           | arm    | lab-broonie     | gcc-10   | bcm2835_defconfig            | 1          

  Details:     https://kernelci.org/test/plan/id/6420c0c4b39872b1489c950a

  Results:     50 PASS, 2 FAIL, 1 SKIP
  Full config: bcm2835_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/bcm2835_defconfig/gcc-10/lab-broonie/baseline-bcm2835-rpi-b-rev2.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/bcm2835_defconfig/gcc-10/lab-broonie/baseline-bcm2835-rpi-b-rev2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c0c5b39872b1489c9541
        failing since 12 days (last pass: v6.3-rc1-336-gbec2983895c5, first fail: v6.3-rc2-254-g7f35d1f08bb0)

    2023-03-26T22:01:17.962835  + set +x
    2023-03-26T22:01:17.966547  <8>[   17.664877] <LAVA_SIGNAL_ENDRUN 0_dmesg 231771_1.5.2.4.1>
    2023-03-26T22:01:18.099683  / # #
    2023-03-26T22:01:18.202578  export SHELL=/bin/sh
    2023-03-26T22:01:18.203247  #
    2023-03-26T22:01:18.305447  / # export SHELL=/bin/sh. /lava-231771/environment
    2023-03-26T22:01:18.306160  
    2023-03-26T22:01:18.408481  / # . /lava-231771/environment/lava-231771/bin/lava-test-runner /lava-231771/1
    2023-03-26T22:01:18.409619  
    2023-03-26T22:01:18.416139  / # /lava-231771/bin/lava-test-runner /lava-231771/1 
    ... (14 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
bcm2836-rpi-2-b              | arm    | lab-collabora   | gcc-10   | bcm2835_defconfig            | 1          

  Details:     https://kernelci.org/test/plan/id/6420c0c839855039ac9c9511

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: bcm2835_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/bcm2835_defconfig/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/bcm2835_defconfig/gcc-10/lab-collabora/baseline-bcm2836-rpi-2-b.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c0c839855039ac9c951a
        failing since 74 days (last pass: v6.2-rc3-243-g3145d9dfed32, first fail: v6.2-rc3-296-g79fa898551af)

    2023-03-26T22:01:28.555794  / # #

    2023-03-26T22:01:28.658367  export SHELL=/bin/sh

    2023-03-26T22:01:28.659136  #

    2023-03-26T22:01:28.761176  / # export SHELL=/bin/sh. /lava-9779338/environment

    2023-03-26T22:01:28.761971  

    2023-03-26T22:01:28.864053  / # . /lava-9779338/environment/lava-9779338/bin/lava-test-runner /lava-9779338/1

    2023-03-26T22:01:28.865347  

    2023-03-26T22:01:28.875647  / # /lava-9779338/bin/lava-test-runner /lava-9779338/1

    2023-03-26T22:01:28.988781  + export 'TESTRUN_ID=1_bootrr'

    2023-03-26T22:01:28.989333  + cd /lava-9779338/1/tests/1_bootrr
 
    ... (12 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
beagle-xm                    | arm    | lab-baylibre    | gcc-10   | omap2plus_defconfig          | 1          

  Details:     https://kernelci.org/test/plan/id/6420c4be0f6dbfddc19c9506

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: omap2plus_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-beagle-xm.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/omap2plus_defconfig/gcc-10/lab-baylibre/baseline-beagle-xm.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c4be0f6dbfddc19c9507
        failing since 17 days (last pass: v6.3-rc1-217-g03913bd4e939, first fail: v6.3-rc1-281-g08705758878ed) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
cubietruck                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...CONFIG_SMP=n | 1          

  Details:     https://kernelci.org/test/plan/id/6420c367b8b13b4b6a9c951a

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+CONFIG_SMP=n
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-baylibre/baseline-cubietruck.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_SMP=n/gcc-10/lab-baylibre/baseline-cubietruck.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c367b8b13b4b6a9c9523
        failing since 68 days (last pass: v6.1-13341-g16cba743aec0, first fail: v6.2-rc4-302-g48ed69cf21b2)

    2023-03-26T22:12:39.515496  <8>[   17.208246] <LAVA_SIGNAL_ENDRUN 0_dmesg 3446859_1.5.2.4.1>
    2023-03-26T22:12:39.624683  / # #
    2023-03-26T22:12:39.727758  export SHELL=/bin/sh
    2023-03-26T22:12:39.728662  #
    2023-03-26T22:12:39.830578  / # export SHELL=/bin/sh. /lava-3446859/environment
    2023-03-26T22:12:39.831450  
    2023-03-26T22:12:39.933335  / # . /lava-3446859/environment/lava-3446859/bin/lava-test-runner /lava-3446859/1
    2023-03-26T22:12:39.935082  
    2023-03-26T22:12:39.939899  / # /lava-3446859/bin/lava-test-runner /lava-3446859/1
    2023-03-26T22:12:40.020906  + export 'TESTRUN_ID=1_bootrr' 
    ... (11 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
cubietruck                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+crypto    | 1          

  Details:     https://kernelci.org/test/plan/id/6420c575f56cb22e9b9c950e

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+crypto
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-baylibre/baseline-cubietruck.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-baylibre/baseline-cubietruck.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c575f56cb22e9b9c9517
        failing since 68 days (last pass: v6.1-13341-g16cba743aec0, first fail: v6.2-rc4-302-g48ed69cf21b2)

    2023-03-26T22:21:19.183467  + set +x<8>[   24.164844] <LAVA_SIGNAL_ENDRUN 0_dmesg 3446901_1.5.2.4.1>
    2023-03-26T22:21:19.184120  
    2023-03-26T22:21:19.293336  / # #
    2023-03-26T22:21:19.397077  export SHELL=/bin/sh
    2023-03-26T22:21:19.398214  #
    2023-03-26T22:21:19.500604  / # export SHELL=/bin/sh. /lava-3446901/environment
    2023-03-26T22:21:19.501797  
    2023-03-26T22:21:19.604372  / # . /lava-3446901/environment/lava-3446901/bin/lava-test-runner /lava-3446901/1
    2023-03-26T22:21:19.606215  
    2023-03-26T22:21:19.610892  / # /lava-3446901/bin/lava-test-runner /lava-3446901/1 
    ... (12 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
cubietruck                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig           | 1          

  Details:     https://kernelci.org/test/plan/id/6420c5e814c65821869c9529

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-cubietruck.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig/gcc-10/lab-baylibre/baseline-cubietruck.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c5e814c65821869c9532
        failing since 68 days (last pass: v6.1-13341-g16cba743aec0, first fail: v6.2-rc4-302-g48ed69cf21b2)

    2023-03-26T22:23:25.438292  <8>[   14.756333] <LAVA_SIGNAL_ENDRUN 0_dmesg 3446947_1.5.2.4.1>
    2023-03-26T22:23:25.550641  / # #
    2023-03-26T22:23:25.652142  export SHELL=/bin/sh
    2023-03-26T22:23:25.653181  #
    2023-03-26T22:23:25.755433  / # export SHELL=/bin/sh. /lava-3446947/environment
    2023-03-26T22:23:25.755893  
    2023-03-26T22:23:25.857265  / # . /lava-3446947/environment/lava-3446947/bin/lava-test-runner /lava-3446947/1
    2023-03-26T22:23:25.857816  
    2023-03-26T22:23:25.864404  / # /lava-3446947/bin/lava-test-runner /lava-3446947/1
    2023-03-26T22:23:25.965143  + export 'TESTRUN_ID=1_bootrr' 
    ... (11 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
da850-lcdk                   | arm    | lab-baylibre    | gcc-10   | davinci_all_defconfig        | 2          

  Details:     https://kernelci.org/test/plan/id/6420c5b8e125cfa5859c9506

  Results:     3 PASS, 3 FAIL, 1 SKIP
  Full config: davinci_all_defconfig
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/davinci_all_defconfig/gcc-10/lab-baylibre/baseline-da850-lcdk.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/davinci_all_defconfig/gcc-10/lab-baylibre/baseline-da850-lcdk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.dmesg.emerg: https://kernelci.org/test/case/id/6420c5b8e125cfa5859c9509
        failing since 23 days (last pass: v6.2-1218-g2d4a54ee969d, first fail: v6.2-13370-g5872d227b73f)
        30 lines

    2023-03-26T22:22:24.816131  kern  :alert : Register r6 information: NULL pointer
    2023-03-26T22:22:24.861046  kern  :alert : Register r7 information: non-paged memory
    2023-03-26T22:22:24.861349  kern  :alert : Register r8 information: slab kmalloc-512 start c4e20600 pointer offset 4 size 512
    2023-03-26T22:22:24.861539  kern  :alert : Register r9 information: non-paged memory
    2023-03-26T22:22:24.861911  kern  :alert : Register r10 information: NULL pointer
    2023-03-26T22:22:24.862108  kern  :alert : Register r11 information: 5-page vmalloc region starting at 0xbf3e2000 allocated at load_module+0x898/0x1b10
    2023-03-26T22:22:24.863910  kern  :alert : Register r12 information: non-paged memory
    2023-03-26T22:22:24.952145  kern  :emerg : Internal error: Oops: 17 [#1] PREEMPT ARM
    2023-03-26T22:22:24.952430  kern  :emerg : Process udevd (pid: 73, stack limit = 0x0173d782)
    2023-03-26T22:22:24.952609  kern  :emerg : Stack: (0xc898dcf8 to 0xc898e000) 
    ... (8 line(s) more)  

  * baseline.dmesg.alert: https://kernelci.org/test/case/id/6420c5b8e125cfa5859c950a
        failing since 427 days (last pass: v5.16-11577-gffd79fec234d, first fail: v5.17-rc1-180-g86539e2bdb99)
        18 lines

    2023-03-26T22:22:24.769590  kern  :alert : BUG: Bad page state in process kworker/u2:0  pfn:c3400
    2023-03-26T22:22:24.769882  kern  :alert : BUG: Bad page state in process kworker/u2:0  pfn:c3800
    2023-03-26T22:22:24.770073  kern  :alert : 8<--- cut here ---
    2023-03-26T22:22:24.770246  kern  :alert : Unable to handle kernel NULL pointer dereference at virtual address 00000060 when read
    2023-03-26T22:22:24.770416  kern  :alert : [00000060] *pgd=c4a9e831, *pte=00000000, *ppte=00000000
    2023-03-26T22:22:24.770586  kern  :alert : Register r0 information: non-paged memory
    2023-03-26T22:22:24.813118  kern  :alert : Register r1 information: 2-page vmalloc region starting at 0xc898c000 allocated at kernel_clone+0x98/0x3a0
    2023-03-26T22:22:24.813401  kern  :alert : Register r2 information: non-slab/vmalloc memory
    2023-03-26T22:22:24.813582  kern  :alert : Register r3 information: non-paged memory
    2023-03-26T22:22:24.813751  kern  :alert : R<8><LAVA_SIGNAL_TESTCASE TEST_CASE_ID=alert RESULT=fail UNITS=lines MEASUREMENT=18> 
    ... (2 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6qp-wandboard-revd1       | arm    | lab-pengutronix | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc3988c10c73cd9c9517

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-pengutronix/baseline-imx6qp-wandboard-revd1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc3988c10c73cd9c9518
        failing since 192 days (last pass: v6.0-rc4-291-g83a56f559828, first fail: v6.0-rc5-197-ga6a750a2f4166) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx6ul-14x14-evk             | arm    | lab-nxp         | gcc-10   | multi_v7_defconfig+crypto    | 1          

  Details:     https://kernelci.org/test/plan/id/6420c45f6bd10236d89c950b

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+crypto
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-nxp/baseline-imx6ul-14x14-evk.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-nxp/baseline-imx6ul-14x14-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c45f6bd10236d89c9512
        failing since 80 days (last pass: v6.1-rc4-339-g185da88d03ed, first fail: v6.2-rc2-332-g2d3ca5d07c8e)

    2023-03-26T22:16:43.916577  + set +x
    2023-03-26T22:16:43.917658  <8>[   58.192083] <LAVA_SIGNAL_ENDRUN 0_dmesg 1187151_1.5.2.4.1>
    2023-03-26T22:16:44.025683  
    2023-03-26T22:16:45.184518  / # #export SHELL=/bin/sh
    2023-03-26T22:16:45.189943  
    2023-03-26T22:16:46.735817  / # export SHELL=/bin/sh. /lava-1187151/environment
    2023-03-26T22:16:46.741307  
    2023-03-26T22:16:49.561452  / # . /lava-1187151/environment/lava-1187151/bin/lava-test-runner /lava-1187151/1
    2023-03-26T22:16:49.567176  
    2023-03-26T22:16:49.567352  / # /lava-1187151/bin/lava-test-runner /lava-1187151/1 
    ... (18 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx7d-sdb                    | arm    | lab-nxp         | gcc-10   | multi_v7_defconfig+crypto    | 1          

  Details:     https://kernelci.org/test/plan/id/6420c44b0fc889802e9c95f7

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+crypto
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-nxp/baseline-imx7d-sdb.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-nxp/baseline-imx7d-sdb.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c44b0fc889802e9c95fe
        failing since 80 days (last pass: v6.1-rc6-391-gf445421fe4c7, first fail: v6.2-rc2-332-g2d3ca5d07c8e)

    2023-03-26T22:16:26.182667  <8>[   38.353800] <LAVA_SIGNAL_ENDRUN 0_dmesg 1187149_1.5.2.4.1>
    2023-03-26T22:16:26.286711  / # #
    2023-03-26T22:16:27.446424  export SHELL=/bin/sh
    2023-03-26T22:16:27.452120  #
    2023-03-26T22:16:28.999203  / # export SHELL=/bin/sh. /lava-1187149/environment
    2023-03-26T22:16:29.004830  
    2023-03-26T22:16:31.824094  / # . /lava-1187149/environment/lava-1187149/bin/lava-test-runner /lava-1187149/1
    2023-03-26T22:16:31.829716  
    2023-03-26T22:16:31.842649  / # /lava-1187149/bin/lava-test-runner /lava-1187149/1
    2023-03-26T22:16:31.921663  + export 'TESTRUN_ID=1_bootrr' 
    ... (22 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
imx7ulp-evk                  | arm    | lab-nxp         | gcc-10   | multi_v7_defconfig+crypto    | 1          

  Details:     https://kernelci.org/test/plan/id/6420c4c2c310f3ee179c959d

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: multi_v7_defconfig+crypto
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-nxp/baseline-imx7ulp-evk.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-nxp/baseline-imx7ulp-evk.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420c4c2c310f3ee179c95a4
        failing since 80 days (last pass: v6.1-rc6-391-gf445421fe4c7, first fail: v6.2-rc2-332-g2d3ca5d07c8e)

    2023-03-26T22:18:28.917544  + set +x
    2023-03-26T22:18:28.918671  <8>[   66.868035] <LAVA_SIGNAL_ENDRUN 0_dmesg 1187148_1.5.2.4.1>
    2023-03-26T22:18:29.027473  / # #
    2023-03-26T22:18:30.187485  export SHELL=/bin/sh
    2023-03-26T22:18:30.193182  #
    2023-03-26T22:18:30.193500  / # export SHELL=/bin/sh
    2023-03-26T22:18:31.741036  / # . /lava-1187148/environment
    2023-03-26T22:18:34.567581  /lava-1187148/bin/lava-test-runner /lava-1187148/1
    2023-03-26T22:18:34.573663  . /lava-1187148/environment
    2023-03-26T22:18:34.573957  / # /lava-1187148/bin/lava-test-runner /lava-1187148/1 
    ... (12 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
jetson-tk1                   | arm    | lab-baylibre    | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/6420c3363d703409f79c9596

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-baylibre/baseline-jetson-tk1.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-baylibre/baseline-jetson-tk1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c3363d703409f79c9597
        failing since 331 days (last pass: v5.18-rc2-366-ga3e1163f7eb1a, first fail: v5.18-rc4-464-g425675974eb3) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
jh7100-starfi...isionfive-v1 | riscv  | lab-clabbe      | gcc-10   | defconfig+kselftest          | 1          

  Details:     https://kernelci.org/test/plan/id/6420c1edc3e2035d1e9c9524

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig+kselftest
  Compiler:    gcc-10 (riscv64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/riscv/defconfig+kselftest/gcc-10/lab-clabbe/baseline-jh7100-starfive-visionfive-v1.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/riscv/defconfig+kselftest/gcc-10/lab-clabbe/baseline-jh7100-starfive-visionfive-v1.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/riscv/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c1edc3e2035d1e9c9525
        failing since 47 days (last pass: v6.2-rc7-120-g11b3628314a5, first fail: v6.2-rc7-136-g39693c3ba989) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
kontron-kswit...0-mmt-6g-2gs | arm    | lab-kontron     | gcc-10   | multi_v7_defconfig+crypto    | 1          

  Details:     https://kernelci.org/test/plan/id/6420c45b653e0630c69c9598

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+crypto
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-kontron/baseline-kontron-kswitch-d10-mmt-6g-2gs.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+crypto/gcc-10/lab-kontron/baseline-kontron-kswitch-d10-mmt-6g-2gs.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c45b653e0630c69c9599
        new failure (last pass: v6.3-rc3-370-g8269040171a02) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
meson-sm1-khadas-vim3l       | arm64  | lab-baylibre    | gcc-10   | defconfig+CON...BIG_ENDIAN=y | 1          

  Details:     https://kernelci.org/test/plan/id/6420ca82eb07143d409c9505

  Results:     5 PASS, 1 FAIL, 1 SKIP
  Full config: defconfig+CONFIG_CPU_BIG_ENDIAN=y
  Compiler:    gcc-10 (aarch64-linux-gnu-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm64/defconfig+CONFIG_CPU_BIG_ENDIAN=y/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm64/defconfig+CONFIG_CPU_BIG_ENDIAN=y/gcc-10/lab-baylibre/baseline-meson-sm1-khadas-vim3l.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/arm64be/rootfs.cpio.gz 

  * baseline.bootrr.deferred-probe-empty: https://kernelci.org/test/case/id/6420ca82eb07143d409c950e
        failing since 2 days (last pass: v6.3-rc3-288-g2dc3fa9ff5523, first fail: v6.3-rc3-370-g8269040171a02)

    2023-03-26T22:43:06.336735  / # #
    2023-03-26T22:43:06.440451  export SHELL=/bin/sh
    2023-03-26T22:43:06.442021  <3>[   18.248809] mmc2: tuning execution failed: -5
    2023-03-26T22:43:06.442720  <3>[   18.248861] mmc2: error -5 whilst initialising SDIO card
    2023-03-26T22:43:06.443313  #
    2023-03-26T22:43:06.545861  / # export SHELL=/bin/sh. /lava-3446982/environment
    2023-03-26T22:43:06.546957  
    2023-03-26T22:43:06.649214  / # . /lava-3446982/environment/lava-3446982/bin/lava-test-runner /lava-3446982/1
    2023-03-26T22:43:06.651819  
    2023-03-26T22:43:06.666106  / # /lava-3446982/bin/lava-test-runner /lava-3446982/1 
    ... (18 line(s) more)  

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc423c1a78535b9c9511

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc423c1a78535b9c9512
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc4088c10c73cd9c9522

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc4088c10c73cd9c9523
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc2d528a54b11f9c9508

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc2d528a54b11f9c9509
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi     | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc453c1a78535b9c951a

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc453c1a78535b9c951b
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi     | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc52536a79cd749c9511

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc52536a79cd749c9512
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv2-uefi     | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc4088c10c73cd9c9525

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv2-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc4088c10c73cd9c9526
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc443c1a78535b9c9517

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc443c1a78535b9c9518
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc413c1a78535b9c950b

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc413c1a78535b9c950c
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3          | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc3288c10c73cd9c950b

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc3288c10c73cd9c950c
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi     | arm    | lab-baylibre    | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc403c1a78535b9c9508

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-baylibre/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc403c1a78535b9c9509
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi     | arm    | lab-broonie     | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc3e528a54b11f9c9519

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-broonie/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc3e528a54b11f9c951a
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
qemu_arm-virt-gicv3-uefi     | arm    | lab-collabora   | gcc-10   | multi_v7_defconfig+debug     | 1          

  Details:     https://kernelci.org/test/plan/id/6420cc2c88c10c73cd9c9505

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+debug
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3-uefi.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+debug/gcc-10/lab-collabora/baseline-qemu_arm-virt-gicv3-uefi.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420cc2c88c10c73cd9c9506
        failing since 165 days (last pass: v6.0-5324-g7871897dadfa9, first fail: v6.0-11312-g1778d6da389c) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
rk3288-rock2-square          | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/6420c4244a558c5fb19c951e

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-rock2-square.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-rock2-square.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c4244a558c5fb19c951f
        failing since 101 days (last pass: v6.1-6378-g44d433ee7540, first fail: v6.1-10792-g244d284981da) 

platform                     | arch   | lab             | compiler | defconfig                    | regressions
-----------------------------+--------+-----------------+----------+------------------------------+------------
rk3288-veyron-jaq            | arm    | lab-collabora   | gcc-10   | multi_v7_defc...G_ARM_LPAE=y | 1          

  Details:     https://kernelci.org/test/plan/id/6420c320368b9bd2759c95f2

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y
  Compiler:    gcc-10 (arm-linux-gnueabihf-gcc (Debian 10.2.1-6) 10.2.1 20210110)
  Plain log:   https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-veyron-jaq.txt
  HTML log:    https://storage.kernelci.org//next/pending-fixes/v6.3-rc3-550-g902f54bbd6c7/arm/multi_v7_defconfig+CONFIG_EFI=y+CONFIG_ARM_LPAE=y/gcc-10/lab-collabora/baseline-rk3288-veyron-jaq.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/20230310.0/armel/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/6420c320368b9bd2759c95f3
        failing since 101 days (last pass: v6.1-6378-g44d433ee7540, first fail: v6.1-10792-g244d284981da)