Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)

From: Veronika Kabatova <vkabatov@redhat•com>
To: Will Deacon <will@kernel•org>
Cc: catalin marinas <catalin.marinas@arm•com>,
	linux-arm-kernel@lists•infradead.org,
	CKI Project <cki-project@redhat•com>
Subject: Re: ❌ FAIL: Test report for kernel	5.11.0-rc7 (arm-next)
Date: Wed, 10 Feb 2021 10:24:31 -0500 (EST)	[thread overview]
Message-ID: <27613639.27281944.1612970671896.JavaMail.zimbra@redhat.com> (raw)
In-Reply-To: <2112681230.27250520.1612958496505.JavaMail.zimbra@redhat.com>

----- Original Message -----
> From: "Veronika Kabatova" <vkabatov@redhat•com>
> To: "Will Deacon" <will@kernel•org>
> Cc: "catalin marinas" <catalin.marinas@arm•com>, "CKI Project" <cki-project@redhat•com>,
> linux-arm-kernel@lists•infradead.org
> Sent: Wednesday, February 10, 2021 1:01:36 PM
> Subject: Re: ❌ FAIL: Test report for kernel	5.11.0-rc7 (arm-next)
> 
> 
> 
> ----- Original Message -----
> > From: "Will Deacon" <will@kernel•org>
> > To: "CKI Project" <cki-project@redhat•com>
> > Cc: "catalin marinas" <catalin.marinas@arm•com>,
> > linux-arm-kernel@lists•infradead.org
> > Sent: Wednesday, February 10, 2021 10:29:46 AM
> > Subject: Re: ❌ FAIL: Test report for kernel	5.11.0-rc7 (arm-next)
> > 
> > Hi CKI folks,
> > 
> > Please can you help me with the error below?
> > 
> > On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
> > > We ran automated tests on a recent commit from this kernel tree:
> > > 
> > >        Kernel repo:
> > >        https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
> > >             Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature'
> > >             into
> > >             for-kernelci
> > > 
> > > The results of these automated tests are provided below.
> > > 
> > >     Overall result: FAILED (see details below)
> > >              Merge: OK
> > >            Compile: OK
> > >              Tests: FAILED
> > > 
> > > All kernel binaries, config files, and logs are available for download
> > > here:
> > > 
> > >   https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431
> > > 
> > > One or more kernel tests failed:
> > > 
> > >     aarch64:
> > >      ❌ Boot test
> > 
> > [...]
> > 
> > >     Host 2:
> > >        ❌ Boot test
> > >        ⚡⚡⚡ selinux-policy: serge-testsuite
> > >        ⚡⚡⚡ storage: software RAID testing
> > >        🚧 ⚡⚡⚡ xfstests - ext4
> > >        🚧 ⚡⚡⚡ xfstests - xfs
> > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > >        🚧 ⚡⚡⚡ IPMI driver test
> > >        🚧 ⚡⚡⚡ IPMItool loop stress test
> > >        🚧 ⚡⚡⚡ Storage blktests
> > >        🚧 ⚡⚡⚡ Storage block - filesystem fio test
> > >        🚧 ⚡⚡⚡ Storage block - queue scheduler test
> > >        🚧 ⚡⚡⚡ Storage nvme - tcp
> > >        🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
> > >        🚧 ⚡⚡⚡ stress: stress-ng
> > 
> > Which system (e.g. soc) is host 2 and are there are known infra issues at
> > the moment? I did push some changes which affect the early boot path, so we
> > may well be running into a kernel bug, but I'd just like to make sure
> > before
> > we dive in trying to debug that, especially as we haven't seen failures on
> > other systems (and host 1 seems ok).
> > 
> 
> Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
> kernel just a few days back okay. The last messages I can see in the raw
> console log from this run are:
> 
> EFI stub: Booting Linux Kernel...
> EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services and installing virtual address map...
> 
> and then it times out after hour and half. I'm not aware of any ongoing
> issues, however sometimes the link between the lab controller and the
> machines can sometimes go wrong after reboot and lead to a similarly
> looking problem.
> 
> I'll resubmit the test job on that same machine to check if that was
> the case and let you know right after it boots.
> 

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value

But this message is not present with the stable run that I mentioned
previously.

We didn't change the builder container for a while now and since I tested
things on the same machine one run after other, there should also be no
distro/package changes involved and the only difference should be the
kernel.

Veronika

> 
> Veronika
> 
> > In the meantime, I'll push a new branch without those early changes to see
> > if that does any better.
> > 
> > Thanks,
> > 
> > Will
> > 
> > 
> > 
> 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists•infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel