Subject: [linux-next] 5.17.0-rc1-next-20220124 kernel fails to boot on my powerpc box

From: Abdul Haleem <[email-masked]>
Date: Tue, 25 Jan 2022 20:22:21 +0530

Greeting's

Today's next kernel 5.17.0-rc1-next-20220124(first bad) fails to boot on 
my powerpc box.

I see it booted fine last Friday for kernel 5.16.0-next-20220121

Boot logs:

Removing IBM Power 842 compression device
device-mapper: multipath: 253:0: Failing path 8:0.
scsi 0:0:0:0: alua: Detached
scsi 0:0:1:0: alua: Detached
kexec_core: Starting new kernel
kexec: waiting for cpu 16 (physical 16) to enter 1 state
kexec: waiting for cpu 17 (physical 17) to enter 1 state
kexec: waiting for cpu 1 (physical 1) to enter 2 state
kexec: waiting for cpu 3 (physical 3) to enter 2 state
kexec: waiting for cpu 8 (physical 8) to enter 2 state
kexec: waiting for cpu 9 (physical 9) to enter 2 state
kexec: waiting for cpu 10 (physical 10) to enter 2 state
kexec: waiting for cpu 11 (physical 11) to enter 2 state
kexec: waiting for cpu 13 (physical 13) to enter 2 state
kexec: waiting for cpu 16 (physical 16) to enter 2 state
kexec: waiting for cpu 17 (physical 17) to enter 2 state
kexec: Starting switchover sequence.
y
Reserving 1024MB of memory at 128MB for crashkernel (System RAM: 30720MB)
hash-mmu: Page sizes from device-tree:
hash-mmu: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, 
tlbiel=1, penc=0
hash-mmu: base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, 
tlbiel=1, penc=7
hash-mmu: base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, 
tlbiel=1, penc=56
hash-mmu: base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, 
tlbiel=1, penc=1
hash-mmu: base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, 
tlbiel=1, penc=8
hash-mmu: base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, 
tlbiel=0, penc=0
hash-mmu: base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, 
tlbiel=0, penc=3
Enabling pkeys with max key count 31
Activating Kernel Userspace Access Prevention
Activating Kernel Userspace Execution Prevention
Using 1TB segments
hash-mmu: Initializing hash mmu with SLB
Linux version 5.17.0-rc1-next-20220124-autotest 
([email-masked]) (gcc (GCC) 8.3.1 20190507 (Red 
Hat 8.3.1-4), GNU ld version 2.30-58.el8) #1 SMP Tue Jan 25 04:47:57 CST 
2022
Found initrd at 0xc000000005800000:0xc000000007d2892c
Using pSeries machine description
printk: bootconsole [udbg0] enabled
Partition configured for 24 cpus.
CPU maps initialized for 8 threads per core
numa: Partition configured for 32 NUMA nodes.
-----------------------------------------------------
phys_mem_size     = 0x780000000
dcache_bsize      = 0x80
icache_bsize      = 0x80
cpu_features      = 0x0001c07b8f5f9187
   possible        = 0x000ffbfbcf5fb187
   always          = 0x0000000380008181
cpu_user_features = 0xdc0065c2 0xeff00000
mmu_features      = 0x7c006e01
firmware_features = 0x0000009fc45bfc57
vmalloc start     = 0xc008000000000000
IO start          = 0xc00a000000000000
vmemmap start     = 0xc00c000000000000
hash-mmu: ppc64_pft_size    = 0x1c
hash-mmu: htab_hash_mask    = 0x1fffff
-----------------------------------------------------
numa:   NODE_DATA [mem 0x77ff1ef00-0x77ff23fff]
rfi-flush: fallback displacement flush available
rfi-flush: mttrig type flush available
count-cache-flush: flush disabled.
link-stack-flush: software flush enabled.
stf-barrier: eieio barrier available
lpar: H_BLOCK_REMOVE supports base psize:0 psize:0 block size:8
lpar: H_BLOCK_REMOVE supports base psize:0 psize:2 block size:8
lpar: H_BLOCK_REMOVE supports base psize:0 psize:10 block size:8
lpar: H_BLOCK_REMOVE supports base psize:2 psize:2 block size:8
lpar: H_BLOCK_REMOVE supports base psize:2 psize:10 block size:8
PPC64 nvram contains 15360 bytes
PV qspinlock hash table entries: 4096 (order: 0, 65536 bytes, linear)
barrier-nospec: using ORI speculation barrier
Zone ranges:
   Normal   [mem 0x0000000000000000-0x000000077fffffff]
Movable zone start for each node
Early memory node ranges
   node   0: [mem 0x0000000000000000-0x000000077fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000077fffffff]
percpu: Embedded 10 pages/cpu s590744 r0 d64616 u1048576
Fallback order for Node 0: 0
Built 1 zonelists, mobility grouping on.  Total pages: 491040
Policy zone: Normal
Kernel command line: 
BOOT_IMAGE=/vmlinuz-5.16.0-rc5-next-20211220-autotest 
root=UUID=8b32580a-0de0-4694-882d-7db2567ad115 ro crashkernel=1024M 
biosdevname=0 ibmvfc.mq=0
Unknown kernel command line parameters 
"BOOT_IMAGE=/vmlinuz-5.16.0-rc5-next-20211220-autotest biosdevname=0", 
will be passed to user space.
Dentry cache hash table entries: 4194304 (order: 9, 33554432 bytes, linear)
Inode-cache hash table entries: 2097152 (order: 8, 16777216 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 30158976K/31457280K available (12992K kernel code, 5696K rwdata, 
4160K rodata, 3904K init, 2588K bss, 1298304K reserved, 0K cma-reserved)
SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=24, Nodes=32
trace event string verifier disabled
rcu: Hierarchical RCU implementation.
rcu:     RCU event tracing is enabled.
rcu:     RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=24.
rcu:     RCU debug extended QS entry/exit.
     Trampoline variant of Tasks RCU enabled.
     Rude variant of Tasks RCU enabled.
     Tracing variant of Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=24
NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
random: get_random_u64 called from start_kernel+0x65c/0x8bc with crng_init=0
time_init: 56 bit decrementer (max: 7fffffffffffff)
clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 
0x761537d007, max_idle_ns: 440795202126 ns
clocksource: timebase mult[1f40000] shift[24] registered
Console: colour dummy device 80x25
printk: console [hvc0] enabled
printk: console [hvc0] enabled
printk: bootconsole [udbg0] disabled
printk: bootconsole [udbg0] disabled
pid_max: default: 32768 minimum: 301
LSM: Security Framework initializing
Yama: becoming mindful.
SELinux:  Initializing.
Mount-cache hash table entries: 65536 (order: 3, 524288 bytes, linear)
Mountpoint-cache hash table entries: 65536 (order: 3, 524288 bytes, linear)
cblist_init_generic: Setting adjustable number of callback queues.
cblist_init_generic: Setting shift to 4 and lim to 1.
cblist_init_generic: Setting shift to 4 and lim to 1.
cblist_init_generic: Setting shift to 4 and lim to 1.
POWER9 performance monitor hardware support registered
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
smp: Brought up 1 node, 24 CPUs
numa: Node 0 CPUs: 0-23
Big cores detected but using small core scheduling
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, 
max_idle_ns: 19112604462750000 ns
futex hash table entries: 8192 (order: 4, 1048576 bytes, linear)
NET: Registered PF_NETLINK/PF_ROUTE protocol family
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(1643112925.010:1): state=initialized 
audit_enabled=0 res=1
thermal_sys: Registered thermal governor 'fair_share'
thermal_sys: Registered thermal governor 'step_wise'
cpuidle: using governor menu
pstore: Registered nvram as persistent store backend
EEH: pSeries platform initialized
software IO TLB: tearing down default memory pool
PCI: Probing PCI hardware
EEH: No capable adapters found: recovery disabled.
pseries-rng: Registering arch random hook.
kprobes: kprobe jump-optimization is enabled. All kprobes are optimized 
if possible.
HugeTLB registered 16.0 MiB page size, pre-allocated 0 pages
HugeTLB registered 16.0 GiB page size, pre-allocated 0 pages
iommu: Default domain type: Translated
iommu: DMA domain TLB invalidation policy: strict mode
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
EDAC MC: Ver: 3.0.0
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
NetLabel:  unlabeled traffic allowed by default
clocksource: Switched to clocksource timebase
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
NET: Registered PF_INET protocol family
IP idents hash table entries: 262144 (order: 5, 2097152 bytes, linear)
tcp_listen_portaddr_hash hash table entries: 16384 (order: 2, 262144 
bytes, linear)
TCP established hash table entries: 262144 (order: 5, 2097152 bytes, linear)
TCP bind hash table entries: 65536 (order: 4, 1048576 bytes, linear)
TCP: Hash tables configured (established 262144 bind 65536)
UDP hash table entries: 16384 (order: 3, 524288 bytes, linear)
UDP-Lite hash table entries: 16384 (order: 3, 524288 bytes, linear)
NET: Registered PF_UNIX/PF_LOCAL protocol family
PCI: CLS 0 bytes, default 128
Trying to unpack rootfs image as initramfs...
IOMMU table initialized, virtual merging enabled
vas: API is supported only with radix page tables
hv-24x7: read 1530 catalog entries, created 509 event attrs (0 
failures), 275 descs
Initialise system trusted keyrings
workingset: timestamp_bits=38 max_order=19 bucket_order=0
zbud: loaded
NET: Registered PF_ALG protocol family
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
io scheduler mq-deadline registered
io scheduler kyber registered
atomic64_test: passed
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
rdac: device handler registered
hp_sw: device handler registered
emc: device handler registered
alua: device handler registered
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci-pci: OHCI PCI platform driver
uhci_hcd: USB Universal Host Controller Interface driver
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
mousedev: PS/2 mouse device common for all mice
rtc-generic rtc-generic: registered as rtc0
rtc-generic rtc-generic: setting system clock to 2022-01-25T12:15:25 UTC 
(1643112925)
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_size 
new:65536 old:0
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_sg 
new:510 old:0
nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sg_len 
new:4080 old:0
NX-GZIP is not supported. Returned=-524
hid: raw HID events driver (C) Jiri Kosina
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
drop_monitor: Initializing network drop monitor service
Initializing XFRM netlink socket
NET: Registered PF_INET6 protocol family
Segment Routing with IPv6
In-situ OAM (IOAM) with IPv6
NET: Registered PF_PACKET protocol family
registered taskstats version 1
Loading compiled-in X.509 certificates
Freeing initrd memory: 38016K
alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1))
Loaded X.509 cert 'Build time autogenerated kernel key: 
984d2d0a058cae59d928008a328cb345ccb6f5b1'
zswap: loaded using pool lzo/zbud
pstore: Using crash dump compression: deflate
alg: No test for 842 (842-nx)
INFO: task swapper/0:1 blocked for more than 122 seconds.
       Not tainted 5.17.0-rc1-next-20220124-autotest #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:swapper/0       state:D stack:    0 pid:    1 ppid:     0 
flags:0x00000800
Call Trace:
[c000000003483960] [c0000000034839a0] 0xc0000000034839a0 (unreliable)
[c000000003483b50] [c00000000001e328] __switch_to+0x278/0x490
[c000000003483bb0] [c000000000c9c2f8] __schedule+0x318/0xa00
[c000000003483c70] [c000000000c9ca48] schedule+0x68/0x130
[c000000003483ca0] [c000000000c9d048] schedule_preempt_disabled+0x18/0x30
[c000000003483cc0] [c000000000c9f49c] __mutex_lock.isra.11+0x36c/0x6f0
[c000000003483d50] [c000000000296228] kprobe_free_init_mem+0x48/0xe8
[c000000003483da0] [c0000000000127c4] kernel_init+0x44/0x1a0
[c000000003483e10] [c00000000000ce64] ret_from_kernel_thread+0x5c/0x64
INFO: task kworker/17:1:145 blocked for more than 122 seconds.
       Not tainted 5.17.0-rc1-next-20220124-autotest #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/17:1    state:D stack:    0 pid:  145 ppid:     2 
flags:0x00000800
Workqueue: events kprobe_optimizer
Call Trace:
[c00000004ccbb6d0] [000000077a820000] 0x77a820000 (unreliable)
[c00000004ccbb8c0] [c00000000001e328] __switch_to+0x278/0x490
[c00000004ccbb920] [c000000000c9c2f8] __schedule+0x318/0xa00
[c00000004ccbb9e0] [c000000000c9ca48] schedule+0x68/0x130
[c00000004ccbba10] [c000000000ca47c8] schedule_timeout+0x348/0x3f0
[c00000004ccbbaf0] [c000000000c9ec20] wait_for_completion+0x140/0x270
[c00000004ccbbb60] [c0000000001fb08c] __wait_rcu_gp+0x21c/0x230
[c00000004ccbbbc0] [c0000000001fb0e0] synchronize_rcu_tasks+0x40/0x70
[c00000004ccbbc20] [c0000000002927d0] kprobe_optimizer+0x1b0/0x3a0
[c00000004ccbbc90] [c0000000001724b8] process_one_work+0x288/0x560
[c00000004ccbbd30] [c000000000172808] worker_thread+0x78/0x620
[c00000004ccbbdc0] [c00000000017e7fc] kthread+0x11c/0x130
[c00000004ccbbe10] [c00000000000ce64] ret_from_kernel_thread+0x5c/0x64

call traces never ends and could not login

-- 
Regard's

Abdul Haleem
IBM Linux Technology Center