Subject: Re: [PATCH v3 19/35] x86/io_apic: Cleanup trigger/polarity helpers

From: Qian Cai <[email-masked]>
Date: Tue, 10 Nov 2020 01:31:29 -0500

On Sat, 2020-10-24 at 22:35 +0100, David Woodhouse wrote:
> From: Thomas Gleixner <[email-masked]>
> 
> 'trigger' and 'polarity' are used throughout the I/O-APIC code for handling
> the trigger type (edge/level) and the active low/high configuration. While
> there are defines for initializing these variables and struct members, they
> are not used consequently and the meaning of 'trigger' and 'polarity' is
> opaque and confusing at best.
> 
> Rename them to 'is_level' and 'active_low' and make them boolean in various
> structs so it's entirely clear what the meaning is.
> 
> Signed-off-by: Thomas Gleixner <[email-masked]>
> Signed-off-by: David Woodhouse <[email-masked]>
> ---
>  arch/x86/include/asm/hw_irq.h       |   6 +-
>  arch/x86/kernel/apic/io_apic.c      | 244 +++++++++++++---------------
>  arch/x86/pci/intel_mid_pci.c        |   8 +-
>  drivers/iommu/amd/iommu.c           |  10 +-
>  drivers/iommu/intel/irq_remapping.c |   9 +-
>  5 files changed, 130 insertions(+), 147 deletions(-)

Reverting the rest of patchset up to this commit on next-20201109 fixed an
endless soft-lockups issue booting an AMD server below. I noticed that the
failed boots always has this IOMMU IO_PAGE_FAULT before those soft-lockups:

[ 3404.093354][    T1] AMD-Vi: Interrupt remapping enabled
[ 3404.098593][    T1] AMD-Vi: Virtual APIC enabled
[ 3404.107783][  T340] pci 0000:00:14.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0000 address=0xfffffffdf8020200 flags=0x0008]
[ 3404.120644][    T1] AMD-Vi: Lazy IO/TLB flushing enabled
[ 3404.126011][    T1] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 3404.133173][    T1] software IO TLB: mapped [mem 0x0000000068dcf000-0x000000006cdcf000] (64MB)

.config (if ever matters):
https://cailca.coding.net/public/linux/mm/git/files/master/x86.config

good boot dmesg (with the commits reverted):
http://people.redhat.com/qcai/dmesg.txt

== system info ==
Dell Poweredge R6415
AMD EPYC 7401P 24-Core Processor
24576 MB memory, 239 GB disk space

[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
[  OK  ] Reached target Local File Systems (Pre).
         Mounting /boot...
[  OK  ] Created slice system-lvm2\x2dpvscan.slice.
[ 3740.376500][ T1058] XFS (sda1): Mounting V5 Filesystem
[ 3740.438474][ T1058] XFS (sda1): Ending clean mount
[ 3765.159433][    C0] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 3765.166929][    C0] Modules linked in: acpi_cpufreq(+) ip_tables x_tables sd_mod ahci libahci tg3 bnxt_en megaraid_sas libata firmware_class libphy dm_mirror dm_region_hash dm_log dm_mod
[ 3765.183576][    C0] irq event stamp: 26230104
[ 3765.187954][    C0] hardirqs last  enabled at (26230103): [<ffffffff82800b9e>] asm_common_interrupt+0x1e/0x40
[ 3765.197873][    C0] hardirqs last disabled at (26230104): [<ffffffff8277e45a>] sysvec_apic_timer_interrupt+0xa/0xa0
[ 3765.208303][    C0] softirqs last  enabled at (26202664): [<ffffffff82a0061b>] __do_softirq+0x61b/0x95d
[ 3765.217699][    C0] softirqs last disabled at (26202591): [<ffffffff82800ec2>] asm_call_irq_on_stack+0x12/0x20
[ 3765.227702][    C0] CPU: 0 PID: 1 Comm: systemd Not tainted 5.10.0-rc2-next-20201109+ #2
[ 3765.235793][    C0] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.9.3 06/25/2019
[ 3765.244065][    C0] RIP: 0010:lock_acquire+0x1f4/0x820
lock_acquire at kernel/locking/lockdep.c:5404
[ 3765.249211][    C0] Code: ff ff ff 48 83 c4 20 65 0f c1 05 a7 ba 9e 7e 83 f8 01 4c 8b 54 24 08 0f 85 60 04 00 00 41 52 9d 48 b8 00 00 00 00 00 fc ff df <48> 01 c3 c7 03 00 00 00 00 c7 43 08 00 00 00 00 48 8b0
[ 3765.268657][    C0] RSP: 0018:ffffc9000006f9f8 EFLAGS: 00000246
[ 3765.274587][    C0] RAX: dffffc0000000000 RBX: 1ffff9200000df42 RCX: 1ffff9200000df28
[ 3765.282420][    C0] RDX: 1ffff11020645769 RSI: 00000000ffffffff RDI: 0000000000000001
[ 3765.290256][    C0] RBP: 0000000000000001 R08: fffffbfff164cb10 R09: fffffbfff164cb10
[ 3765.298090][    C0] R10: 0000000000000246 R11: fffffbfff164cb0f R12: ffff88812be555b0
[ 3765.305922][    C0] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3765.313750][    C0] FS:  00007f12bb8c59c0(0000) GS:ffff8881b7000000(0000) knlGS:0000000000000000
[ 3765.322537][    C0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3765.328985][    C0] CR2: 00007f0c2d828fd0 CR3: 000000011868a000 CR4: 00000000003506f0
[ 3765.336820][    C0] Call Trace:
[ 3765.339979][    C0]  ? rcu_read_unlock+0x40/0x40
[ 3765.344609][    C0]  __d_move+0x2a2/0x16f0
__seqprop_spinlock_assert at include/linux/seqlock.h:277
(inlined by) __d_move at fs/dcache.c:2861
[ 3765.348711][    C0]  ? d_move+0x47/0x70
[ 3765.352560][    C0]  ? _raw_spin_unlock+0x1a/0x30
[ 3765.357275][    C0]  d_move+0x47/0x70
write_seqcount_t_end at include/linux/seqlock.h:560
(inlined by) write_sequnlock at include/linux/seqlock.h:901
(inlined by) d_move at fs/dcache.c:2916
[ 3765.360951][    C0]  ? vfs_rename+0x9ac/0x1270
[ 3765.365402][    C0]  vfs_rename+0x9ac/0x1270
vfs_rename at fs/namei.c:4330
[ 3765.369687][    C0]  ? vfs_link+0x830/0x830
[ 3765.373878][    C0]  ? do_renameat2+0x58c/0x930
[ 3765.378419][    C0]  do_renameat2+0x58c/0x930
do_renameat2 at fs/namei.c:4455
[ 3765.382790][    C0]  ? __ia32_sys_link+0x80/0x80
[ 3765.387418][    C0]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 3765.392563][    C0]  ? rcu_read_lock_sched_held+0x9c/0xd0
[ 3765.397965][    C0]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 3765.403112][    C0]  ? lockdep_hardirqs_on_prepare+0x27c/0x3d0
[ 3765.408955][    C0]  ? trace_hardirqs_on+0x1c/0x150
[ 3765.413841][    C0]  ? asm_common_interrupt+0x1e/0x40
[ 3765.418905][    C0]  ? strncpy_from_user+0x14c/0x330
[ 3765.423878][    C0]  ? strncpy_from_user+0x6b/0x330
[ 3765.428764][    C0]  ? getname_flags+0x88/0x3e0
[ 3765.433307][    C0]  __x64_sys_rename+0x75/0x90
__x64_sys_rename at fs/namei.c:4504
[ 3765.437848][    C0]  do_syscall_64+0x33/0x40
[ 3765.442129][    C0]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3765.447882][    C0] RIP: 0033:0x7f12b9b759eb
[ 3765.452166][    C0] Code: e8 5a 0a 08 00 85 c0 0f 95 c0 0f b6 c0 f7 d8 5b c3 66 0f 1f 44 00 00 b8 ff ff ff ff 5b c3 90 f3 0f 1e fa b8 52 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 69 e48
[ 3765.471611][    C0] RSP: 002b:00007fff76252538 EFLAGS: 00000246 ORIG_RAX: 0000000000000052
[ 3765.479880][    C0] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f12b9b759eb
[ 3765.487715][    C0] RDX: 000055ea2997f400 RSI: 00007fff76252570 RDI: 000055ea2997f450
[ 3765.495548][    C0] RBP: 00007fff76252570 R08: 0000000000000000 R09: 0000000065732e32
[ 3765.503374][    C0] R10: 0000000000000003 R11: 0000000000000246 R12: 000055ea2996fcf8
[ 3765.511201][    C0] R13: 000055ea2996fcf8 R14: 000055ea277097f0 R15: 0000000000000000
[ 3765.939428][   C14] watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [systemd-udevd:976]
[ 3765.947707][   C14] Modules linked in: acpi_cpufreq(+) ip_tables x_tables sd_mod ahci libahci tg3 bnxt_en megaraid_sas libata firmware_class libphy dm_mirror dm_region_hash dm_log dm_mod
[ 3765.964357][   C14] irq event stamp: 535074
[ 3765.968564][   C14] hardirqs last  enabled at (535073): [<ffffffff82800c42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 3765.979266][   C14] hardirqs last disabled at (535074): [<ffffffff8277e45a>] sysvec_apic_timer_interrupt+0xa/0xa0
[ 3765.989532][   C14] softirqs last  enabled at (535072): [<ffffffff82a0061b>] __do_softirq+0x61b/0x95d
[ 3765.998761][   C14] softirqs last disabled at (535067): [<ffffffff82800ec2>] asm_call_irq_on_stack+0x12/0x20
[ 3766.008590][   C14] CPU: 14 PID: 976 Comm: systemd-udevd Tainted: G             L    5.10.0-rc2-next-20201109+ #2
[ 3766.018851][   C14] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.9.3 06/25/2019
[ 3766.027130][   C14] RIP: 0010:path_init+0x1dd/0x1380
__seqprop_spinlock_sequence at include/linux/seqlock.h:277
(inlined by) path_init at fs/namei.c:2219
[ 3766.032112][   C14] Code: 03 0f 8e 79 11 00 00 44 8b 3d 5f f0 6e 01 41 f6 c7 01 74 2e 48 b8 00 00 00 00 00 fc ff df 48 89 d1 48 c1 e9 03 48 01 c1 f3 90 <0f> b6 01 84 c0 74 08 3c 03 0f 8e 9a 0d 00 00 44 8b 3a1
[ 3766.051565][   C14] RSP: 0018:ffffc90006bdfa78 EFLAGS: 00000202
[ 3766.057493][   C14] RAX: 0000000000000000 RBX: ffffc90006bdfcc0 RCX: fffffbfff06426b0
[ 3766.065330][   C14] RDX: ffffffff83213580 RSI: 00000000d93ef353 RDI: ffffc90006bdfd00
[ 3766.073163][   C14] RBP: ffffc90006bdfb08 R08: fffffbfff164cb0d R09: fffffbfff164cb0d
[ 3766.080997][   C14] R10: 0000000000000246 R11: fffffbfff164cb0c R12: ffff8881159d0060
[ 3766.088831][   C14] R13: ffffc90006bdfcf8 R14: 0000000000000041 R15: 000000000000039d
[ 3766.096665][   C14] FS:  00007f0c2ec45980(0000) GS:ffff88866dec0000(0000) knlGS:0000000000000000
[ 3766.105454][   C14] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3766.111900][   C14] CR2: 00007f79350ceb84 CR3: 0000000160a84000 CR4: 00000000003506e0
[ 3766.119737][   C14] Call Trace:
[ 3766.122899][   C14]  ? debug_mutex_init+0x31/0x60
[ 3766.127614][   C14]  path_openat+0x154/0x1b00
path_openat at fs/namei.c:3364
[ 3766.131988][   C14]  ? rcu_read_lock_held+0xb0/0xb0
[ 3766.136881][   C14]  ? __lock_acquire+0x1dad/0x3890
[ 3766.141767][   C14]  ? path_parentat.isra.62+0x190/0x190
[ 3766.147090][   C14]  ? rcu_read_lock_sched_held+0x9c/0xd0
[ 3766.152496][   C14]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 3766.157646][   C14]  do_filp_open+0x171/0x240
[ 3766.162017][   C14]  ? may_open_dev+0xc0/0xc0
[ 3766.166388][   C14]  ? lock_downgrade+0x700/0x700
[ 3766.171104][   C14]  ? rcu_read_unlock+0x40/0x40
[ 3766.175735][   C14]  ? rwlock_bug.part.1+0x90/0x90
[ 3766.180543][   C14]  ? __check_object_size+0x454/0x670
[ 3766.185692][   C14]  ? do_raw_spin_unlock+0x4f/0x250
[ 3766.190668][   C14]  ? __alloc_fd+0x184/0x500
[ 3766.195031][   C14]  do_sys_openat2+0x2db/0x5b0
[ 3766.199576][   C14]  ? memset+0x1f/0x40
[ 3766.203422][   C14]  ? file_open_root+0x200/0x200
[ 3766.208136][   C14]  do_sys_open+0x85/0xd0
[ 3766.212245][   C14]  ? filp_open+0x50/0x50
[ 3766.216353][   C14]  ? syscall_trace_enter.isra.21+0x54/0x1f0
[ 3766.222107][   C14]  do_syscall_64+0x33/0x40
[ 3766.226389][   C14]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3766.232145][   C14] RIP: 0033:0x7f0c2d828552
[ 3766.236426][   C14] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 15 50 2d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 645
[ 3766.255873][   C14] RSP: 002b:00007ffc65d36000 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[ 3766.264147][   C14] RAX: ffffffffffffffda RBX: 0000564d81e50770 RCX: 00007f0c2d828552
[ 3766.271982][   C14] RDX: 0000000000080000 RSI: 00007ffc65d36150 RDI: 00000000ffffff9c
[ 3766.279817][   C14] RBP: 0000000000000008 R08: 0000000000000008 R09: 0000000000000001
[ 3766.287651][   C14] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f0c2e733347
[ 3766.295486][   C14] R13: 00007f0c2e733347 R14: 0000000000000001 R15: 0000564d81e2d1b8
[ 3766.469430][   C28] watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [mount:1058]
[ 3766.477105][   C28] Modules linked in: acpi_cpufreq(+) ip_tables x_tables sd_mod ahci libahci tg3 bnxt_en megaraid_sas libata firmware_class libphy dm_mirror dm_region_hash dm_log dm_mod
[ 3766.493753][   C28] irq event stamp: 88756
[ 3766.497868][   C28] hardirqs last  enabled at (88755): [<ffffffff82800c42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 3766.508481][   C28] hardirqs last disabled at (88756): [<ffffffff8277e45a>] sysvec_apic_timer_interrupt+0xa/0xa0
[ 3766.518665][   C28] softirqs last  enabled at (88748): [<ffffffff82a0061b>] __do_softirq+0x61b/0x95d
[ 3766.527805][   C28] softirqs last disabled at (88743): [<ffffffff82800ec2>] asm_call_irq_on_stack+0x12/0x20
[ 3766.537550][   C28] CPU: 28 PID: 1058 Comm: mount Tainted: G             L    5.10.0-rc2-next-20201109+ #2
[ 3766.547210][   C28] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.9.3 06/25/2019
[ 3766.555482][   C28] RIP: 0010:prepend_path.isra.6+0x349/0xdb0
__seqprop_spinlock_sequence at include/linux/seqlock.h:277
(inlined by) read_seqbegin at include/linux/seqlock.h:838
(inlined by) read_seqbegin_or_lock at include/linux/seqlock.h:1142
(inlined by) read_seqbegin_or_lock at include/linux/seqlock.h:1139
(inlined by) prepend_path at fs/d_path.c:99
[ 3766.561241][   C28] Code: 95 f0 fe ff ff 5a 41 54 9d 41 0f b6 55 00 84 d2 74 09 80 fa 03 0f 8e 75 09 00 00 45 8b 26 41 f6 c4 01 0f 84 5b 05 00 00 f3 90 <41> 0f b6 55 00 84 d2 74 e8 80 fa 03 7f e3 4c 89 f7 4ce
[ 3766.580687][   C28] RSP: 0018:ffffc9000670fa18 EFLAGS: 00000202
[ 3766.586618][   C28] RAX: ffff88812bd7d070 RBX: dffffc0000000000 RCX: 1ffff1102a164dc8
[ 3766.594449][   C28] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81bafabb
[ 3766.602287][   C28] RBP: ffffc9000670fb68 R08: 0000000000000001 R09: ffffffff81bb033d
[ 3766.610118][   C28] R10: ffff8881517d8340 R11: fffffbfff14b86e4 R12: 000000000000039d
[ 3766.617956][   C28] R13: fffffbfff06426b0 R14: ffffffff83213580 R15: ffffc9000670fbc0
[ 3766.625789][   C28] FS:  00007f96bb7a6080(0000) GS:ffff8881b71c0000(0000) knlGS:0000000000000000
[ 3766.634579][   C28] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3766.641026][   C28] CR2: 00007f96ba7343f0 CR3: 0000000146c76000 CR4: 00000000003506e0
[ 3766.648859][   C28] Call Trace:
[ 3766.652016][   C28]  ? dentry_path_raw+0x10/0x10
[ 3766.656645][   C28]  ? rcu_read_unlock+0x40/0x40
[ 3766.661276][   C28]  ? memcpy+0x38/0x60
[ 3766.665128][   C28]  d_path+0x3fd/0x660
[ 3766.668976][   C28]  ? prepend_path.isra.6+0xdb0/0xdb0
[ 3766.674129][   C28]  ? __alloc_pages_nodemask+0x650/0x760
[ 3766.679543][   C28]  ? __alloc_pages_slowpath.constprop.112+0x2120/0x2120
[ 3766.686345][   C28]  mnt_warn_timestamp_expiry+0x25a/0x270
mnt_warn_timestamp_expiry at fs/namespace.c:2555 (discriminator 1)
[ 3766.691839][   C28]  ? get_mountpoint+0x370/0x370
[ 3766.696554][   C28]  ? do_raw_spin_unlock+0x4f/0x250
[ 3766.701536][   C28]  ? _raw_spin_unlock+0x1a/0x30
[ 3766.706253][   C28]  ? vfs_create_mount+0x371/0x4a0
[ 3766.711150][   C28]  path_mount+0xe6b/0x1720
do_new_mount_fc at fs/namespace.c:2839
(inlined by) do_new_mount at fs/namespace.c:2898
(inlined by) path_mount at fs/namespace.c:3227
[ 3766.715438][   C28]  ? finish_automount+0x750/0x750
[ 3766.720328][   C28]  ? getname_flags+0x88/0x3e0
[ 3766.724877][   C28]  do_mount+0xc6/0xe0
[ 3766.728726][   C28]  ? path_mount+0x1720/0x1720
[ 3766.733275][   C28]  ? _copy_from_user+0xab/0xf0
[ 3766.737906][   C28]  ? memdup_user+0x4f/0x80
[ 3766.742193][   C28]  __x64_sys_mount+0x15d/0x1b0
[ 3766.746828][   C28]  do_syscall_64+0x33/0x40
[ 3766.751111][   C28]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3914.819439][   C36] watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [systemd-journal:902]
[ 3914.827884][   C36] Modules linked in: acpi_cpufreq(+) ip_tables x_tables sd_mod ahci libahci tg3 bnxt_en megaraid_sas libata firmware_class libphy dm_mirror dm_region_hash dm_log dm_mod
[ 3914.844543][   C36] irq event stamp: 196688
[ 3914.848748][   C36] hardirqs last  enabled at (196687): [<ffffffff82800c42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 3914.859450][   C36] hardirqs last disabled at (196688): [<ffffffff8277e45a>] sysvec_apic_timer_interrupt+0xa/0xa0
[ 3914.869709][   C36] softirqs last  enabled at (196684): [<ffffffff82a0061b>] __do_softirq+0x61b/0x95d
[ 3914.878930][   C36] softirqs last disabled at (196679): [<ffffffff82800ec2>] asm_call_irq_on_stack+0x12/0x20
[ 3914.888759][   C36] CPU: 36 PID: 902 Comm: systemd-journal Tainted: G             L    5.10.0-rc2-next-20201109+ #2
[ 3914.899195][   C36] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.9.3 06/25/2019
[ 3914.907471][   C36] RIP: 0010:path_init+0x1f3/0x1380
[ 3914.912448][   C36] Code: 00 00 00 00 00 fc ff df 48 89 d1 48 c1 e9 03 48 01 c1 f3 90 0f b6 01 84 c0 74 08 3c 03 0f 8e 9a 0d 00 00 44 8b 3a 41 f6 c7 01 <75> e6 48 b8 00 00 00 00 00 fc ff df 48 8d 7b 44 48 89a
[ 3914.931901][   C36] RSP: 0018:ffffc900065dfc08 EFLAGS: 00000202
[ 3914.937837][   C36] RAX: 0000000000000000 RBX: ffffc900065dfd40 RCX: fffffbfff06426b0
[ 3914.945675][   C36] RDX: ffffffff83213580 RSI: 00000000d93ef353 RDI: ffffc900065dfd80
[ 3914.953514][   C36] RBP: ffffc900065dfc98 R08: fffffbfff164cb0d R09: fffffbfff164cb0d
[ 3914.961349][   C36] R10: 0000000000000246 R11: fffffbfff164cb0c R12: ffff8881159d5920
[ 3914.969186][   C36] R13: ffffc900065dfd78 R14: 0000000000000041 R15: 000000000000039d
[ 3914.977028][   C36] FS:  00007fc5f95f9980(0000) GS:ffff8881b7240000(0000) knlGS:0000000000000000
[ 3914.985814][   C36] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3914.992262][   C36] CR2: 000055ea2994ba42 CR3: 000000013702a000 CR4: 00000000003506e0
[ 3915.000097][   C36] Call Trace:
[ 3915.003257][   C36]  path_lookupat.isra.61+0x20/0x440
[ 3915.008323][   C36]  ? find_held_lock+0x33/0x1c0
[ 3915.012954][   C36]  filename_lookup.part.73+0x15c/0x290
[ 3915.018283][   C36]  ? __might_fault+0xb5/0x160
[ 3915.022831][   C36]  ? do_symlinkat+0x190/0x190
[ 3915.027375][   C36]  ? strncpy_from_user+0x6b/0x330
[ 3915.032268][   C36]  ? getname_flags+0x88/0x3e0
[ 3915.036809][   C36]  ? kmem_cache_alloc+0x244/0x270
[ 3915.041701][   C36]  do_faccessat+0xc0/0x5a0
[ 3915.045987][   C36]  ? stream_open+0x60/0x60
[ 3915.050269][   C36]  do_syscall_64+0x33/0x40
[ 3915.054552][   C36]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3915.060306][   C36] RIP: 0033:0x7fc5f862697b
[ 3915.064590][   C36] Code: 77 05 c3 0f 1f 40 00 48 8b 15 09 f5 2c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff c3 0f 1f 40 00 f3 0f 1e fa b8 15 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 d9 f48
[ 3915.084044][   C36] RSP: 002b:00007ffdefba1198 EFLAGS: 00000246 ORIG_RAX: 0000000000000015
[ 3915.092322][   C36] RAX: ffffffffffffffda RBX: 00007ffdefba3cf0 RCX: 00007fc5f862697b
[ 3915.100163][   C36] RDX: 00007fc5f91032e8 RSI: 0000000000000000 RDI: 0000556a2401c0d0
[ 3915.107996][   C36] RBP: 00007ffdefba11e0 R08: 0000000000000000 R09: 0000000000000000
[ 3915.115832][   C36] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 3915.123667][   C36] R13: 0000000000000000 R14: 0000000000000009 R15: 0000556a254de950
[ 3917.159439][    C0] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [systemd:1]
[ 3917.166926][    C0] Modules linked in: acpi_cpufreq(+) ip_tables x_tables sd_mod ahci libahci tg3 bnxt_en megaraid_sas libata firmware_class libphy dm_mirror dm_region_hash dm_log dm_mod
[ 3917.183572][    C0] irq event stamp: 94238206
[ 3917.187944][    C0] hardirqs last  enabled at (94238205): [<ffffffff82800b9e>] asm_common_interrupt+0x1e/0x40
[ 3917.197857][    C0] hardirqs last disabled at (94238206): [<ffffffff8277e45a>] sysvec_apic_timer_interrupt+0xa/0xa0
[ 3917.208289][    C0] softirqs last  enabled at (94219940): [<ffffffff82a0061b>] __do_softirq+0x61b/0x95d
[ 3917.217685][    C0] softirqs last disabled at (94219881): [<ffffffff82800ec2>] asm_call_irq_on_stack+0x12/0x20
[ 3917.227688][    C0] CPU: 0 PID: 1 Comm: systemd Tainted: G             L    5.10.0-rc2-next-20201109+ #2
[ 3917.237167][    C0] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.9.3 06/25/2019
[ 3917.245437][    C0] RIP: 0010:lock_acquire+0x21b/0x820
[ 3917.250585][    C0] Code: fc ff df 48 01 c3 c7 03 00 00 00 00 c7 43 08 00 00 00 00 48 8b 84 24 90 00 00 00 65 48 33 04 25 28 00 00 00 0f 85 1a 05 00 00 <48> 81 c4 98 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3c
[ 3917.270033][    C0] RSP: 0018:ffffc9000006f9f8 EFLAGS: 00000246
[ 3917.275959][    C0] RAX: 0000000000000000 RBX: fffff5200000df42 RCX: 1ffff9200000df28
[ 3917.283795][    C0] RDX: 1ffff11020645769 RSI: 00000000ffffffff RDI: 0000000000000001
[ 3917.291627][    C0] RBP: 0000000000000001 R08: fffffbfff164cb10 R09: fffffbfff164cb10
[ 3917.299463][    C0] R10: 0000000000000246 R11: fffffbfff164cb0f R12: ffff88812be555b0
[ 3917.307297][    C0] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3917.315133][    C0] FS:  00007f12bb8c59c0(0000) GS:ffff8881b7000000(0000) knlGS:0000000000000000
[ 3917.323921][    C0] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3917.330368][    C0] CR2: 00007f0c2d828fd0 CR3: 000000011868a000 CR4: 00000000003506f0
[ 3917.338202][    C0] Call Trace:
[ 3917.341360][    C0]  ? rcu_read_unlock+0x40/0x40
[ 3917.345987][    C0]  __d_move+0x2a2/0x16f0
[ 3917.350096][    C0]  ? d_move+0x47/0x70
[ 3917.353944][    C0]  ? _raw_spin_unlock+0x1a/0x30
[ 3917.358656][    C0]  d_move+0x47/0x70
[ 3917.362330][    C0]  ? vfs_rename+0x9ac/0x1270
[ 3917.366786][    C0]  vfs_rename+0x9ac/0x1270
[ 3917.371068][    C0]  ? vfs_link+0x830/0x830
[ 3917.375261][    C0]  ? do_renameat2+0x58c/0x930
[ 3917.379804][    C0]  do_renameat2+0x58c/0x930
[ 3917.384173][    C0]  ? __ia32_sys_link+0x80/0x80
[ 3917.388800][    C0]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 3917.393948][    C0]  ? rcu_read_lock_sched_held+0x9c/0xd0
[ 3917.399354][    C0]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 3917.404503][    C0]  ? lockdep_hardirqs_on_prepare+0x27c/0x3d0
[ 3917.410346][    C0]  ? trace_hardirqs_on+0x1c/0x150
[ 3917.415234][    C0]  ? asm_common_interrupt+0x1e/0x40
[ 3917.420294][    C0]  ? strncpy_from_user+0x14c/0x330
[ 3917.425269][    C0]  ? strncpy_from_user+0x6b/0x330
[ 3917.430158][    C0]  ? getname_flags+0x88/0x3e0
[ 3917.434699][    C0]  __x64_sys_rename+0x75/0x90
[ 3917.439239][    C0]  do_syscall_64+0x33/0x40
[ 3917.443520][    C0]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 3917.449274][    C0] RIP: 0033:0x7f12b9b759eb
[ 3917.453557][    C0] Code: e8 5a 0a 08 00 85 c0 0f 95 c0 0f b6 c0 f7 d8 5b c3 66 0f 1f 44 00 00 b8 ff ff ff ff 5b c3 90 f3 0f 1e fa b8 52 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 69 e48
[ 3917.473003][    C0] RSP: 002b:00007fff76252538 EFLAGS: 00000246 ORIG_RAX: 0000000000000052
[ 3917.481272][    C0] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f12b9b759eb
[ 3917.489107][    C0] RDX: 000055ea2997f400 RSI: 00007fff76252570 RDI: 000055ea2997f450
[ 3917.496942][    C0] RBP: 00007fff76252570 R08: 0000000000000000 R09: 0000000065732e32
[ 3917.504776][    C0] R10: 0000000000000003 R11: 0000000000000246 R12: 000055ea2996fcf8
[ 3917.512610][    C0] R13: 000055ea2996fcf8 R14: 000055ea277097f0 R15: 0000000000000000

[ 3634.443982][   C37] NMI backtrace for cpu 37
[ 3634.443984][   C37] CPU: 37 PID: 906 Comm: udevadm Tainted: G             L    5.10.0-rc2-next-20201109+ #3
[ 3634.443986][   C37] Hardware name: Dell Inc. PowerEdge R6415/07YXFK, BIOS 1.9.3 06/25/2019
[ 3634.443987][   C37] RIP: 0010:io_serial_in+0x5c/0x90
[ 3634.443989][   C37] Code: 48 8d 7b 40 0f b6 8b f1 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 d3 e6 80 3c 02 00 75 1e 03 73 40 89 f2 ec <48> 83 c4 08 0f b6 c0 5b c3 89 74 24 04 e8 d2 90 81 ff4
[ 3634.443991][   C37] RSP: 0018:ffffc90004a38780 EFLAGS: 00000002
[ 3634.443993][   C37] RAX: dffffc0000000000 RBX: ffffffffa0b1c988 RCX: 0000000000000000
[ 3634.443995][   C37] RDX: 00000000000002fd RSI: 00000000000002fd RDI: ffffffffa0b1c9c8
[ 3634.443996][   C37] RBP: 0000000000000020 R08: 0000000000000004 R09: fffff520009470f3
[ 3634.443998][   C37] R10: 0000000000000003 R11: fffff520009470f3 R12: fffffbfff4163984
[ 3634.443999][   C37] R13: fffffbfff416393b R14: ffffffffa0b1c9d8 R15: ffffffffa0b1c988
[ 3634.444001][   C37] FS:  00007f6947084980(0000) GS:ffff88866e440000(0000) knlGS:0000000000000000
[ 3634.444002][   C37] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3634.444004][   C37] CR2: 00007fd220b94816 CR3: 000000015371e000 CR4: 00000000003506e0
[ 3634.444005][   C37] Call Trace:
[ 3634.444006][   C37]  <IRQ>
[ 3634.444007][   C37]  wait_for_xmitr+0x7c/0x1b0
[ 3634.444008][   C37]  ? wait_for_xmitr+0x1b0/0x1b0
[ 3634.444010][   C37]  serial8250_console_putchar+0x11/0x50
[ 3634.444011][   C37]  uart_console_write+0x39/0xc0
[ 3634.444012][   C37]  serial8250_console_write+0x74c/0x8a0
[ 3634.444013][   C37]  ? serial8250_start_tx+0x810/0x810
[ 3634.444015][   C37]  ? rcu_read_unlock+0x40/0x40
[ 3634.444016][   C37]  ? do_raw_spin_lock+0x121/0x290
[ 3634.444017][   C37]  ? rwlock_bug.part.1+0x90/0x90
[ 3634.444018][   C37]  ? prb_read_valid+0x61/0x90
[ 3634.444020][   C37]  ? prb_final_commit+0x10/0x10
[ 3634.444021][   C37]  console_unlock+0x721/0xa20
[ 3634.444022][   C37]  ? do_syslog.part.22+0x5d0/0x5d0
[ 3634.444023][   C37]  ? lock_acquire+0x1c8/0x820
[ 3634.444024][   C37]  ? lock_downgrade+0x700/0x700
[ 3634.444026][   C37]  vprintk_emit+0x201/0x2c0
[ 3634.444027][   C37]  printk+0x9a/0xc0
[ 3634.444028][   C37]  ? record_print_text.cold.38+0x11/0x11
[ 3634.444029][   C37]  ? __module_text_address+0xe/0x140
[ 3634.444031][   C37]  ? nmi_cpu_backtrace.cold.10+0x18/0xc5
[ 3634.444032][   C37]  show_trace_log_lvl+0x228/0x2bb
[ 3634.444033][   C37]  ? nmi_cpu_backtrace.cold.10+0x18/0xc5
[ 3634.444035][   C37]  ? nmi_cpu_backtrace.cold.10+0x18/0xc5
[ 3634.444036][   C37]  ? lapic_can_unplug_cpu+0x11/0x70
[ 3634.444037][   C37]  dump_stack+0x99/0xcb
[ 3634.444038][   C37]  nmi_cpu_backtrace.cold.10+0x18/0xc5
[ 3634.444040][   C37]  ? lapic_can_unplug_cpu+0x70/0x70
[ 3634.444041][   C37]  nmi_trigger_cpumask_backtrace+0x232/0x2a0
[ 3634.444042][   C37]  rcu_dump_cpu_stacks+0x1ad/0x1f9
[ 3634.444043][   C37]  rcu_sched_clock_irq.cold.90+0x4f7/0x968
[ 3634.444045][   C37]  ? tick_sched_do_timer+0x140/0x140
[ 3634.444046][   C37]  update_process_times+0x75/0xb0
[ 3634.444047][   C37]  tick_sched_timer+0xc8/0xf0
[ 3634.444048][   C37]  __hrtimer_run_queues+0x529/0xbb0
[ 3634.444050][   C37]  ? enqueue_hrtimer+0x330/0x330
[ 3634.444051][   C37]  ? ktime_get_update_offsets_now+0xd6/0x2b0
[ 3634.444052][   C37]  hrtimer_interrupt+0x2c2/0x750
[ 3634.444054][   C37]  __sysvec_apic_timer_interrupt+0x13c/0x520
[ 3634.444055][   C37]  asm_call_irq_on_stack+0x12/0x20
[ 3634.444056][   C37]  </IRQ>
[ 3634.444057][   C37]  sysvec_apic_timer_interrupt+0x85/0xa0
[ 3634.444059][   C37]  asm_sysvec_apic_timer_interrupt+0x12/0x20