Subject: Re: Bug inkvm_set_irq

From: "Michael S. Tsirkin" <[email-masked]>
Date: Fri, 4 Mar 2011 11:35:00 +0200

On Fri, Mar 04, 2011 at 10:22:03AM +0100, Jean-Philippe Menil wrote:
> Le 03/03/2011 16:55, Michael S. Tsirkin a écrit :
> >On Thu, Mar 03, 2011 at 04:26:11PM +0100, Jean-Philippe Menil wrote:
> >>Le 03/03/2011 15:47, Michael S. Tsirkin a écrit :
> >>>On Tue, Mar 01, 2011 at 03:39:12PM +0100, Jean-Philippe Menil wrote:
> >>>>so this time the bug is:
> >>>>
> >>>>[17882.612303] BUG: unable to handle kernel paging request at
> >>>>0000000000002458
> >>>>[17882.612342] IP: [<ffffffffa03898a0>] kvm_set_irq+0x30/0x140 [kvm]
> >>>>
> >>>>markup_oops give me this:
> >>>>
> >>>>root@ayrshire:~# cat bug-0103.txt | perl markup_oops.pl -m
> >>>>/lib/modules/2.6.37.2-dsiun-110105+/kernel/arch/x86/kvm/kvm.ko
> >>>>/boot/vmlinuz-2.6.37.2-dsiun-110105+
> >>>>vmaoffset = 18446744072102621184 ffffffffa0389871:	48 89 e5   	mov
> >>>>%rsp,%rbp
> >>>>  ffffffffa0389874:	41 57                	push   %r15
> >>>>  ffffffffa0389876:	41 89 cf             	mov    %ecx,%r15d  |  %r15
> >>>>=>   1  %ecx = 1
> >>>>  ffffffffa0389879:	41 56                	push   %r14        |  %r14
> >>>>=>   ffffffffa038aad0
> >>>>  ffffffffa038987b:	41 55                	push   %r13
> >>>>  ffffffffa038987d:	49 89 fd             	mov    %rdi,%r13   |  %edi
> >>>>= 0  %r13 =>   0
> >>>>  ffffffffa0389880:	41 54                	push   %r12        |  %r12 =>   0
> >>>>  ffffffffa0389882:	53                   	push   %rbx
> >>>>  ffffffffa0389883:	89 d3                	mov    %edx,%ebx   |  %ebx =>   1a
> >>>>  ffffffffa0389885:	48 81 ec a8 00 00 00 	sub    $0xa8,%rsp
> >>>>  ffffffffa038988c:	8b 15 00 00 00 00    	mov    0x0(%rip),%edx
> >>>># ffffffffa0389892<kvm_set_irq+0x22>
> >>>>  ffffffffa0389892:	89 b5 3c ff ff ff    	mov    %esi,-0xc4(%rbp) |
> >>>>%esi = 0
> >>>>  ffffffffa0389898:	85 d2                	test   %edx,%edx   |  %edx =>   0
> >>>>  ffffffffa038989a:	0f 85 d5 00 00 00    	jne    ffffffffa0389975
> >>>><kvm_set_irq+0x105>
> >>>>*ffffffffa03898a0:	49 8b 85 58 24 00 00 	mov    0x2458(%r13),%rax |
> >>>>%eax = 0  %r13 = 0<--- faulting instruction
> >>>>  ffffffffa03898a7:	3b 98 28 01 00 00    	cmp    0x128(%rax),%ebx
> >>>>  ffffffffa03898ad:	73 61                	jae    ffffffffa0389910
> >>>><kvm_set_irq+0xa0>
> >>>>  ffffffffa03898af:	89 db                	mov    %ebx,%ebx
> >>>>  ffffffffa03898b1:	48 8b 84 d8 30 01 00 	mov    0x130(%rax,%rbx,8),%rax
> >>>>  ffffffffa03898b8:	00
> >>>>  ffffffffa03898b9:	48 85 c0             	test   %rax,%rax
> >>>>  ffffffffa03898bc:	74 52                	je     ffffffffa0389910
> >>>><kvm_set_irq+0xa0>
> >>>>  ffffffffa03898be:	48 8d 95 40 ff ff ff 	lea    -0xc0(%rbp),%rdx
> >>>>  ffffffffa03898c5:	31 db                	xor    %ebx,%ebx
> >>>>  ffffffffa03898c7:	48 8b 08             	mov    (%rax),%rcx
> >>>>  ffffffffa03898ca:	83 c3 01             	add    $0x1,%ebx
> >>>>  ffffffffa03898cd:	0f 18 09             	prefetcht0 (%rcx)
> >>>>  ffffffffa03898d0:	48 8b 48 e0          	mov    -0x20(%rax),%rcx
> >>>>  ffffffffa03898d4:	48 89 0a             	mov    %rcx,(%rdx)
> >>>>  ffffffffa03898d7:	48 8b 48 e8          	mov    -0x18(%rax),%rcx
> >>>>  ffffffffa03898db:	48 89 4a 08          	mov    %rcx,0x8(%rdx)
> >>>>  ffffffffa03898df:	48 8b 48 f0          	mov    -0x10(%rax),%rcx
> >>>>  ffffffffa03898e3:	48 89 4a 10          	mov    %rcx,0x10(%rdx)
> >>>>  ffffffffa03898e7:	48 8b 48 f8          	mov    -0x8(%rax),%rcx
> >>>>  ffffffffa03898eb:	48 89 4a 18          	mov    %rcx,0x18(%rdx)
> >>>>
> >>>>wich correspond to offset 68a0 (from objdump):
> >>>>
> >>>>kvm_set_irq():
> >>>>/usr/src/GIT/linux-2.6-stable/arch/x86/kvm/../../../virt/kvm/irq_comm.c:161
> >>>>     68a0:       49 8b 85 58 24 00 00    mov    0x2458(%r13),%rax
> >>>>/usr/src/GIT/linux-2.6-stable/arch/x86/kvm/../../../virt/kvm/irq_comm.c:162
> >>>>     68a7:       3b 98 28 01 00 00       cmp    0x128(%rax),%ebx
> >>>>
> >>>>root@ayrshire:~# addr2line -e
> >>>>/lib/modules/2.6.37.2-dsiun-110105+/kernel/arch/x86/kvm/kvm.ko
> >>>>0x68a0
> >>>>/usr/src/GIT/linux-2.6-stable/arch/x86/kvm/../../../virt/kvm/irq_comm.c:161
> >>>>
> >>>>So here kvm->irq_routing is null.
> >>>>
> >>>>How can it be?
> >>>>
> >>>>Regards.
> >>>Not null, this seems to be invalid.
> >>>I suspect use after free where the kvm pointer is
> >>>pointing at some random memory. Use after free?
> >>>Could you please try enabling a slab debugger,
> >>>recompile and rerun the test?
> >>>
> >>Hi,
> >>
> >>I'm not sure to activate the right thing.
> >>Is that what you want?
> >>
> >>CONFIG_SLAB=y
> >>CONFIG_SLABINFO=y
> >>CONFIG_DEBUG_SLAB=y
> >>CONFIG_DEBUG_SLAB_LEAK=y
> >>
> >>Regards.
> >Yes, maybe disable SLAB_LEAK.
> >
> >>-- 
> >>Jean-Philippe Menil - Pôle réseau Service IRTS
> >>DSI Université de Nantes
> >>[email-masked]
> >>Tel : 02.53.48.49.27 - Fax : 02.53.48.49.09
> >--
> >To unsubscribe from this list: send the line "unsubscribe kvm" in
> >the body of a message to [email-masked]
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Hi,
> 
> so this time, here is what markup_oops says:
> 
> root@ayrshire:~# cat oops-0403.txt | perl markup_oops.pl -m
> /lib/modules/2.6.37.2.999-dsiun-110105+/kernel/arch/x86/kvm/kvm.ko
> /boot/vmlinuz-2.6.37.2.999-dsiun-110105+
> vmaoffset = 18446744072102948864 ffffffffa03d9811:    48 89 e5
> mov    %rsp,%rbp
>  ffffffffa03d9814:    41 57                    push   %r15
>  ffffffffa03d9816:    41 89 cf                 mov    %ecx,%r15d  |
> %r15 => 1  %ecx = 1
>  ffffffffa03d9819:    41 56                    push   %r14        |
> %r14 => ffffffffa03daa50
>  ffffffffa03d981b:    41 55                    push   %r13
>  ffffffffa03d981d:    49 89 fd                 mov    %rdi,%r13   |
> %edi = 6b6b6b6b6b6b6b6b  %r13 => 6b6b6b6b6b6b6b6b
>  ffffffffa03d9820:    41 54                    push   %r12        |
> %r12 => 6b6b6b6b6b6b6b6b
>  ffffffffa03d9822:    53                       push   %rbx
>  ffffffffa03d9823:    89 d3                    mov    %edx,%ebx   |
> %ebx => 6b6b6b6b
>  ffffffffa03d9825:    48 81 ec a8 00 00 00     sub    $0xa8,%rsp
>  ffffffffa03d982c:    8b 15 00 00 00 00        mov    0x0(%rip),%edx
> # ffffffffa03d9832 <kvm_set_irq+0x22>
>  ffffffffa03d9832:    89 b5 3c ff ff ff        mov
> %esi,-0xc4(%rbp) |  %esi = 0
>  ffffffffa03d9838:    85 d2                    test   %edx,%edx   |
> %edx => 0
>  ffffffffa03d983a:    0f 85 d5 00 00 00        jne
> ffffffffa03d9915 <kvm_set_irq+0x105>
> *ffffffffa03d9840:    49 8b 85 58 24 00 00     mov
> 0x2458(%r13),%rax |  %eax = 0  %r13 = 6b6b6b6b6b6b6b6b <--- faulting
> instruction
>  ffffffffa03d9847:    3b 98 28 01 00 00        cmp    0x128(%rax),%ebx
>  ffffffffa03d984d:    73 61                    jae
> ffffffffa03d98b0 <kvm_set_irq+0xa0>
>  ffffffffa03d984f:    89 db                    mov    %ebx,%ebx
>  ffffffffa03d9851:    48 8b 84 d8 30 01 00     mov
> 0x130(%rax,%rbx,8),%rax
>  ffffffffa03d9858:    00
>  ffffffffa03d9859:    48 85 c0                 test   %rax,%rax
>  ffffffffa03d985c:    74 52                    je
> ffffffffa03d98b0 <kvm_set_irq+0xa0>
>  ffffffffa03d985e:    48 8d 95 40 ff ff ff     lea    -0xc0(%rbp),%rdx
>  ffffffffa03d9865:    31 db                    xor    %ebx,%ebx
>  ffffffffa03d9867:    48 8b 08                 mov    (%rax),%rcx
>  ffffffffa03d986a:    83 c3 01                 add    $0x1,%ebx
>  ffffffffa03d986d:    0f 18 09                 prefetcht0 (%rcx)
>  ffffffffa03d9870:    48 8b 48 e0              mov    -0x20(%rax),%rcx
>  ffffffffa03d9874:    48 89 0a                 mov    %rcx,(%rdx)
>  ffffffffa03d9877:    48 8b 48 e8              mov    -0x18(%rax),%rcx
>  ffffffffa03d987b:    48 89 4a 08              mov    %rcx,0x8(%rdx)
>  ffffffffa03d987f:    48 8b 48 f0              mov    -0x10(%rax),%rcx
>  ffffffffa03d9883:    48 89 4a 10              mov    %rcx,0x10(%rdx)
>  ffffffffa03d9887:    48 8b 48 f8              mov    -0x8(%rax),%rcx
>  ffffffffa03d988b:    48 89 4a 18              mov    %rcx,0x18(%rdx)
> 
> 
> Is that you wanted, the "6b6b6b6b6b6b6b6b" ?
> 
> Regards.

Yes, excellent. So now we can detect the problem by comparing
kvm with 6b6b6b6b6b6b6b6b, and print out stuff to understand where
this comes from.
I will prepare such a debugging patch.

For that, could you please tell me which kernel version, exactly, are
you using?

-- 
MST