From: Nikolay Aleksandrov <nikolay@cumulusnetworks•com>
To: Johan Schuijt <johan@transip•nl>, Eric Dumazet <eric.dumazet@gmail•com>
Cc: "nikolay@redhat•com" <nikolay@redhat•com>,
"davem@davemloft•net" <davem@davemloft•net>,
"fw@strlen•de" <fw@strlen•de>,
"chutzpah@gentoo•org" <chutzpah@gentoo•org>,
Robin Geuze <robing@transip•nl>,
Frank Schreuder <fschreuder@transip•nl>,
netdev <netdev@vger•kernel.org>
Subject: Re: reproducable panic eviction work queue
Date: Sat, 18 Jul 2015 12:02:37 +0200 [thread overview]
Message-ID: <55AA243D.5020306@cumulusnetworks.com> (raw)
In-Reply-To: <5FD5C17E-B321-404E-80A2-EE46BB8AA746@transip.nl>
On 07/18/2015 11:01 AM, Johan Schuijt wrote:
> Yes, we already found these and are included in our kernel, but even with these patches we still receive the panic.
>
> - Johan
>
>
>> On 18 Jul 2015, at 10:56, Eric Dumazet <eric.dumazet@gmail•com> wrote:
>>
>> On Fri, 2015-07-17 at 21:18 +0000, Johan Schuijt wrote:
>>> Hey guys,
>>>
>>>
>>> We’re currently running into a reproducible panic in the eviction work
>>> queue code when we pin al our eth* IRQ to different CPU cores (in
>>> order to scale our networking performance for our virtual servers).
>>> This only occurs in kernels >= 3.17 and is a result of the following
>>> change:
>>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.18.y&id=b13d3cbfb8e8a8f53930af67d1ebf05149f32c24
>>>
>>>
>>> The race/panic we see seems to be the same as, or similar to:
>>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.18.y&id=65ba1f1ec0eff1c25933468e1d238201c0c2cb29
>>>
>>>
>>> We can confirm that this is directly exposed by the IRQ pinning since
>>> disabling this stops us from being able to reproduce this case :)
>>>
>>>
>>> How te reproduce: in our test-setup we have 4 machines generating UDP
>>> packets which are send to the vulnerable host. These all have a MTU of
>>> 100 (for test purposes) and send UDP packets of a size of 256 bytes.
>>> Within half an hour you will see the following panic:
>>>
>>>
>>> crash> bt
>>> PID: 56 TASK: ffff885f3d9fc210 CPU: 9 COMMAND: "kworker/9:0"
>>> #0 [ffff885f3da03b60] machine_kexec at ffffffff8104a1f7
>>> #1 [ffff885f3da03bb0] crash_kexec at ffffffff810db187
>>> #2 [ffff885f3da03c80] oops_end at ffffffff81015140
>>> #3 [ffff885f3da03ca0] general_protection at ffffffff814f6c88
>>> [exception RIP: inet_evict_bucket+281]
>>> RIP: ffffffff81480699 RSP: ffff885f3da03d58 RFLAGS: 00010292
>>> RAX: ffff885f3da03d08 RBX: dead0000001000a8 RCX:
>>> ffff885f3da03d08
>>> RDX: 0000000000000006 RSI: ffff885f3da03ce8 RDI:
>>> dead0000001000a8
>>> RBP: 0000000000000002 R8: 0000000000000286 R9:
>>> ffff88302f401640
>>> R10: 0000000080000000 R11: ffff88602ec0c138 R12:
>>> ffffffff81a8d8c0
>>> R13: ffff885f3da03d70 R14: 0000000000000000 R15:
>>> ffff881d6efe1a00
>>> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
>>> #4 [ffff885f3da03db0] inet_frag_worker at ffffffff8148075a
>>> #5 [ffff885f3da03e10] process_one_work at ffffffff8107be19
>>> #6 [ffff885f3da03e60] worker_thread at ffffffff8107c6e3
>>> #7 [ffff885f3da03ed0] kthread at ffffffff8108103e
>>> #8 [ffff885f3da03f50] ret_from_fork at ffffffff814f4d7c
>>>
>>>
>>> We would love to receive your input on this matter.
>>>
>>>
>>> Thx in advance,
>>>
>>>
>>> - Johan
>>
>> Check commits 65ba1f1ec0eff1c25933468e1d238201c0c2cb29 &
>> d70127e8a942364de8dd140fe73893efda363293
>>
>> Also please send your mails in text format, not html, and CC netdev ( I
>> did here)
>>
>>>
>>>
>>
>>
>
> N�����r��y���b�X��ǧv�^�){.n�+���z�^�)���w*\x1fjg���\x1e�����ݢj/���z�ޖ��2�ޙ���&�)ߡ�a��\x7f��\x1e�G���h�\x0f�j:+v���w�٥
>
Thank you for the report, I will try to reproduce this locally
Could you please post the full crash log ? Also could you test
with a clean current kernel from Linus' tree or Dave's -net ?
These are available at:
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
respectively.
One last question how many IRQs do you pin i.e. how many cores
do you actively use for receive ?
next prev parent reply other threads:[~2015-07-18 10:02 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <F8D94413-90A2-4F80-AAA2-7A6AB57DF314@transip.nl>
2015-07-18 8:56 ` reproducable panic eviction work queue Eric Dumazet
2015-07-18 9:01 ` Johan Schuijt
2015-07-18 10:02 ` Nikolay Aleksandrov [this message]
2015-07-18 13:31 ` Nikolay Aleksandrov
2015-07-18 15:28 ` Johan Schuijt
2015-07-18 15:30 ` Johan Schuijt
2015-07-18 15:32 ` Nikolay Aleksandrov
2015-07-20 12:47 ` Frank Schreuder
2015-07-20 14:02 ` Nikolay Aleksandrov
2015-07-20 14:30 ` Florian Westphal
2015-07-21 11:50 ` Frank Schreuder
2015-07-21 18:34 ` Florian Westphal
2015-07-22 8:09 ` Frank Schreuder
2015-07-22 8:17 ` Frank Schreuder
2015-07-22 9:11 ` Nikolay Aleksandrov
2015-07-22 10:55 ` Frank Schreuder
2015-07-22 13:58 ` Florian Westphal
2015-07-22 14:03 ` Nikolay Aleksandrov
2015-07-22 14:14 ` Nikolay Aleksandrov
2015-07-22 15:31 ` Frank Schreuder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55AA243D.5020306@cumulusnetworks.com \
--to=nikolay@cumulusnetworks$(echo .)com \
--cc=chutzpah@gentoo$(echo .)org \
--cc=davem@davemloft$(echo .)net \
--cc=eric.dumazet@gmail$(echo .)com \
--cc=fschreuder@transip$(echo .)nl \
--cc=fw@strlen$(echo .)de \
--cc=johan@transip$(echo .)nl \
--cc=netdev@vger$(echo .)kernel.org \
--cc=nikolay@redhat$(echo .)com \
--cc=robing@transip$(echo .)nl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox