From: Denys Fedoryshchenko <nuclearcat@nuclearcat•com>
To: Guillaume Nault <g.nault@alphalink•fr>
Cc: Linux Kernel Network Developers <netdev@vger•kernel.org>
Subject: Re: ppp/pppoe, still panic 4.15.3 in ppp_push
Date: Thu, 15 Feb 2018 12:19:52 +0200 [thread overview]
Message-ID: <bc40be4167a510073e78a83e35ccc3a3@nuclearcat.com> (raw)
In-Reply-To: <20180214172549.GH1422@alphalink.fr>
On 2018-02-14 19:25, Guillaume Nault wrote:
> On Wed, Feb 14, 2018 at 06:49:19PM +0200, Denys Fedoryshchenko wrote:
>> On 2018-02-14 18:47, Guillaume Nault wrote:
>> > On Wed, Feb 14, 2018 at 06:29:34PM +0200, Denys Fedoryshchenko wrote:
>> > > On 2018-02-14 18:07, Guillaume Nault wrote:
>> > > > On Wed, Feb 14, 2018 at 03:17:23PM +0200, Denys Fedoryshchenko wrote:
>> > > > > Hi,
>> > > > >
>> > > > > Upgraded kernel to 4.15.3, still it crashes after while (several
>> > > > > hours,
>> > > > > cannot do bisect, as it is production server).
>> > > > >
>> > > > > dev ppp # gdb ppp_generic.o
>> > > > > GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1
>> > > > > <<skipped>>
>> > > > > Reading symbols from ppp_generic.o...done.
>> > > > > (gdb) list *ppp_push+0x73
>> > > > > 0x681 is in ppp_push (drivers/net/ppp/ppp_generic.c:1663).
>> > > > > 1658 list = list->next;
>> > > > > 1659 pch = list_entry(list, struct channel, clist);
>> > > > > 1660
>> > > > > 1661 spin_lock(&pch->downl);
>> > > > > 1662 if (pch->chan) {
>> > > > > 1663 if (pch->chan->ops->start_xmit(pch->chan, skb))
>> > > > > 1664 ppp->xmit_pending = NULL;
>> > > > > 1665 } else {
>> > > > > 1666 /* channel got unregistered */
>> > > > > 1667 kfree_skb(skb);
>> > > > >
>> > > > >
>> > > > I expect a memory corruption. Do you have the possibility to run with
>> > > > KASAN by any chance?
>> > > I will try to enable it tonight. For now i reverted "drivers, net,
>> > > ppp:
>> > > convert ppp_file.refcnt from atomic_t to refcount_t" for test.
>> > >
>> > This commit looks good to me. Do you have doubts about it because it's
>> > new in 4.15? Does it mean that your last known-good kernel is 4.14?
>>
>> I am just doing "manual" bisect, checking all possibilities, and
>> picking
>> patch to revert randomly.
>>
> Must be a painful process. Are all of your networking modules required?
> With luck, you might be able to isolate a faulty module in fewer steps.
>
>> Yes, correct, my known-good is 4.14.2.
>>
> Good to know.
>
> Let me know if you can get a KASAN trace.
Here we go:
<srv> [24558.921549]
==================================================================
<srv> [24558.922167] BUG: KASAN: use-after-free in
ppp_ioctl+0xa6a/0x1522 [ppp_generic]
<srv> [24558.922776] Write of size 8 at addr ffff8803d35bf3f8 by task
accel-pppd/12622
<srv> [24558.923113]
<srv> [24558.923451] CPU: 0 PID: 12622 Comm: accel-pppd Tainted: G
W 4.15.3-build-0134 #1
<srv> [24558.924058] Hardware name: HP ProLiant DL320e Gen8 v2, BIOS
P80 04/02/2015
<srv> [24558.924406] Call Trace:
<srv> [24558.924753] dump_stack+0x46/0x59
<srv> [24558.925103] print_address_description+0x6b/0x23b
<srv> [24558.925451] ? ppp_ioctl+0xa6a/0x1522 [ppp_generic]
<srv> [24558.925797] kasan_report+0x21b/0x241
<srv> [24558.926136] ppp_ioctl+0xa6a/0x1522 [ppp_generic]
<srv> [24558.926479] ? ppp_nl_newlink+0x1da/0x1da [ppp_generic]
<srv> [24558.926829] ? sock_sendmsg+0x89/0x99
<srv> [24558.927176] ? __vfs_write+0xd9/0x4ad
<srv> [24558.927523] ? kernel_read+0xed/0xed
<srv> [24558.927872] ? SyS_getpeername+0x18c/0x18c
<srv> [24558.928213] ? bit_waitqueue+0x2a/0x2a
<srv> [24558.928561] ? wake_atomic_t_function+0x115/0x115
<srv> [24558.928898] vfs_ioctl+0x6e/0x81
<srv> [24558.929228] do_vfs_ioctl+0xa00/0xb10
<srv> [24558.929571] ? sigprocmask+0x1a6/0x1d0
<srv> [24558.929907] ? sigsuspend+0x13e/0x13e
<srv> [24558.930239] ? ioctl_preallocate+0x14e/0x14e
<srv> [24558.930568] ? SyS_rt_sigprocmask+0xf1/0x142
<srv> [24558.930904] ? sigprocmask+0x1d0/0x1d0
<srv> [24558.931252] SyS_ioctl+0x39/0x55
<srv> [24558.931595] ? do_vfs_ioctl+0xb10/0xb10
<srv> [24558.931942] do_syscall_64+0x1b1/0x31f
<srv> [24558.932288] entry_SYSCALL_64_after_hwframe+0x21/0x86
<srv> [24558.932627] RIP: 0033:0x7f302849d8a7
<srv> [24558.932965] RSP: 002b:00007f3029a52af8 EFLAGS: 00000206
ORIG_RAX: 0000000000000010
<srv> [24558.933578] RAX: ffffffffffffffda RBX: 00007f3027d861e3 RCX:
00007f302849d8a7
<srv> [24558.933927] RDX: 00007f3023f49468 RSI: 000000004004743a RDI:
0000000000003a67
<srv> [24558.934266] RBP: 00007f3029a52b20 R08: 0000000000000000 R09:
000055c8308d8e40
<srv> [24558.934607] R10: 0000000000000008 R11: 0000000000000206 R12:
00007f3023f49358
<srv> [24558.934947] R13: 00007ffe86e5723f R14: 0000000000000000 R15:
00007f3029a53700
<srv> [24558.935288]
<srv> [24558.935626] Allocated by task 12622:
<srv> [24558.935972] ppp_register_net_channel+0x5f/0x5c6 [ppp_generic]
<srv> [24558.936306] pppoe_connect+0xab7/0xc71 [pppoe]
<srv> [24558.936640] SyS_connect+0x14b/0x1b7
<srv> [24558.936975] do_syscall_64+0x1b1/0x31f
<srv> [24558.937319] entry_SYSCALL_64_after_hwframe+0x21/0x86
<srv> [24558.937655]
<srv> [24558.937993] Freed by task 12622:
<srv> [24558.938321] kfree+0xb0/0x11d
<srv> [24558.938658] ppp_release+0x111/0x120 [ppp_generic]
<srv> [24558.938994] __fput+0x2ba/0x51a
<srv> [24558.939332] task_work_run+0x11c/0x13d
<srv> [24558.939676] exit_to_usermode_loop+0x7c/0xaf
<srv> [24558.940022] do_syscall_64+0x2ea/0x31f
<srv> [24558.940368] entry_SYSCALL_64_after_hwframe+0x21/0x86
<srv> [24558.947099]
<srv> [24558.947443] The buggy address belongs to the object at
ffff8803d35bf340
[24558.947443] which belongs to the cache kmalloc-256 of size 256
<srv> [24558.948064] The buggy address is located 184 bytes inside of
[24558.948064] 256-byte region [ffff8803d35bf340, ffff8803d35bf440)
<srv> [24558.948676] The buggy address belongs to the page:
<srv> [24558.949019] page:ffffea000f4d6f00 count:1 mapcount:0 mapping:
(null) index:0xffff8803d35bfc00 compound_mapcount: 0
<srv> [24558.949633] flags: 0x17ffe00000008100(slab|head)
<srv> [24558.949980] raw: 17ffe00000008100 0000000000000000
ffff8803d35bfc00 000000010033002e
<srv> [24558.950597] raw: ffffea000d98c020 ffffea000df57d20
ffff8803f1c0f480 0000000000000000
<srv> [24558.951209] page dumped because: kasan: bad access detected
<srv> [24558.951546]
<srv> [24558.951880] Memory state around the buggy address:
<srv> [24558.952217] ffff8803d35bf280: fb fb fb fb fb fb fb fb fb fb
fb fb fb fb fb fb
<srv> [24558.952818] ffff8803d35bf300: fc fc fc fc fc fc fc fc fb fb
fb fb fb fb fb fb
<srv> [24558.953415] >ffff8803d35bf380: fb fb fb fb fb fb fb fb fb fb
fb fb fb fb fb fb
<srv> [24558.954021]
^
<srv> [24558.954365] ffff8803d35bf400: fb fb fb fb fb fb fb fb fc fc
fc fc fc fc fc fc
<srv> [24558.954969] ffff8803d35bf480: 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00
<srv> [24558.955572]
==================================================================
<srv> [24558.956169] Disabling lock debugging due to kernel taint
next prev parent reply other threads:[~2018-02-15 10:19 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-14 13:17 ppp/pppoe, still panic 4.15.3 in ppp_push Denys Fedoryshchenko
2018-02-14 16:07 ` Guillaume Nault
2018-02-14 16:29 ` Denys Fedoryshchenko
2018-02-14 16:47 ` Guillaume Nault
2018-02-14 16:49 ` Denys Fedoryshchenko
2018-02-14 17:25 ` Guillaume Nault
2018-02-15 10:19 ` Denys Fedoryshchenko [this message]
2018-02-15 15:55 ` Guillaume Nault
2018-02-15 16:01 ` Denys Fedoryshchenko
2018-02-15 19:31 ` Guillaume Nault
2018-02-15 19:34 ` Denys Fedoryshchenko
2018-02-15 19:42 ` Guillaume Nault
2018-02-16 11:13 ` Denys Fedoryshchenko
2018-02-16 18:48 ` Guillaume Nault
2018-02-18 10:01 ` Denys Fedoryshchenko
2018-02-21 18:38 ` Guillaume Nault
2018-02-20 9:05 ` Denys Fedoryshchenko
2018-02-21 10:26 ` Denys Fedoryshchenko
2018-02-21 18:55 ` Guillaume Nault
2018-02-21 19:30 ` Denys Fedoryshchenko
2018-02-21 20:04 ` Cong Wang
2018-02-22 18:30 ` Guillaume Nault
2018-02-22 18:51 ` Denys Fedoryshchenko
2018-02-23 9:38 ` Guillaume Nault
2018-02-23 9:41 ` Denys Fedoryshchenko
2018-02-23 10:07 ` Guillaume Nault
2018-02-23 10:54 ` Denys Fedoryshchenko
2018-02-24 21:22 ` Denys Fedoryshchenko
2018-02-27 10:58 ` Denys Fedoryshchenko
2018-02-27 18:56 ` Guillaume Nault
2018-03-01 20:01 ` Guillaume Nault
2018-03-01 20:07 ` Denys Fedoryshchenko
2018-03-02 17:43 ` Guillaume Nault
2018-03-03 9:33 ` Denys Fedoryshchenko
2018-03-05 17:22 ` Guillaume Nault
2018-02-27 18:54 ` Guillaume Nault
2018-02-15 19:20 ` Guillaume Nault
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bc40be4167a510073e78a83e35ccc3a3@nuclearcat.com \
--to=nuclearcat@nuclearcat$(echo .)com \
--cc=g.nault@alphalink$(echo .)fr \
--cc=netdev@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox