public inbox for netdev@vger.kernel.org 
 help / color / mirror / Atom feed

From: stranche@codeaurora•org
To: eric.dumazet@gmail•com
Cc: "soheil@google•com" <netdev@vger•kernel.org>
Subject: WARN_ON in TLP causing RT throttling
Date: Wed, 26 Sep 2018 17:46:27 -0600	[thread overview]
Message-ID: <7aa9932a59aad7a21c7f8a8146dd0542@codeaurora.org> (raw)

Hi Eric,

Someone recently reported a crash to us on the 4.14.62 kernel where 
excessive
WARNING prints were spamming the logs and causing watchdog bites. The 
kernel
does have the following commit by Soheil:
bffd168c3fc5 "tcp: clear tp->packets_out when purging write queue"

Before this bug we see over 1 second of continuous WARN_ON prints from
tcp_send_loss_probe() like so:

7795.530450:   <2>  tcp_send_loss_probe+0x194/0x1b8
7795.534833:   <2>  tcp_write_timer_handler+0xf8/0x1c4
7795.539492:   <2>  tcp_write_timer+0x4c/0x74
7795.543348:   <2>  call_timer_fn+0xc0/0x1b4
7795.547113:   <2>  run_timer_softirq+0x248/0x81c

Specifically, the prints come from the following check:

	/* Retransmit last segment. */
	if (WARN_ON(!skb))
		goto rearm_timer;

Since skb is always NULL, we know there's nothing on the write queue or 
the
retransmit queue, so we just keep resetting the timer, waiting for more 
data
to be queued. However, we were able to determine that the TCP socket is 
in the
TCP_FIN_WAIT1 state, so we will no longer be sending any data and these 
queues
remain empty.

Would it be appropriate to stop resetting the TLP timer if we detect 
that the
connection is starting to close and we have no more data to send the 
probe with,
or is there some way that this scenario should already be handled?

Unfortunately, we don't have a reproducer for this crash.

Thanks,
Sean

next             reply	other threads:[~2018-09-27  6:01 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-26 23:46 stranche [this message]
2018-09-27  0:09 ` WARN_ON in TLP causing RT throttling Eric Dumazet
2018-09-27 19:14   ` Yuchung Cheng
2018-09-28  0:16     ` stranche
2018-09-28  0:25       ` Eric Dumazet
2018-09-28 16:20         ` stranche
2018-10-02 21:19       ` Yuchung Cheng

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7aa9932a59aad7a21c7f8a8146dd0542@codeaurora.org \
    --to=stranche@codeaurora$(echo .)org \
    --cc=eric.dumazet@gmail$(echo .)com \
    --cc=netdev@vger$(echo .)kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox