From: Zoltan Kiss <zoltan.kiss@citrix•com>
To: Sander Eikelenboom <linux@eikelenboom•it>
Cc: Ian Campbell <Ian.Campbell@citrix•com>,
"David S. Miller" <davem@davemloft•net>, <netdev@vger•kernel.org>,
<xen-devel@lists•xen.org>
Subject: Re: [3.15-rc3] Bisected: xen-netback mangles packets between two guests on a bridge since merge of "TX grant mapping with SKBTX_DEV_ZEROCOPY instead of copy" series.
Date: Fri, 9 May 2014 22:02:58 +0100 [thread overview]
Message-ID: <536D4282.9070309@citrix.com> (raw)
In-Reply-To: <395225650.20140430124506@eikelenboom.it>
Hi,
Sorry for the long silence on this issue, I was busy trying to figure
out what went wrong. Fun facts:
- commenting out that _pskb_pull_tail from tx_submit which
unconditionally pulls up the linear area to 128 bytes seems to solve the
problem
- I could repro the problem only when the sending guest had a 64 bit
kernel, but then even with 3.2. On the other hand, with 32 bit sending
guest it works fine. More exactly I think it boils down to the actual
config, I used XenServer Dom0 config files, see them here:
https://github.com/xenserver/linux-3.x.pg/blob/master/master/kernel-configuration
- with 64 bit Debian 7 kernel as sender it also works, so I guess it's
not about 32/64 bit, but something in the config
- the receiving guest, where wget ran, doesn't matter.
- the "more than MAX_SKB_FRAGS slots" thing was a red herring. A typical
skb layout (on the sender's xenvif_start_xmit) which gets corrupted:
linear area: 66 bytes
0. frag: 52 bytes
1. frag: 1200 bytes
- so I guess the problem is when that pull_tail pulls the whole first
frag into the linear area
- a corrupt packet on the receiver side looks like the following:
- linear buffer: 128 bytes, content is OK
- the content of the frag area is shifted back 4096 bytes in the
TCP stream. So instead of the Nth byte it starts with the (N-4096)th byte
- the length is the same as on the sender side, I've checked by
looking at the IP id fields
- otherwise the stream content looks ok (I used a continuously
incrementing pattern)
- the next packet starts at the right place
- the pulling itself doesn't cause the corruption, I've printed out the
first frag after that, and it still looks OK
- ftrace_printk("%*ph") seems to have problems when the pointer points
to a grant mapped page. I have the impression that it tries to
dereference it when I read the trace buffer, at which point the mapping
and the content is long gone.
I'll continue to look into this next week
Zoli
next prev parent reply other threads:[~2014-05-09 21:03 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-30 10:45 [3.15-rc3] Bisected: xen-netback mangles packets between two guests on a bridge since merge of "TX grant mapping with SKBTX_DEV_ZEROCOPY instead of copy" series Sander Eikelenboom
2014-04-30 15:24 ` Eric Dumazet
2014-04-30 20:40 ` Zoltan Kiss
2014-04-30 20:53 ` Zoltan Kiss
2014-04-30 22:25 ` Sander Eikelenboom
2014-05-01 13:37 ` Zoltan Kiss
2014-05-01 13:59 ` Sander Eikelenboom
2014-05-01 15:46 ` Zoltan Kiss
2014-05-01 17:39 ` Sander Eikelenboom
2014-05-01 17:46 ` Eric Dumazet
2014-05-01 19:39 ` [Xen-devel] " Sander Eikelenboom
2014-05-02 14:00 ` Zoltan Kiss
2014-05-02 14:06 ` Sander Eikelenboom
2014-05-02 14:47 ` Zoltan Kiss
2014-05-02 15:21 ` Eric Dumazet
2014-05-02 15:26 ` Zoltan Kiss
2014-05-02 16:28 ` Sander Eikelenboom
2014-05-02 16:45 ` Zoltan Kiss
2014-05-05 10:19 ` Sander Eikelenboom
2014-05-06 17:07 ` Steven Haigh
2014-05-06 17:13 ` Zoltan Kiss
2014-05-06 17:37 ` Sander Eikelenboom
2014-05-06 18:07 ` Steven Haigh
2014-05-07 8:16 ` [Xen-devel] " David Vrabel
2014-05-16 2:13 ` Steven Haigh
2014-05-06 17:08 ` [Xen-devel] " Zoltan Kiss
2014-05-06 17:10 ` Zoltan Kiss
2014-05-06 17:33 ` Sander Eikelenboom
2014-05-01 13:49 ` Zoltan Kiss
2014-05-01 14:05 ` Sander Eikelenboom
2014-05-01 15:16 ` Zoltan Kiss
2014-05-01 15:40 ` Sander Eikelenboom
2014-05-02 15:35 ` Eric Dumazet
2014-05-02 22:18 ` Sander Eikelenboom
2014-05-09 22:19 ` Neal Cardwell
2014-05-09 21:02 ` Zoltan Kiss [this message]
2014-05-13 13:40 ` Zoltan Kiss
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=536D4282.9070309@citrix.com \
--to=zoltan.kiss@citrix$(echo .)com \
--cc=Ian.Campbell@citrix$(echo .)com \
--cc=davem@davemloft$(echo .)net \
--cc=linux@eikelenboom$(echo .)it \
--cc=netdev@vger$(echo .)kernel.org \
--cc=xen-devel@lists$(echo .)xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox