Subject: RE: netperf udp_rr testing hang

From: "Zhang, Yanmin" <[email-masked]>
Date: Mon, 28 Apr 2008 11:43:20 +0800

On Sun, 2008-04-27 at 16:47 -0700, Brandeburg, Jesse wrote: 
Thanks for your kind response. I think it might be an issue of kernel. Pls. see below comments.

> are you turning on arp_filter in sysctl?
No. I use the default configuration, i.e. arp_filter=0.

> 
> IMO you can't use two IP addresses in the same subnet on the same switch anyway, even with arp filter.
> 
> if you were to assign 192.168.0.X to one interface and 192.168.1.X to the other, *and* then use arp filter it will work okay.
Why does TCP work well? Lab manager just configures 192.168.0.XXX on the dns server.

I tried arp_filter=1 a moment ago and it doesn't work.

I checked document e1000.txt and it says Multiple Interfaces on Same Ethernet Broadcast Network
results in unbalanced receive traffic. But it doesn't say it will break the network.

>>From the tcpdump info, I think kernel on the client machine always drops the first UDP packet unexpectedly, after
the server (lkp-tt02-nic2.tsp.org) sends the first UDP response back. If I disable 192.168.1.160 (eth0:lkp-tt02-x8664.tsp.org)
on server, the testing could go ahead. That also means the issue isn't relevant to arp_filter. If I firstly disable
192.168.1.160 (eth0:lkp-tt02-x8664.tsp.org) and then reenable it, restart testing, the testing also could go ahead, but then
the testing to IP 192.168.1.160 becomes hanging. So it looks like only one ip at the same time could be
available for UDP.

> 
> jesse 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhang, Yanmin
> Sent: Friday, April 25, 2008 12:42 AM
> To: [email-masked]
> Cc: Rick Jones
> Subject: netperf udp_rr testing hang
> 
> I am testing network UDP by netperf V2.4.4.
> 
> I have 2 machines. Every machine has 2 NIC, so 2 IP addresses per machine.
> Client1: 192.168.1.164 (eth1:lkp-h01-nic2.tsp.org) and 192.168.1.169 (eth2:lkp-h01.tsp.org).
> Server1: 192.168.1.160 (eth0:lkp-tt02-x8664.tsp.org) and 192.168.1.153 (eth1:lkp-tt02-nic2.tsp.org).
> 
> They are connected to the same GIGA switch.
> 
> On Server1, start netserver:
> #./netserver&
> 
> Then, on Client1: start netperf:
> #./netperf -t UDP_RR -l 60 -H 192.168.1.153 -L 192.168.1.164 -i 3,3 -I 99,5 -- -r 1,1
> It looks like netperf hangs and exits after 180(or 60?) seconds. The result shows
> RatePerSec is 0.0.
> If I use tcpdump to intercept all packets on eth1 on client1, the dump shows:
> 14:49:14.820924 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 1 win 46 <nop,nop,timestamp 1691048 1803244>
> 14:49:14.821043 IP lkp-h01.tsp.org.ssh > lkp-os.tsp.org.45485: P 176:368(192) ack 49 win 146 <nop,nop,timestamp 1691048 1575061177>
> 14:49:14.821047 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: P 1:257(256) ack 1 win 46 <nop,nop,timestamp 1691048 1803244>
> 14:49:14.821157 IP lkp-tt02-nic2.tsp.org.12865 > lkp-h01.tsp.org.41456: . ack 257 win 54 <nop,nop,timestamp 1803244 1691048>
> 14:49:14.821307 IP lkp-tt02-nic2.tsp.org.12865 > lkp-h01.tsp.org.41456: P 1:257(256) ack 257 win 54 <nop,nop,timestamp 1803244 1691048>
> 14:49:14.821312 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 257 win 54 <nop,nop,timestamp 1691048 1803244>
> 14:49:14.821348 IP lkp-h01.tsp.org.54226 > lkp-tt02-nic2.tsp.org.42164: UDP, length 1
> 14:49:14.821406 IP lkp-tt02-x8664.tsp.org.42164 > lkp-h01.tsp.org.54226: UDP, length 1
> 14:49:14.821415 IP lkp-h01.tsp.org > lkp-tt02-x8664.tsp.org: ICMP lkp-h01.tsp.org udp port 54226 unreachable, length 37
> 
> 
> If I use tcpdump to intercept all packets on eth1 on Server1, the dump shows:
> 23:54:12.320760 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: S 2825016431:2825016431(0) win 5840 <mss 1460,sackOK,timestamp 1691048 0,nop,wscale 7>
> 23:54:12.320858 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 1965002601 win 46 <nop,nop,timestamp 1691048 1803244>
> 23:54:12.321010 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: P 0:256(256) ack 1 win 46 <nop,nop,timestamp 1691048 1803244>
> 23:54:12.321259 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 257 win 54 <nop,nop,timestamp 1691048 1803244>
> 23:54:12.321271 IP lkp-h01.tsp.org.54226 > lkp-tt02-nic2.tsp.org.42164: UDP, length 1
> 
> 
> If I start netperf by below command:
> #./netperf -t UDP_RR -l 60 -H 192.168.1.160 -L 192.168.1.164 -i 3,3 -I 99,5 -- -r 1,1
> The testing really goes ahead and prints correct result after testing. However, tcpdump shows
> the packets just pass between lkp-h01.tsp.org.50303 and lkp-tt02-x8664.tsp.org.41305, not
> lkp-h01-nic2.tsp.org.50303 and lkp-tt02-x8664.tsp.org.41305
> 
> I check source codes of netperf and send_udp_rr really binds the correct local/host IP.
> 
> I tries TCP_RR and it has no hang issue although packets might be sent out from another IP.
> 
> I tested it with kernel 2.6.22/23/24/25.
> 
> Thanks,
> Yanmin