Subject: netperf udp_rr testing hang

From: "Zhang, Yanmin" <[email-masked]>
Date: Fri, 25 Apr 2008 15:42:23 +0800

I am testing network UDP by netperf V2.4.4.

I have 2 machines. Every machine has 2 NIC, so 2 IP addresses per machine.
Client1: 192.168.1.164 (eth1:lkp-h01-nic2.tsp.org) and 192.168.1.169 (eth2:lkp-h01.tsp.org).
Server1: 192.168.1.160 (eth0:lkp-tt02-x8664.tsp.org) and 192.168.1.153 (eth1:lkp-tt02-nic2.tsp.org).

They are connected to the same GIGA switch.

On Server1, start netserver:
#./netserver&

Then, on Client1: start netperf:
#./netperf -t UDP_RR -l 60 -H 192.168.1.153 -L 192.168.1.164 -i 3,3 -I 99,5 -- -r 1,1
It looks like netperf hangs and exits after 180(or 60?) seconds. The result shows
RatePerSec is 0.0.
If I use tcpdump to intercept all packets on eth1 on client1, the dump shows:
14:49:14.820924 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 1 win 46 <nop,nop,timestamp 1691048 1803244>
14:49:14.821043 IP lkp-h01.tsp.org.ssh > lkp-os.tsp.org.45485: P 176:368(192) ack 49 win 146 <nop,nop,timestamp 1691048 1575061177>
14:49:14.821047 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: P 1:257(256) ack 1 win 46 <nop,nop,timestamp 1691048 1803244>
14:49:14.821157 IP lkp-tt02-nic2.tsp.org.12865 > lkp-h01.tsp.org.41456: . ack 257 win 54 <nop,nop,timestamp 1803244 1691048>
14:49:14.821307 IP lkp-tt02-nic2.tsp.org.12865 > lkp-h01.tsp.org.41456: P 1:257(256) ack 257 win 54 <nop,nop,timestamp 1803244 1691048>
14:49:14.821312 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 257 win 54 <nop,nop,timestamp 1691048 1803244>
14:49:14.821348 IP lkp-h01.tsp.org.54226 > lkp-tt02-nic2.tsp.org.42164: UDP, length 1
14:49:14.821406 IP lkp-tt02-x8664.tsp.org.42164 > lkp-h01.tsp.org.54226: UDP, length 1
14:49:14.821415 IP lkp-h01.tsp.org > lkp-tt02-x8664.tsp.org: ICMP lkp-h01.tsp.org udp port 54226 unreachable, length 37

If I use tcpdump to intercept all packets on eth1 on Server1, the dump shows:
23:54:12.320760 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: S 2825016431:2825016431(0) win 5840 <mss 1460,sackOK,timestamp 1691048 0,nop,wscale 7>
23:54:12.320858 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 1965002601 win 46 <nop,nop,timestamp 1691048 1803244>
23:54:12.321010 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: P 0:256(256) ack 1 win 46 <nop,nop,timestamp 1691048 1803244>
23:54:12.321259 IP lkp-h01.tsp.org.41456 > lkp-tt02-nic2.tsp.org.12865: . ack 257 win 54 <nop,nop,timestamp 1691048 1803244>
23:54:12.321271 IP lkp-h01.tsp.org.54226 > lkp-tt02-nic2.tsp.org.42164: UDP, length 1

If I start netperf by below command:
#./netperf -t UDP_RR -l 60 -H 192.168.1.160 -L 192.168.1.164 -i 3,3 -I 99,5 -- -r 1,1
The testing really goes ahead and prints correct result after testing. However, tcpdump shows
the packets just pass between lkp-h01.tsp.org.50303 and lkp-tt02-x8664.tsp.org.41305, not
lkp-h01-nic2.tsp.org.50303 and lkp-tt02-x8664.tsp.org.41305

I check source codes of netperf and send_udp_rr really binds the correct local/host IP.

I tries TCP_RR and it has no hang issue although packets might be sent out from another IP.

I tested it with kernel 2.6.22/23/24/25.

Thanks,
Yanmin