Subject: Re: bonding can't change to another slave if you ifdown the active slave

From: Weiping Pan <[email-masked]>
Date: Mon, 07 Mar 2011 11:13:27 +0800

On 03/05/2011 09:49 PM, Nicolas de Pesloüan wrote:
> Le 05/03/2011 03:53, Andy Gospodarek a écrit :
>> On Fri, Mar 04, 2011 at 10:15:17AM +0800, Weiping Pan wrote:
>>> Hi,
>>>
>>> I'm doing some Linux bonding driver test, and I find a problem in
>>> balance-rr mode.
>>> That's it can't change to another slave if you ifdown the active slave.
>>> Any comments are warmly welcomed!
>>>
>>> regards
>>> Weiping Pan
>>>
>>> My host is Fedora 14, and I install VirtualBox (4.0.2), and enable 4
>>> nics for the guest system.
>>
>> Does this mean you are passing 4 NICs from your host to your guest
>> (maybe via direct pci-device assignment to the guest) or are you
>> creating 4 virtual devices on the host that are in a bridge group on the
>> host?
>
> VirtualBox does not allow assignment of pci-device to the guest. The 
> network interfaces on the guest are pure virtual one, with several 
> modes available. In order to help you trouble shooting this problem, 
> we need to know the mode form each of the virtual interfaces. Possible 
> modes are NAT, bridged, internal-network, and host-only-network.
>
> Please provide the output of the following command:
>
> VBoxManage showvminfo <your-vm-uuid> | grep ^NIC
>
> To display your vm uuid, use the following command:
>
> VBoxManage list vms
[root@localhost ~]# VBoxManage showvminfo 
67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 |grep ^NIC
NIC 1:           MAC: 0800270481A8, Attachment: Bridged Interface 
'eth0', Cable connected: on, Trace: off (file: none), Type: 82540EM, 
Reported speed: 0 Mbps, Boot priority: 0
NIC 2:           MAC: 08002778F641, Attachment: Bridged Interface 
'eth0', Cable connected: on, Trace: off (file: none), Type: 82540EM, 
Reported speed: 0 Mbps, Boot priority: 0
NIC 3:           MAC: 080027C408BA, Attachment: Bridged Interface 
'eth0', Cable connected: on, Trace: off (file: none), Type: 82540EM, 
Reported speed: 0 Mbps, Boot priority: 0
NIC 4:           MAC: 080027DB339A, Attachment: Bridged Interface 
'eth0', Cable connected: on, Trace: off (file: none), Type: 82540EM, 
Reported speed: 0 Mbps, Boot priority: 0
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled

And when guest starts, i find that:
NIC 1: eth7
NIC 2: eth6
NIC 3: eth9
NIC 4: eth8

>
>>
>> [...]
>>> [root@localhost ~]# ifconfig eth7 down
>>
>> This is not a great way to test link failure with bonding.  The best way
>> is to actually pull the cable so the interface is truly down.
>
> To virtually plug or unplug the cable from a virtual interface, use 
> the following command, replacing the # with the interface number (from 
> 1 to 8):
>
> VBoxManage controlvm setlinkstate# on
> VBoxManage controlvm setlinkstate# off
I repeat my test with your guide, but it still doesn't work!

First on my host,
ifconfig eth0:0 192.168.1.100 netmask 255.255.255.0 up

And restart my guest,
[root@localhost ~]# ifconfig
eth6      Link encap:Ethernet  HWaddr 08:00:27:78:F6:41
           inet addr:10.66.65.128  Bcast:10.66.65.255  Mask:255.255.254.0
           inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:22 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:3608 (3.5 KiB)  TX bytes:1152 (1.1 KiB)

eth7      Link encap:Ethernet  HWaddr 08:00:27:04:81:A8
           inet addr:10.66.65.53  Bcast:10.66.65.255  Mask:255.255.254.0
           inet6 addr: fe80::a00:27ff:fe04:81a8/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:23 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:3668 (3.5 KiB)  TX bytes:1152 (1.1 KiB)

eth8      Link encap:Ethernet  HWaddr 08:00:27:DB:33:9A
           inet addr:10.66.65.237  Bcast:10.66.65.255  Mask:255.255.254.0
           inet6 addr: fe80::a00:27ff:fedb:339a/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:147 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:14783 (14.4 KiB)  TX bytes:1152 (1.1 KiB)

eth9      Link encap:Ethernet  HWaddr 08:00:27:C4:08:BA
           inet addr:10.66.65.125  Bcast:10.66.65.255  Mask:255.255.254.0
           inet6 addr: fe80::a00:27ff:fec4:8ba/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:147 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:14783 (14.4 KiB)  TX bytes:1152 (1.1 KiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:16 errors:0 dropped:0 overruns:0 frame:0
           TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:880 (880.0 b)  TX bytes:880 (880.0 b)

[root@localhost ~]# ifconfig eth6 down
[root@localhost ~]# ifconfig eth7 down
[root@localhost ~]# ifconfig eth8 down
[root@localhost ~]# ifconfig eth9 down
[root@localhost ~]# ip route show
[root@localhost ~]# ip neigh show
[root@localhost ~]# ifconfig
lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:16 errors:0 dropped:0 overruns:0 frame:0
           TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:880 (880.0 b)  TX bytes:880 (880.0 b)

[root@localhost ~]# dmesg -c &> /dev/null
[root@localhost ~]# modprobe bonding mode=0 miimon=100
[root@localhost ~]# ifconfig bond0 192.168.1.5 netmask 255.255.255.0 up
[root@localhost ~]# ifenslave bond0 eth6
[root@localhost ~]# ifenslave bond0 eth7
[root@localhost ~]# dmesg
[  164.865840] bonding: Ethernet Channel Bonding Driver: v3.6.0 
(September 26, 2009)
[  164.865845] bonding: MII link monitoring set to 100 ms
[  181.186201] ADDRCONF(NETDEV_UP): bond0: link is not ready
[  191.549252] bonding: bond0: enslaving eth6 as an active interface 
with a down link.
[  191.552653] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX
[  191.586166] bonding: bond0: link status definitely up for interface eth6.
[  191.586315] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[  193.420974] e1000: eth7 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX
[  193.434907] bonding: bond0: enslaving eth7 as an active interface 
with an up link.

[root@localhost ~]# ifconfig
bond0     Link encap:Ethernet  HWaddr 08:00:27:78:F6:41
           inet addr:192.168.1.5  Bcast:192.168.1.255  Mask:255.255.255.0
           inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link
           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
           RX packets:95 errors:0 dropped:0 overruns:0 frame:0
           TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:13415 (13.1 KiB)  TX bytes:4140 (4.0 KiB)

eth6      Link encap:Ethernet  HWaddr 08:00:27:78:F6:41
           inet addr:10.66.65.128  Bcast:10.66.65.255  Mask:255.255.254.0
           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
           RX packets:48 errors:0 dropped:0 overruns:0 frame:0
           TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:7464 (7.2 KiB)  TX bytes:1822 (1.7 KiB)

eth7      Link encap:Ethernet  HWaddr 08:00:27:78:F6:41
           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
           RX packets:47 errors:0 dropped:0 overruns:0 frame:0
           TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:5951 (5.8 KiB)  TX bytes:2318 (2.2 KiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:16 errors:0 dropped:0 overruns:0 frame:0
           TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:880 (880.0 b)  TX bytes:880 (880.0 b)

[root@localhost ~]# ping 192.168.1.100 -c 5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=1.98 ms
64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.955 ms
64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.209 ms
64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.277 ms
64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.289 ms

--- 192.168.1.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.209/0.742/1.984/0.678 ms

[root@localhost ~]# ip route show
192.168.1.0/24 dev bond0  proto kernel  scope link  src 192.168.1.5
10.66.64.0/23 dev eth6  proto kernel  scope link  src 10.66.65.128  
metric 1
default via 10.66.65.254 dev eth6  proto static
[root@localhost ~]# ip neigh show
192.168.1.100 dev bond0 lladdr 64:31:50:3a:b0:b5 STALE

And on host,
[root@localhost ~]# VBoxManage controlvm 
67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate2 off

Then on guest,
[root@localhost ~]# ethtool eth6
Settings for eth6:
         Supported ports: [ TP ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Full
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Full
         Advertised pause frame use: No
         Advertised auto-negotiation: Yes
         Speed: Unknown!
         Duplex: Unknown! (255)
         Port: Twisted Pair
         PHYAD: 0
         Transceiver: internal
         Auto-negotiation: on
         MDI-X: Unknown
         Supports Wake-on: umbg
         Wake-on: d
         Current message level: 0x00000007 (7)
         Link detected: no

[root@localhost ~]# dmesg
[  164.865840] bonding: Ethernet Channel Bonding Driver: v3.6.0 
(September 26, 2009)
[  164.865845] bonding: MII link monitoring set to 100 ms
[  181.186201] ADDRCONF(NETDEV_UP): bond0: link is not ready
[  191.549252] bonding: bond0: enslaving eth6 as an active interface 
with a down link.
[  191.552653] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX
[  191.586166] bonding: bond0: link status definitely up for interface eth6.
[  191.586315] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[  193.420974] e1000: eth7 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX
[  193.434907] bonding: bond0: enslaving eth7 as an active interface 
with an up link.
[  202.018085] bond0: no IPv6 routers present
[  238.834001] e1000: eth7 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX
[  434.010205] e1000: eth6 NIC Link is Down
[  434.011661] bonding: bond0: link status definitely down for interface 
eth6, disabling it

[root@localhost ~]# ping 192.168.1.100 -c 5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
 From 192.168.1.5 icmp_seq=2 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=3 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=4 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=5 Destination Host Unreachable

--- 192.168.1.100 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4001ms
pipe 3

[root@localhost ~]# ip route show
192.168.1.0/24 dev bond0  proto kernel  scope link  src 192.168.1.5
[root@localhost ~]# ip neigh show
192.168.1.100 dev bond0  FAILED

ping on the guest while tcpdump on the host,
on guest:
[root@localhost ~]# ping 192.168.1.100
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
 From 192.168.1.5 icmp_seq=2 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=3 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=4 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=6 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=7 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=8 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=10 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=11 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=12 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=14 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=15 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=16 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=18 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=19 Destination Host Unreachable
 From 192.168.1.5 icmp_seq=20 Destination Host Unreachable
^C
--- 192.168.1.100 ping statistics ---
21 packets transmitted, 0 received, +15 errors, 100% packet loss, time 
20005ms
pipe 3

on host:
[root@localhost ~]# tcpdump -i eth0 -p arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:00:50.474242 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:50.474256 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:52.469651 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:52.469661 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:52.632719 ARP, Request who-has dhcp-65-29.nay.redhat.com tell 
corerouter.nay.redhat.com, length 46
11:00:53.192150 ARP, Request who-has dhcp-65-14.nay.redhat.com tell 
corerouter.nay.redhat.com, length 46
11:00:53.471246 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:53.471257 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:54.474627 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:54.474636 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:56.472050 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:56.472060 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:57.475211 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:57.475220 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:58.476840 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:58.476849 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:00:58.624738 ARP, Request who-has dhcp-65-29.nay.redhat.com tell 
corerouter.nay.redhat.com, length 46
11:01:00.477029 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
11:01:00.477038 ARP, Request who-has 192.168.1.100 tell 192.168.1.5, 
length 28
^C
19 packets captured
19 packets received by filter
0 packets dropped by kernel
[root@localhost ~]# ip neigh show
192.168.1.5 dev eth0 lladdr 08:00:27:78:f6:41 STALE
10.66.65.254 dev eth0 lladdr 00:1d:45:20:d5:ff REACHABLE

regards
Weiping Pan