Subject: Re: how to mix bridges and bonding inc. vlans correctly on Kernel > 3.10

From: Vlad Yasevich <[email-masked]>
Date: Wed, 13 Nov 2013 12:49:47 -0500

On 11/13/2013 12:46 PM, Stefan Priebe - Profihost AG wrote:
>
>> Am 13.11.2013 um 18:37 schrieb Vlad Yasevich <[email-masked]>:
>>
>>> On 11/13/2013 12:22 PM, Stefan Priebe - Profihost AG wrote:
>>>
>>>>> Am 13.11.2013 um 17:44 schrieb Vlad Yasevich <[email-masked]>:
>>>>>
>>>>> On 11/13/2013 10:17 AM, Stefan Priebe - Profihost AG wrote:
>>>>> Am 13.11.2013 16:05, schrieb Vlad Yasevich:
>>>>>>> On 11/13/2013 09:20 AM, Stefan Priebe - Profihost AG wrote:
>>>>>>> Hi Falico,
>>>>>>> Am 13.11.2013 15:12, schrieb Veaceslav Falico:
>>>>>>>> On Wed, Nov 13, 2013 at 02:58:40PM +0100, Stefan Priebe - Profihost AG
>>>>>>>> wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> while my vlans, bridging and bonding stuff was working until 3.9 i
>>>>>>>>> never
>>>>>>>>> thought about how it is right. So maybe i was always wrong.
>>>>>>>>>
>>>>>>>>> I've this:
>>>>>>>>>
>>>>>>>>> eth2
>>>>>>>>>       \
>>>>>>>>>        -- bond1 -- vmbr1
>>>>>>>>>       /
>>>>>>>>> eth3
>>>>>>>>>
>>>>>>>>> This works fine and as expected now i want to have a vlan using the
>>>>>>>>> bonding and using a bridge.
>>>>>>>>>
>>>>>>>>> I the past i had this:
>>>>>>>>> eth2
>>>>>>>>>       \
>>>>>>>>>        -- bond1 -- vmbr1
>>>>>>>>>       /              \
>>>>>>>>> eth3                 \ vmbr1.3000
>>>>>>>>>                             \ ---- tap114i1
>>>>>>>>>
>>>>>>>>> This was working fine until 3.9.X since 3.10. Right now using 3.10 i
>>>>>>>>> need to put eth2 and eth3 into promisc mode to get it working ;-( this
>>>>>>>>> is bad!
>>>>>>>>
>>>>>>>> As a guess - do you use arp monitoring for bonding? Try using miimon -
>>>>>>>> there were some issues with it in 3.10, which were fixed by some huge
>>>>>>>> patchsets that will never hit 3.10 stable.
>>>>>>>> Also, the bonding configuration would be welcome.
>>>>>>>
>>>>>>> Debian Bonding konfiguration looks like this:
>>>>>>> auto bond1
>>>>>>> iface bond1 inet manual
>>>>>>>           slaves eth2 eth3
>>>>>>>           bond-mode 802.3ad
>>>>>>>           bond_miimon 100
>>>>>>>           bond_updelay 200
>>>>>>>           bond_downdelay 0
>>>>>>>
>>>>>>> This should be miimon using lacp and not arp isn't it?
>>>>>>> Anything more needed?
>>>>>>
>>>>>> Hmm..  With 802.3ad mode, when the bond is a port on the bridge, the
>>>>>> bond should place all of its ports into promiscuous mode.  Do you see
>>>>>> the the kernel messages that say that?
>>>>>
>>>>> No it does not - i only see:
>>>>> # dmesg -c|egrep "promiscuous|forward"
>>>>> [    5.445161] device bond0 entered promiscuous mode
>>>>> [    7.670701] device bond1 entered promiscuous mode
>>>>> [    7.845472] vmbr0: port 1(bond0) entered forwarding state
>>>>> [    7.845474] vmbr0: port 1(bond0) entered forwarding state
>>>>> [    8.269769] vmbr1: port 1(bond1) entered forwarding state
>>>>> [    8.269771] vmbr1: port 1(bond1) entered forwarding state
>>>>
>>>> So this can happen if bond interfaces are down at the time of joining
>>>> the bridge.
>>>
>>> About which of my tried configuration do you speak? The bond itself is always up it's the internet connection. Or do you mean the vlan bond?
>>
>> The issue is in the formation for vmbr0 and vmbr1.  When bond0 and
>> bond1 enter promisc mode as part of becoming a bridge port, they should
>> put some subset of their ports into promisc mode as well (in 802.3ad
>> mode it should happen on all bond slaves).
>>
>> What I see in my config is something like:
>>
>> [ 2640.725700] device bond0 entered promiscuous mode
>> [ 2640.726623] device eth0 entered promiscuous mode
>> [ 2640.727655] br0: port 1(bond0) entered forwarding state
>> [ 2640.728718] br0: port 1(bond0) entered forwarding state
>>
>> This is my quick configuration:
>>
>>   br0 ----> bond0 -------> eth0
>>
>> I wanted to make sure in this case eth0 is properly configured in
>> promisc mode.
>>
>> If eth0 is somehow DOWN, when bond0 is added to the bridge, then it
>> will never get promisc mode set (at least in my quick test).
>> I am not sure if this what's happening in your case, but that's
>> the only explanation I can come up with for the above log you provided.
>
> But this one works fine it only does not work with vlans.
>

The fact that it does not make lower level devices enter promisc mode
indicates that it may not be working correctly.

> Is there an easy way to check where those packages get dropped.
>

dropwatch should be able to help.

-vlad

> Stefan
>
>>
>>
>> -vlad
>>
>>>
>>>> The promisc flag propagation to lower interfaces only happens when those interfaces are up.  So if for whatever reason,
>>>> the lower interfaces happen to be down during the crating of this
>>>> hierarchy, the promiscuity flag may not get set on the lower level port
>>>> device an we may end up in this situation.
>>>>
>>>> -vlad
>>>>
>>>>> Now adding variant 1:
>>>>> # dmesg -c|egrep "promiscuous|forward"
>>>>> [   85.919382] device tap113i0 entered promiscuous mode
>>>>> [   85.965018] vmbr0: port 2(tap113i0) entered forwarding state
>>>>> [   85.965023] vmbr0: port 2(tap113i0) entered forwarding state
>>>>> [   86.263292] device tap113i1 entered promiscuous mode
>>>>> [   86.314151] device vmbr1.3000 entered promiscuous mode
>>>>> [   86.314153] device vmbr1 entered promiscuous mode
>>>>> [   86.314192] vmbr1v3000: port 1(vmbr1.3000) entered forwarding state
>>>>> [   86.314196] vmbr1v3000: port 1(vmbr1.3000) entered forwarding state
>>>>> [   86.318116] vmbr1v3000: port 2(tap113i1) entered forwarding state
>>>>> [   86.318120] vmbr1v3000: port 2(tap113i1) entered forwarding state
>>>>> [  101.382129] vmbr1v3000: port 1(vmbr1.3000) entered forwarding state
>>>>>
>>>>> Now it looks like this:
>>>>> # ip a l|grep PROMISC
>>>>> 13: tap113i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc
>>>>> htb master vmbr0 state UNKNOWN qlen 500
>>>>> 14: tap113i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc
>>>>> htb master vmbr1v3000 state UNKNOWN qlen 500
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>>
>>>>> Main question is - is this one correct:
>>>>>>>>> eth2
>>>>>>>>>       \
>>>>>>>>>        -- bond1 -- vmbr1
>>>>>>>>>       /              \
>>>>>>>>> eth3                 \ vmbr1.3000
>>>>>>>>>                             \ ---- tap114i1
>>>>>
>>>>> <= does not work at all
>>>>>
>>>>> or this one?:
>>>>>>>>> eth2
>>>>>>>>>       \
>>>>>>>>>        -- bond1 -- vmbr1
>>>>>>>>>       /     \
>>>>>>>>> eth3        ----- bond1.3000 --- vmbr1v3000
>>>>>>>>>                                       \ ---- tap114i1
>>>>>
>>>>> <= works if i manually put eth2 and eth3 into promiscous mode.
>>>>>
>>>>>> -vlad
>>>>>>
>>>>>>> One thing i forgot the one with vmbr1.3000 does not work at all eben not
>>>>>>> with promisc mode. The one below works fine if i set eth2 and eth3 into
>>>>>>> promisc mode.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>>>> I also tried this one without success:
>>>>>>>>> eth2
>>>>>>>>>       \
>>>>>>>>>        -- bond1 -- vmbr1
>>>>>>>>>       /     \
>>>>>>>>> eth3        ----- bond1.3000 --- vmbr1v3000
>>>>>>>>>                                       \ ---- tap114i1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>>>>>>> the body of a message to [email-masked]
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>>>>> the body of a message to [email-masked]
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>