Linux Networking Bridge
This is the second post in a series of posts showing how to set up networking in Linux using low-level tools.
It’s time to talk about bridging (switching) and VLANs.
Bridging, or Switching
The fist post introduced LANs and broadcast domains. An Ethernet bridge, or more commonly, a switch, connects multiple networks segments into a common broadcast domain. If you are interested in this, see the Wikipedia page on bridging for details.
In Linux we can create a software defined switch by adding multiple network interfaces (NICs) to a PC and then connect them to the bridge module. In this setup these interfaces are called ports, and we don’t set IP addresses on them. Instead, we do that on the bridge, and on interfaces on top of the bridge.
Virtual LANs, VLANs
To kick things up a notch we need to introduce one more concept before moving on – VLANs!
A VLAN, or virtual LAN, is one of the true corner stones in most network setups, and as such is really deserves a blog post of its own.
However, for the purpose of this post, consider VLANs a way for us to group ports in separate broadcast domains. I.e., isolate certain end devices from each other; e.g., an office network from a process control network.
br0 ______|______ |#_#_#_#_#_#_#| / | : | \ eth0 eth1 : eth2 eth3 : VLAN 1 : VLAN 2 Here we have configured the bridge (switch) to assign ports eth0 and eth1 to VLAN 1, and eth2 and eth3 to VLAN 2. Ports in each VLAN can only communicate with each other, the bridge ensures a true separation between both VLANs.
If a device on port eth0 (member of VLAN 1) wants to communicate with a device on port eth3 (member of VLAN 2) it must be routed somehow. For this to work we must either connect a router to ports eth1 and eth2, or let interface br0 be a member of both VLANs.
A port that is member of more than one VLAN is often referred to as a trunk port, and a port facing an end device is called access port.
Port VLAN memberships can be tagged or untagged. A tagged port is usually a trunk port, and an untagged port is usually an access port. There are always exceptions to these rules, but for most cases this is a good starting point.
To route traffic between VLAN 1 and VLAN 2 we create the following setup (it’s starting to look a bit crazy now):
IP: 192.168.1.1 IP: 192.168.2.1 br0.1 br0.2 \ / br0 ______|______ |#_#_#_#_#_#_#| / | : | \ eth0 eth1 : eth2 eth3 : VLAN 1 : VLAN 2 Since br0 now is a tagged member of both VLANs we need to create VLAN interfaces on top of it to be able to set IP addresses. These are the gateway addresses each end device will use in their IP network setup.
That is basically it, remember to enable IP forwarding … now let’s get hands-on with the command line!
In the next section we use the names vlan1 and vlan2 instead of br0.1 and br0.2 , respectively. The naming is not only create confusion, but to a) show that any name can be used, and b) simplify and follow the terminology used in Westermo WeOS.
Creating a Bridge in Linux
There are actually two variants of the standard bridge in mainline Linux; old-style and new-style. The latter, which we will focus on in this blog post, has native support for VLAN filtering.
# ip link add br0 type bridge # ip link set br0 type bridge vlan_filtering 1 Note: recent versions of Debian based systems, like Ubuntu, have enabled bridge firewalling by default. This may completely disable all or some forwarding of traffic on bridges. Causing a lot of head scratching! See Bridge Forwarding Problem for a fix!
Now, add a couple of ports to the bridge:
# ip link set eth0 master br0 # ip link set eth1 master br0 To see the ports we use the bridge(8) command, which is also part of the iproute2 tool suite:
# bridge link 2: eth0: mtu 1500 master br0 state disabled priority 32 cost 100 4: eth1: mtu 1500 master br0 state disabled priority 32 cost 4 To see the default VLAN assignments of ports:
# bridge vlan show port vlan ids eth0 1 PVID Egress Untagged eth1 1 PVID Egress Untagged br0 1 PVID Egress Untagged So these ports look OK, the default VLAN ID assigned to ports is 1. Lets add the other two, but now we need to tell the bridge to use VLAN ID 2 instead. We also set the pvid and untagged flags since we want to treat these ports as access ports (untagged), and assign their default VLAN (ID 2) on ingress (pvid). Remember to remove from their default VLAN (ID 1) as well:
# ip link set eth2 master br0 # ip link set eth3 master br0 # bridge vlan add vid 2 dev eth2 pvid untagged # bridge vlan add vid 2 dev eth3 pvid untagged # bridge vlan del vid 1 dev eth2 # bridge vlan del vid 1 dev eth3 To see static and learned MAC addresses (c.f. the arp command):
# bridge fdb show 00:80:e1:42:55:a3 dev eth0 vlan 1 master br0 permanent 00:80:e1:42:55:a3 dev eth0 master br0 permanent 33:33:00:00:00:01 dev eth0 self permanent 00:e0:4c:68:03:06 dev eth1 vlan 1 master br0 permanent 00:e0:4c:68:03:06 dev eth1 master br0 permanent 33:33:00:00:00:01 dev eth1 self permanent . 33:33:00:00:00:01 dev br0 self permanent In our use-case we have two different VLANs, so we need to change the bridge port itself to be a tagged VLAN member, otherwise we cannot distinguish between frames on different VLANs and thus cannot set up our VLAN interfaces on top, like this:
vlan1 vlan2 Layer-3 :: IP Networking \ / ------------------------------- br0 ______|_______ Layer-2 :: Switching [#_#_#_#_#_#_#] / | : | \ ------------------------------- eth0 eth1 : eth2 eth3 Layer-1 :: Link layer : VLAN 1 : VLAN 2 Let’s change br0 to be a tagged member of VLAN 1 and 2:
# bridge vlan add vid 1 dev br0 self # bridge vlan add vid 2 dev br0 self # bridge vlan show port vlan ids eth0 1 PVID Egress Untagged eth1 1 PVID Egress Untagged eth2 2 PVID Egress Untagged eth3 2 PVID Egress Untagged br0 1 2 Now we add our VLAN interface on top of br0 so we can communicate with the outside world. Some prefer naming VLAN interfaces br0.1 , but here we use vlan1 since we will only use one bridge:
# ip link add name vlan1 link br0 type vlan id 1 # ip addr add 192.168.1.1/24 dev vlan1 # ip link add name vlan2 link br0 type vlan id 2 # ip addr add 192.168.2.1/24 dev vlan2 Bring everything up by taking up the bridge and its ports:
# ip link set eth0 up # ip link set eth1 up # ip link set eth2 up # ip link set eth3 up # ip link set br0 up # ip link set vlan1 up # ip link set vlan2 up This is a good time to have a look at the available interfaces:
# ip -brief link show lo UNKNOWN 00:00:00:00:00:00 eth0 UP 00:80:e1:42:55:a0 eth1 UP 00:80:e1:42:55:a1 eth2 UP 00:80:e1:42:55:a2 eth3 UP 00:80:e1:42:55:a3 br0 UP 00:80:e1:42:55:a0 vlan1@br0 UP 00:80:e1:42:55:a0 vlan2@br0 UP 00:80:e1:42:55:a0
As you can see, the vlan1 interface is created on top of br0 , vlan1@br0 . The addresses of all interfaces can be inspected with the ip address command. For a quick overview, use the -brief switch:
# ip -br addr show lo UNKNOWN 127.0.0.1/8 eth0 UP eth1 UP eth2 UP eth3 UP br0 UP vlan1@br0 UP 192.168.1.1/24 vlan2@br0 UP 192.168.2.1/24 Here we have automatically configured IPv6 addresses on eth1 and br0, this should be disabled since IP addresses in a our bridge setup should only be set on the VLAN interfaces.
Summary and More
In this post we covered the theory of Ethernet bridges and VLANs, and then proceeded to provide an example of how to set this up a single bridge up in Linux.
But wait, what if we want to connect two separate bridges, on two PCs, with multiple VLANs on each? Let’s extend the image used previously, and add a syntax for denoting VLAN memberships: 1U means untagged member of VLAN 1, 2U means untagged in VLAN 2, and 1T means tagged member of VLAN 1, etc.
vlan1 vlan2 vlan1 vlan2 \ / \ / br0 1T,2T br0 1T,2T,3T ____|____ ____|__________ [#_#_#_#_#] [#_#_#_#_#_#_#_#] / | \ / | | \ \ eth2 eth1 eth0----------eth0 eth1 eth2 eth3 eth4 2U 1U 1T,2T 1U 2U 3U 3U The image shows two devices with one bridge each. The right-hand bridge has more ports and VLANs, but they are interconnected using port eth0 on each bridge. This shared link, VLAN “trunk” (see above), serves as the backbone for this network.
Notice how VLAN 3 only exists on the right-hand bridge, both bridges filter traffic going out and coming in on the trunk from port eth0, to prevent VLAN 3 from reaching beyond its boundary (port eth3 and eth4).
EOF
Future posts will cover how the Linux bridge can be used with single board computers that support switching in hardware, i.e., offloading of the otherwise CPU intensive parts.
Feel free to contact Westermo for more information, help designing your network, and hands on training on our products.
© Westermo Engineers — Powered by Jekyll & whiteglass — Subscribe via RSS
(Vlan-aware) Bridges on Linux
Lately I’ve had some conversations about how Linux sucks at bridging tagged VLANs into VMs, which just isn’t true anymore.
With recent Kernels Linux bridges have become vlan-aware and now allow configuring any bridge port like a port of any decent network switch with respect to 802.1q VLANS. A port can present a VLAN as untagged traffic as well as a number of VLANs in tagged mode. As can be expected, SVIs can be configured as vlan interfaces of a bridge, too.
The old brctl utility has been integrated within the iproute suite as part of of ip link. The commands map as follows:
brctl addbr br0 ip link add br0 type bridge [ forward_delay FORWARD_DELAY ] . [ vlan_filtering VLAN_FILTERING ] [ vlan_default_pvid VLAN_D_PVID ] . [ nf_call_iptables NF_CALL_IPT ] [ nf_call_ip6tables NF_CALL_IP6TABLES ] [ nf_call_arptables NF_CALL_ARPTABLES ] brctl addif br0 eth0 ip link set eth0 master br0
The full list of bridge related parameters of ip link contains all the glory:
$ ip link add br0 type bridge help Usage: . bridge [ forward_delay FORWARD_DELAY ] [ hello_time HELLO_TIME ] [ max_age MAX_AGE ] [ ageing_time AGEING_TIME ] [ stp_state STP_STATE ] [ priority PRIORITY ] [ group_fwd_mask MASK ] [ group_address ADDRESS ] [ vlan_filtering VLAN_FILTERING ] [ vlan_protocol VLAN_PROTOCOL ] [ vlan_default_pvid VLAN_DEFAULT_PVID ] [ mcast_snooping MULTICAST_SNOOPING ] [ mcast_router MULTICAST_ROUTER ] [ mcast_query_use_ifaddr MCAST_QUERY_USE_IFADDR ] [ mcast_querier MULTICAST_QUERIER ] [ mcast_hash_elasticity HASH_ELASTICITY ] [ mcast_hash_max HASH_MAX ] [ mcast_last_member_count LAST_MEMBER_COUNT ] [ mcast_startup_query_count STARTUP_QUERY_COUNT ] [ mcast_last_member_interval LAST_MEMBER_INTERVAL ] [ mcast_membership_interval MEMBERSHIP_INTERVAL ] [ mcast_querier_interval QUERIER_INTERVAL ] [ mcast_query_interval QUERY_INTERVAL ] [ mcast_query_response_interval QUERY_RESPONSE_INTERVAL ] [ mcast_startup_query_interval STARTUP_QUERY_INTERVAL ] [ nf_call_iptables NF_CALL_IPTABLES ] [ nf_call_ip6tables NF_CALL_IP6TABLES ] [ nf_call_arptables NF_CALL_ARPTABLES ] Where: VLAN_PROTOCOL :=
The vlan_filtering option of the iproute command maps to a not very well hidden magic switch in sysfs, which might be relevant for checking support for vlan-awareness:
echo 1 > /sys/class/net/br0/bridge/vlan_filtering
On Debian based systems I’d recommend using ifupdown2 for the configuration network interfaces. It’s nearly a drop-in replacement for classic ifupdown network configuration and is developed by the folks at CumulusNetworks, who among other things developed vlan-aware-ness for bridges. It’s no surprise that it comes with config options to configure vlan-aware bridges from within /etc/network/interfaces
auto br0 iface br0 bridge-ports bond0 bridge-vlan-aware yes bridge-vids 1013 4002 iface bond0 bridge-vids 100 101 200 201 1013 2000 [..] iface cr02_eth1 bridge-vids 1013 2000 2004 2006 3002 iface br0.1013 address 10.132.252.22/28
Adding/removing ports to/from the bridge manually could be done with the new bridge utility also shipped by the iproute suite:
bridge vlan < add | del >vid VLAN_ID dev DEV [ pvid ] [ untagged ] [ self ] [ master ] bridge vlan show [ dev DEV ] [ vid VLAN_ID ]
Firewalling / netfilter
Previously there were global options to enable/disable netfilter interaction with Linux bridges (for iptables, ip6tables, arptables). These have been moved to per-bridge options as indicated by the highlighted nf_call_* parameters shown in the ip link help output above. This might come in handy when handling multiple bridges with different firewalling requirements.
Applications
Vlan-aware bridges come in really handy, when running Linux-based virtualization hosts, for example using KVM. With a simple hook for KVM/libvirt installed as / etc/libvirt/hooks/qemu which uses the interface names from the libvirt VM configuration to configure untagged VLANs on VM interfaces. Using the following interface configuration and the simple hook above, the interface will be configured as a member of br0 as an untagged port serving VLAN 1013:
My talk on Contemporay Linux Networking from DENOG9 contains some more hands-on glimpses into current Linux networking features.
Leave a Reply Cancel reply
You must be logged in to post a comment.