Linux Switching – Interconnecting Namespaces
Switching in software on Linux is one of the important parts when using virtualization technologies like KVM or LXC. Typical hosts do not provide one or more physical adapters for each NIC of a virtual machine in KVM or per container when using LXC. Something else must take the part to interconnect the virtual network interfaces.
The software switching classical tool is the linuxbridge, which is available in the Linux kernel for a long time. The frontend to manage the linuxbridge is brctl. The newer tool is the Openvswitch (at http://openvswitch.org/). The main frontend is ovs-vsctl.
In this post I will show multiple solutions to interconnect Linux namespaces using a software based switch. A performance analysis of these solutions will be discussed in another article later. Starting with network namespaces helps to understand the more complex situations when using KVM or LXC.
tap interfaces
Linux tap interfaces created with ip tuntap cannot be used to attach network namespaces to linuxbridges or the openvswitch.
veth pair
The simple solution to connect two network namespaces is the usage of one veth pair. This has been discussed in a previous artice.
Connecting namespaces using a veth pair
The command sequence has been discussed in a previous article, but we show the commands here again
# add the namespaces ip netns add ns1 ip netns add ns2 # create the veth pair ip link add tap1 type veth peer name tap2 # move the interfaces to the namespaces ip link set tap1 netns ns1 ip link set tap2 netns ns2 # bring up the links ip netns exec ns1 ip link set dev tap1 up ip netns exec ns2 ip link set dev tap2 up # now assign the ip addresses
linux bridge and two veth pairs
When more than two network namespaces (or KVM or LXC instances) must be connected a switch should be used. Linux offers as one solution the well known linux bridge.
Connecting namespaces using a linux bridge and two veth pairs
We need for this setup one switch, and two connectors. In this setup we use a linuxbridge and two veth pairs.
The commands to create this setup are:
# add the namespaces ip netns add ns1 ip netns add ns2 # create the switch BRIDGE=br-test brctl addbr $BRIDGE brctl stp $BRIDGE off ip link set dev $BRIDGE up # #### PORT 1 # create a port pair ip link add tap1 type veth peer name br-tap1 # attach one side to linuxbridge brctl addif br-test br-tap1 # attach the other side to namespace ip link set tap1 netns ns1 # set the ports to up ip netns exec ns1 ip link set dev tap1 up ip link set dev br-tap1 up # #### PORT 2 # create a port pair ip link add tap2 type veth peer name br-tap2 # attach one side to linuxbridge brctl addif br-test br-tap2 # attach the other side to namespace ip link set tap2 netns ns2 # set the ports to up ip netns exec ns2 ip link set dev tap2 up ip link set dev br-tap2 up #
openvswitch and two veth pairs
Another solution is to use the openvswitch instead of the „old“ linuxbrige. The configuration is nearly the same as for the linuxbridge.
Connecting namespaces using the openvswitch and two veth pairs
We need for this setup one switch, and two connectors. In this setup we use an openvswitch and two veth pairs.
The commands to create this setup are:
# add the namespaces ip netns add ns1 ip netns add ns2 # create the switch BRIDGE=ovs-test ovs-vsctl add-br $BRIDGE # #### PORT 1 # create a port pair ip link add tap1 type veth peer name ovs-tap1 # attach one side to ovs ovs-vsctl add-port $BRIDGE ovs-tap1 # attach the other side to namespace ip link set tap1 netns ns1 # set the ports to up ip netns exec ns1 ip link set dev tap1 up ip link set dev ovs-tap1 up # #### PORT 2 # create a port pair ip link add tap2 type veth peer name ovs-tap2 # attach one side to ovs ovs-vsctl add-port $BRIDGE ovs-tap2 # attach the other side to namespace ip link set tap2 netns ns2 # set the ports to up ip netns exec ns2 ip link set dev tap2 up ip link set dev ovs-tap2 up #
openvswitch and two openvswitch ports
Another solution is to use the openvswitch and make use of the openvswitch internal ports. This avoids the usage of the veth pairs, which must be used in all other solutions.
Connecting namespaces using the openvswitch and two openvswitch ports
We need for this setup one switch, and two connectors. In this setup we use an openvswitch and two openvswitch ports.
The commands to create this setup are:
# add the namespaces ip netns add ns1 ip netns add ns2 # create the switch BRIDGE=ovs-test ovs-vsctl add-br $BRIDGE # #### PORT 1 # create an internal ovs port ovs-vsctl add-port $BRIDGE tap1 -- set Interface tap1 type=internal # attach it to namespace ip link set tap1 netns ns1 # set the ports to up ip netns exec ns1 ip link set dev tap1 up # #### PORT 2 # create an internal ovs port ovs-vsctl add-port $BRIDGE tap2 -- set Interface tap2 type=internal # attach it to namespace ip link set tap2 netns ns2 # set the ports to up ip netns exec ns2 ip link set dev tap2 up
Performance
In another article I will show some performance numbers for the four presented solutions. There are noticeable differences with respect to throughput and CPU usage.
Linux server as an unmanaged switch
I have a small Ubuntu x64 server, which provides dhcp, naming, file and such services to my home network. It’s simply connected to unmanaged SOHO switch laying next to it and just works. But, it has unused 4xGbE i350 NIC card in PCIe slot and I had this idea of using this NIC instead of the current switch. It probably won’t have any advantages if it will be also plain unmanaged switch like the current dedicated device but I wanted to learn something (and I could maybe turn it into managed switch later, maybe implement some IPS/IDS and further stuff). How I was surprised to find out I have no idea how. I did my share of searching of course, but it left me confused even more. First of all, why all howtos and articles configure bridges to achieve switching? With unmanaged switch all ports are equal, has no IPs and one of them leads to gateway. Is that even possible to create that on Linux server (ignoring it may not be really useful considering low price of a small switch)?
Clearly. But that was my question, is it the only possible way? Bridged iface has an IP address and requires ip forwarding. While it mostly provides the comparable service, it is not the same thing.
What do you mean by that? A bridge does not require either an IP address or IP forwarding. Though virtually everyone actually does use IP on their networks. I would be very surprised if you were an exception, since all of the services you mention use IP.
There’s clearly something about the concept I do not understand. Small switch uses no IP addresses, it’s completely transparent and does all the work on L2, working simply with MAC addresses. Now, when I go and configure Linux bridge across multiple ports, I get a a new, virtual interface. It has an IP address and I believe it needs routing/kernel IP forwarding. If that’s true, they are basically very different devices (though providing comparable service). As I said, it is not about making something work, it’s about understanding how things work inside so that difference is important.
1 Answer 1
The bridge is a software implementation of a Layer 2 switch, where each interface added into the bridge is a switchport.
Let’s say you have your 4 i350 ports in bridge br0 with two other PCs plugged in:
[ Remote PC A - MAC aa:aa:aa:aa:aa ] ---- [ i350 #1 ] ---- .------------. [ i350 #2 ] ---- | Linux PC | [ i350 #3 ] ---- | bridge br0 | [ Remote PC B - MAC bb:bb:bb:bb:bb ] ---- [ i350 #4 ] ---- '------------' Remote PC A tries to reach Remote PC B.
PC A will ARP out to the broadcast destination MAC address (all ff ) with its own source MAC (all aa ).
The bridge will receive that ARP Request and learn that MAC aa is down bridgeport i350 #1 .
The bridge then forwards that broadcast out all the other i350 ports, because that is the correct behaviour when a switch recieves a broadcast packet.
Remote PC B receives the ARP, and generates a reply to PC A with destination MAC of PC A (all aa ) and its own source MAC (all bb ).
The bridge will receive that ARP Reply and learn that MAC bb is down bridgeport i350 #4 .
The bridge sees the destination MAC is aa , and the bridge knows that MAC aa is available down port i350 #1 and so sends the frame out that NIC.
You can add an IP address to a bridge and use it like a regular network interface, route with it, NAT with it, etc but you don’t have to.
The above happens even if kernel tunable net.ipv4.ip_forward=0 is set, because the bridge driver is not performing IP forwarding or IP routing or IP NAT, it is performing Layer 2 packet switching like any other unmanaged Layer 2 switch.
You can also run Spanning Tree on the bridge if you are connecting it to a larger network.
You said you’re using Ubuntu. The following should work for Ubuntu 20.04 using NetworkManager:
/etc/NetworkManager/system-connections/br0.nmconnection [connection] id=br0 type=bridge interface-name=br0 permissions= [bridge] stp=false [ipv4] dns-search= ignore-auto-dns=true ignore-auto-routes=true method=disabled never-default=true [ipv6] addr-gen-mode=stable-privacy dns-search= ignore-auto-dns=true ignore-auto-routes=true method=ignore never-default=true [proxy] Repeat this file for each i350 named netX , using each NIC’s correct MAC address:
/etc/NetworkManager/system-connections/netX.nmconnection [connection] id=netX type=ethernet interface-name=netX master=br0 metered=2 permissions= slave-type=bridge [ethernet] mac-address=xx:xx:xx:xx:xx:xx mac-address-blacklist= [bridge-port] Configure linux machine as bridge/switch and end device
At my home, I have two desktop PCs in two rooms. The router / DSL modem is in one of these rooms. Now I want to configure a home server (having 2 LAN ports, running 24/7) in the corridor between the two rooms, using only one LAN cable at each door. This gives me the following physical configuration:
(door) (door) .----/-/----. .-----/-/----------._ FritzBox | | | .----´´ DSL Router PC1 Server | PC2 As just said, the server has 2 network interfaces and is running Ubuntu. What I need now is a network configuration which enables both the server and PC1 to connect to the router. I think the server needs to serve as a bridge or switch. Currently, all computers are configured having static IP addresses. If I’m understanding it correctly, a bridge / switch doesn’t have its own IP address, but as the server needs to be configured as an own end device, it needs to have one. My first question is, do I have to configure both interfaces separately, giving both the same static IP address? My next question is, how do I bridge the two physical networks into one? I have basic understanding (but am always confused again and again) of bridges and switches, but I don’t know how to configure it in software. I only know that it’s possible to do so 🙂 The third question is: Is it possible to configure this in a way that network packets from/to PC1 to/from the router only go through hardware or only consume low CPU in the server? Can you help me? Thanks in advance!