Introduction
When you want to deploy Virtual Machines with multiple interfaces like ingress/egress, trusted/untrusted or production/management, it is always a mission to get the routing table in the VM correct.
In particular in a OpenStack/Linux environment, DHCP will be used to on every interface installing a default route on every interface. This leads to unpredictable and undesirable traffic forwarding patterns. This article describes how you can use Nuage Networks VSP to set DHCP options in such a way that only a single default route installed and other static routes are installed on top to steer traffic over the other interfaces.
The solution will basically leverage these DHCP options:
- DHCP Option 3 – Router Option – Setting this to a value of 0.0.0.0 will block the installation of a default route on the interface by the Linux Networking Stack.
- DHCP Option 121 — Classless Static Route Option – Setting this to a series of destination subnets and next-hops will properly install the static routes on Linux Networking Stack.
The example in this article will be show using both API and GUI.
Sample Network Topology
The target network topology is shown in the diagram below
To model this in Nuage Networks, a few subnets are created
- Domain Trusted – Subnets foo ( 10.10.10.0/24 ) and bar ( 10.10.11.0/24 )
- Domain Untrusted – Subnets UntrustedSN ( 20.20.20.0/24 )
And a couple of VMs are deployed in the domain in each domain.
Without any manipulation of DHCP options, the route table for the two VMs is as follows:
$ uname -n singlenic-vm $ ip -4 a 1: lo: mtu 16436 qdisc noqueue inet 127.0.0.1/8 scope host lo 2: eth0: mtu 1500 qdisc pfifo fast qlen 1000 inet 10.10.11.2/24 brd 10.10.11.255 scope global eth0 $ ip r default via 10.10.11.1 dev eth0 10.10.11.0/24 dev eth0 src 10.10.11.2 $ uname -n dualnic-vm $ ip -4 a 1: lo: mtu 16436 qdisc noqueue inet 127.0.0.1/8 scope host lo 2: eth0: mtu 1500 qdisc pfifo fast qlen 1000 inet 10.10.10.2/24 brd 10.10.10.255 scope global eth0 3: eth1: mtu 1500 qdisc pfifo fast qlen 1000 inet 20.20.20.2/24 brd 20.20.20.255 scope global eth1 $ ip route default via 20.20.20.1 dev eth1 default via 10.10.10.1 dev eth0 10.10.10.0/24 dev eth0 src 10.10.10.2 20.20.20.0/24 dev eth1 src 20.20.20.2 As can be seen, the second VM has two default routes installed, causing any outgoing traffic to go out over an unpredictable interface. In my case, it seemed like eth0 (via 10.10.10.1 ) was preferred by the Linux kernel.
Blocking installation of default route
A first step is to get rid of the undesirable default route. This can be done by setting DHCP Option 3 – Router to a value of 0.0.0.0 . DHCP Options can be configured on the Domain / Zone / Subnet level.
The screenshot below shows how to do this on the VSD Architect
The same can be done via the API / VSD CLI. The set of commands below show how to use the VSD CLI to accomplish this:
[root@os-controller ~]# vsd create dhcpoption --in subnet 4325ecde-e249-4aec-9146-f4f28be7b09c -p type=03 length=04 value=00000000 [Success] dhcpoption has been created with actualValues | [u'0.0.0.0'] | | length | 04 | | value | 00000000 | | entityScope | ENTERPRISE | | actualType | 3 | | parentType | subnet | | lastUpdatedBy | 8a6f0e20-a4db-4878-ad84-9cc61756cd5e | | externalID | | | lastUpdatedDate | 1485314316000 | | parentID | 4325ecde-e249-4aec-9146-f4f28be7b09c | | owner | 8a6f0e20-a4db-4878-ad84-9cc61756cd5e | | creationDate | 1485314316000 | | type | 03 | | ID | d934ca38-949b-4d15-ab65-491966341f96 | +-----------------+--------------------------------------+ This will prevent the local dhcp client to hand out any default route towards the VM over the interface in that particular subnet.
(note that you may have to verify this via VNC if this change has cut off your own access)
$ ip r default via 20.20.20.1 dev eth1 10.10.10.0/24 dev eth0 src 10.10.10.2 20.20.20.0/24 dev eth1 src 20.20.20.2 Setting a static route on VM
Setting a static route can be done with DHCP Option 121 — Classless Static Route Option since this allows specifying the CIDR and next hop.
As with any DHCP Option, it is to be coverted to HEX so it can immediately be passed into . For DHCP ption 121, the route needs to be converted for the API Values.
In our case we want to assign a route for the servers:
= 08 ; 10 ; 10.10.10.01 – CIDR Length ; CIDR without Zeros ; Next-Hop
= 080A0A0A0A01 – Cleaned up HEX value
The corresponding API call is as follows:
[root@os-controller ~]# vsd create dhcpoption --in subnet 4325ecde-e249-4aec-9146-f4f28be7b09c -p type=79 length=06 value=080A0A0A0A01 [Success] dhcpoption has been created with actualValues | [u'10.0.0.0/8', u'10.10.10.1'] | | length | 06 | | value | 080A0A0A0A01 | | entityScope | ENTERPRISE | | actualType | 121 | | parentType | subnet | | lastUpdatedBy | 8a6f0e20-a4db-4878-ad84-9cc61756cd5e | | externalID | | | lastUpdatedDate | 1485315170000 | | parentID | 4325ecde-e249-4aec-9146-f4f28be7b09c | | owner | 8a6f0e20-a4db-4878-ad84-9cc61756cd5e | | creationDate | 1485315170000 | | type | 79 | | ID | 016ea7ed-7ab8-4af2-a9b9-97f192513e86 | +-----------------+--------------------------------------+ The resulting setting in the GUI looks like this:
You may find that using the GUI for setting Option 121 is not that straightforward, especially when provisioning multiple static routes. For that, I would recommend using the API / SDK / VSD CLI instead.
Results
Once you are done, you can reset the DHCP client on the VM, and you will see the new static routes getting populated, with a PING to the single-nic-vm being successful.
$ ip r default via 20.20.20.1 dev eth1 10.10.10.0/24 dev eth0 src 10.10.10.2 20.20.20.0/24 dev eth1 src 20.20.20.2 $ ping 10.10.11.2 PING 10.10.11.2 (10.10.11.2): 56 data bytes ^C --- 10.10.11.2 ping statistics --- $ sudo cirros-dhcpc up eth0 udhcpc (v1.20.1) started Sending discover. Sending select for 10.10.10.2. Lease of 10.10.10.2 obtained, lease time 268435455 route: SIOCADDRT: Invalid argument packets transmitted, 0 packets received, 100% packet loss $ ip r default via 20.20.20.1 dev eth1 10.0.0.0/8 via 10.10.10.1 dev eth0 10.10.10.0/24 dev eth0 src 10.10.10.2 20.20.20.0/24 dev eth1 src 20.20.20.2 $ ping 10.10.11.2 PING 10.10.11.2 (10.10.11.2): 56 data bytes 64 bytes from 10.10.11.2: seq=0 ttl=62 time=60.965 ms 64 bytes from 10.10.11.2: seq=1 ttl=62 time=1.626 ms 64 bytes from 10.10.11.2: seq=2 ttl=62 time=1.456 ms ^C --- 10.10.11.2 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss Hopefully by now you have a better understanding on how to manage and push static routes via DHCP options when using Nuage Networks. Good luck trying this out for yourself !
Статические маршруты через DHCP
Настраиваем передачу статических маршрутов от isc-dhcp-server клиентам UNIX и Windows.
Добавляем в его конфиг опции:
Первая для Windows(поддерживается Windows XP/2003 и выше), вторая — в соответствии со стандартом. По своей сути опции одинаковы.
# MS routes: adds extras to supplement routers option option ms-classless-static-routes code 249 = array of unsigned integer 8; # RFC3442 routes: overrides routers option option rfc3442-classless-static-routes code 121 = array of unsigned integer 8; subnet 192.168.0.0 netmask 255.255.255.0 < option ms-classless-static-routes 23, 192,168,100 192,168,0,30; option rfc3442-classless-static-routes 23, 192,168,100, 192,168,0,30; option routers 192.168.0.200;
Приведённый пример соответствует маршруту на 192.168.100.0/23 через 192.168.0.30
В одной опции могут задаваться несколько маршрутов. Например, для такой таблицы маршрутов:
195.98.64.65 192.168.0.30 255.255.255.255 UGH 0 0 0 eth0 195.98.64.1 192.168.0.30 255.255.255.255 UGH 0 0 0 eth0 195.98.64.66 192.168.0.30 255.255.255.224 UG 0 0 0 eth0 213.184.232.32 192.168.0.30 255.255.255.224 UG 0 0 0 eth0 192.168.100.0 192.168.0.30 255.255.254.0 UG 0 0 0 eth0
опции будут выглядеть как
option ms-classless-static-routes 23, 192,168,100 192,168,0,30, 32, 195,98,64,65, 192,168,0,30, 32, 195,98,64,65, 192,168,0,30, 27, 27, 213,184,232,32, 192,168,0,30;
option rfc3442-classless-static-routes 23, 192,168,100 192,168,0,30, 32, 195,98,64,65, 192,168,0,30, 32, 195,98,64,65, 192,168,0,30, 27, 27, 213,184,232,32, 192,168,0,30;
CentOS 7 DHCP Client - How to use "classless-static-routes" ("code 121")?
According to this answer. https://unix.stackexchange.com/a/457577/61742 . I can use the following configuration to use pushed routes from my DHCP server in my DHCP client by placing in the /etc/dhclient.conf or /etc/dhcp3/dhclient.conf file the following configuration.
option classless-static-routes code 121 = array of < ip-address, ip-address >; [root@localhost ~]# ps -eaf | grep dhcli root 780 650 0 15:02 ? 00:00:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp0s8.pid -lf /var/lib/NetworkManager/dhclient-00cb8299-feb9-55b6-a378-3fdc720e0bc6-enp0s8.lease -cf /var/lib/NetworkManager/dhclient-enp0s8.conf enp0s8 root 783 650 0 15:02 ? 00:00:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp0s17.pid -lf /var/lib/NetworkManager/dhclient-8512e951-6012-c639-73b1-5b4d7b469f7f-enp0s17.lease -cf /var/lib/NetworkManager/dhclient-enp0s17.conf enp0s17 root 2218 1152 0 15:36 pts/0 00:00:00 grep --color=auto dhcli NOTE: Note the value for the "-cf" parameter (files: /var/lib/NetworkManager/dhclient-enp0s8.conf and /var/lib/NetworkManager/dhclient-enp0s17.conf ). My question is: What is the correct way to use "classless-static-routes" (code 121) on a CentOS 7 DHCP client? Thanks! PLUS: This is the configuration of my ISC KEA DHCP (DHCPv4).
cat /usr/local/etc/kea/kea-dhcp4.conf [. ] "option-def": [ < "name": "rfc3442-classless-static-routes", "code": 121, "space": "dhcp4", "type": "record", "record-types": "uint8,uint8,uint8,ipv4-address" >], "option-data": [< "name": "rfc3442-classless-static-routes", "data": "10,1,4, 10.1.6.4" >] [. ] The goal is that requests to 10.1.4.0/24 be routed to ip 10.1.6.4/32 in my DHCP clients. NOTE: Is the same thing I would get with the command ip route add 10.1.4.0/24 dev enp0s8 via 10.1.6.4 in each my DHCP clients.
How can I configure my DHCP server to distribute IP routes?
I have a DHCP server (Linux, IPCop, dnsmasq) running on my default gateway server 192.168.0.1. I have a VPN endpoint on another server in the network (192.168.0.4). Is it possible to configure the DHCP server so it sends not only the default gateway but also routing information for the VPN (192.168.1.*) to DHCP clients when they request an IP address? The DHCP clients run Windows and Linux. I tried to find something in the official documentation (http://www.ipcop.org/2.0.0/en/admin/html/custom-dnsmasq-local.html) but failed. The existence of RFC3442 indicates that it might be possible somehow.
2 Answers 2
This can be done by adding the following lines to dhcpd.conf :
option rfc3442-classless-static-routes code 121 = array of integer 8; option rfc3442-classless-static-routes 24, 192, 168, 1, 192, 168, 0, 4; option ms-classless-static-routes code 249 = array of integer 8; option ms-classless-static-routes 24, 192, 168, 1, 192, 168, 0, 4; This distributes a route entry for network 192.168.1.0/24 using the gateway 192.168.0.4.
The meaning of the bytes is (in brackets the value from the example above):
WW, D1, D2, D3, R1, R2, R3, R4 WW = destination network mask width (24) D1..D3 = destination network address (192.168.1.*) R1..R4 = router address (192.168.0.4) Note: The number of D1..DN bytes varies depending on the network mask. See RFC3442 for details.