Cumulus Linux VX is a wonderful tool for folks like us who would like to play around linux networking.
In this lab we will build simple Vxlan setup using cumulus, and getting some hands on on the fundamentals.
Lets start by building a basic L2 bridge first:
Using CumulusVX3.7.2, the simple bridge between swp5-6 is easily achieved with below command:
cumulus@sw1:~$ net add bridge bridge ports swp5-6
This one CLI will automatically create the below bridge interfaces in /etc/network/interfaces, hence the bridging once you do a “net commit”
+ auto bridge
+ iface bridge
+ bridge-ports swp5 swp6
+ bridge-vlan-aware yes
This saves you form manually editing those files, and good for people whom are new to linux networking (i.e. me)
The SVI is also done via one line
net add vlan 1 ip address 1.1.1.10/24
causing the vlan interface to be created again in /etc/network/interfaces
+
+auto bridge
+iface bridge
+ bridge-vids 1
+ bridge-vlan-aware yes
+
+auto vlan1
+iface vlan1
+ address 1.1.1.10/24
+ vlan-id 1
+ vlan-raw-device bridge
Quick check of connectivity:
Done. Now move on to build the simple 3 nodes topology.
I will be using ospf as IGP. For that, simply assign the loopback and ospf configs on each of the 3 cumulus switches
net add loopback lo 100.100.100.1/32
net add interface swp1 ip address 10.10.10.1/24
net add ospf router-id 1.0.0.1
net add ospf network 10.10.10.0/24 area 0
Once IGP is configured, the underlay is set and we can configure the Vxlan overlap for the 1.1.1.x/24 network.
VTEPs will be configured on sw1 and sw3. sw2 is not aware of the overlay but just transport, thus no changes required.
On sw3, we will create the vxlan 1 and statically define the VTEP endpoints (i.e. both local and remote end). We will use the loopback address for the VTEP.
Once the vxlan is configured, we will need to bridge the VTEP interface with the End host port (i.e. VPC3) so the overlay traffic can reach the VTEP:
Command
——- ————————– ——————————————————
net add vxlan vni-1 vxlan id 1
net add vxlan vni-1 vxlan local-tunnelip 100.100.100.3
net add vxlan vni-1 vxlan remoteip 100.100.100.1
net add vxlan vni-1 bridge access 1
net add bridge bridge ports swp6
——- ————————– ——————————————————
We would apply similar config on the other end (sw1), and wont repeat here.
Lets verify the configuration after all that.
cumulus@sw3:~$ net show bridge vlan
Interface VLAN Flags VNI
——— —- ——————— —
swp6 1 PVID, Egress Untagged
vni-1 1 PVID, Egress Untagged 1
cumulus@sw1:~$ net show bridge vlan
Interface VLAN Flags VNI
——— —- ——————— —
swp5 1 PVID, Egress Untagged
swp6 1 PVID, Egress Untagged
bridge 1
vni-1 1 PVID, Egress Untagged 1
cumulus@cumulus:~$ net show interface
State Name Spd MTU Mode LLDP Summary
—– —— — —– ———— ———- ——————–
UP lo N/A 65536 Loopback IP: 127.0.0.1/8
lo IP: 100.100.100.3/32
lo IP: ::1/128
UP eth0 1G 1500 Mgmt
UP swp1 1G 1500 Interface/L3 sw2 (swp2) IP: 20.20.20.1/24
UP swp6 1G 1500 Access/L2 Master: bridge(UP)
UP bridge N/A 1500 Bridge/L2
UP vni-1 N/A 1500 Access/L2 Master: bridge(UP)
The Vxlan configuration seems to be in the right place. However, when we try to verify the overlay connectivity, there is an issue:
VPCS-3> ping 1.1.1.1
host (1.1.1.1) not reachable
Lets verify the connectivity from sw3. Logically when VPC3 first sends a packet to VPC1, it would ARP first and sw3 would flood the ARP packet out. In this case, the frame will be encapsulated in Vxlan format and out to the remote VTEP tunnel (i.e. Sw1 VTEP)
cumulus@cumulus:~$ ping 100.100.100.1
connect: Network is unreachable
So the interfaces are up, but the remote VTEP is unreachable
show ip route
=============
Codes: K – kernel route, C – connected, S – static, R – RIP,
O – OSPF, I – IS-IS, B – BGP, E – EIGRP, N – NHRP,
T – Table, v – VNC, V – VNC-Direct, A – Babel, D – SHARP,
F – PBR,
> – selected route, * – FIB route
O>* 10.10.10.0/24 [110/200] via 20.20.20.2, swp1, 02:05:05
O 20.20.20.0/24 [110/100] is directly connected, swp1, 02:06:25
C>* 20.20.20.0/24 is directly connected, swp1, 02:06:26
C>* 100.100.100.3/32 is directly connected, lo, 01:18:16
It turns out the underlay network (i.e. IGP OSPF) does not know how to reach the remote VTEP, and traffic is simply dropped.
So now adding the loopback network in ospf and try again
sw3
show ip route
=============
Codes: K – kernel route, C – connected, S – static, R – RIP,
O – OSPF, I – IS-IS, B – BGP, E – EIGRP, N – NHRP,
T – Table, v – VNC, V – VNC-Direct, A – Babel, D – SHARP,
F – PBR,
> – selected route, * – FIB route
O>* 10.10.10.0/24 [110/200] via 20.20.20.2, swp1, 02:10:17
O 20.20.20.0/24 [110/100] is directly connected, swp1, 02:11:37
C>* 20.20.20.0/24 is directly connected, swp1, 02:11:38
O>* 100.100.100.1/32 [110/200] via 20.20.20.2, swp1, 00:00:18
O 100.100.100.3/32 [110/0] is directly connected, lo, 00:00:35
C>* 100.100.100.3/32 is directly connected, lo, 01:23:28
VPCS-3> ping 1.1.1.2
84 bytes from 1.1.1.2 icmp_seq=1 ttl=64 time=2.590 ms
84 bytes from 1.1.1.2 icmp_seq=2 ttl=64 time=6.060 ms
84 bytes from 1.1.1.2 icmp_seq=3 ttl=64 time=3.723 ms
84 bytes from 1.1.1.2 icmp_seq=4 ttl=64 time=5.991 ms
84 bytes from 1.1.1.2 icmp_seq=5 ttl=64 time=3.545 ms
Note sw3 does not have route/knowledge of 1.1.1.x/24. This is because that network has been encapsulated into the overlay vxlan tunnel.
A sniff on the wire would show the the vxlan header and User traffic has been tucked inside.
Open sandbox 