This is an explanation how to configure vMX routers for VXLAN with Multicast control plane. The configuration can be used on MX routers with maybe some small adaptations. I didn’t try to run this config on a hardware MX router.
For people interested in getting the JNCIP-DC certification. This is one of the topics required for the exam. However the exam focusses on QFX switches and the configuration is a bit different.
I’m using the vmx virtual router which you can just download from the Juniper website.
It might sound obvious, but also use the trial license on your vmx. Some things work without and somethings don’t. Don’t waste your time figuring out what doesn’t and just register the trial license.
Scheme
This will be our setup. The top routers fuction as spine devices. The routers down are functioning as leaf routers. We are using manual vxlan, so we need to configure multicast on this network too. Vmx1 will act as the RP for our multicast group.
Config
Here is the config in full. I will try to explain block by block what is done. This configuration is from vmx1. This is a spine device. This router also functions as the Rendezvous Point of our multicast tree. I tried to keep this config to the very minimum. This is just for lab and testing purposes. Please include necessary security and monitoring configuration to production configurations.
system { host-name vmx1; root-authentication { encrypted-password ""; ## SECRET-DATA } services { ssh; } } chassis { fpc 0 { pic 0 { tunnel-services { bandwidth 1g; } } lite-mode; } } interfaces { ge-0/0/0 { disable; } ge-0/0/1 { unit 0 { family inet { address 172.16.5.2/24; } } } ge-0/0/3 { description "to vmx4 - 172.16.3.0/24"; unit 0 { family inet { address 172.16.3.2/24; } } } ge-0/0/4 { unit 0 { family inet { address 172.16.1.2/24; } } } fxp0 { unit 0 { family inet { address 192.168.1.11/24; } } } lo0 { unit 0 { family inet { address 172.16.100.12/32; } } } } routing-options { autonomous-system 64512; forwarding-table { export loadbalance; } } protocols { bgp { group underlay { type external; export direct; multipath multiple-as; neighbor 172.16.5.1 { peer-as 64516; } neighbor 172.16.3.1 { peer-as 64515; } neighbor 172.16.1.1 { peer-as 64514; } } } pim { rp { local { address 172.16.100.12; group-ranges { 224.0.0.0/4; } } } interface fxp0.0 { disable; } interface ge-0/0/1.0 { mode sparse; } interface lo0.0 { mode sparse; } interface ge-0/0/3.0 { mode sparse; } interface ge-0/0/4.0 { mode sparse; } } lldp { interface all; } } policy-options { policy-statement direct { term t1 { from protocol direct; then accept; } } policy-statement loadbalance { then { load-balance per-packet; } } } switch-options { vtep-source-interface lo0.0; }
System
Nothing special here. We put in a hostname and a root password. For remote management I enabled ssh.
system { host-name vmx1; root-authentication { encrypted-password ""; ## SECRET-DATA } services { ssh; } }
Chassis
Tunnel services need to be enabled to provide encapsulation capabilities. It will enable the vtep interface.
And the lite-mode is something specific for vMX. It’s to make the vMX use less resources. You can read about it here.
chassis { fpc 0 { pic 0 { tunnel-services { bandwidth 1g; } } lite-mode; } }
Interfaces
Configuration of the interfaces. Nothing special here either. We configure an ipv4 address to the neighbors and for the lo0 and fxp0 interface.
interfaces { ge-0/0/0 { disable; } ge-0/0/1 { unit 0 { family inet { address 172.16.5.2/24; } } } ge-0/0/3 { description "to vmx4 - 172.16.3.0/24"; unit 0 { family inet { address 172.16.3.2/24; } } } ge-0/0/4 { unit 0 { family inet { address 172.16.1.2/24; } } } fxp0 { unit 0 { family inet { address 192.168.1.11/24; } } } lo0 { unit 0 { family inet { address 172.16.100.12/32; } } } }
Routing options
Because we are using eBGP routing the AS numbers are important. Every router in the network should have it’s own AS number.
The forwarding-table statement will allow us to loadbalance traffic in the network (when multiple paths are available).
routing-options { autonomous-system 64512; forwarding-table { export loadbalance; } }
Protocols
protocols { bgp { group underlay { type external; export direct; multipath multiple-as; neighbor 172.16.5.1 { peer-as 64516; } neighbor 172.16.3.1 { peer-as 64515; } neighbor 172.16.1.1 { peer-as 64514; } } } pim { rp { local { address 172.16.100.12; group-ranges { 224.0.0.0/4; } } } interface fxp0.0 { disable; } interface ge-0/0/1.0 { mode sparse; } interface lo0.0 { mode sparse; } interface ge-0/0/3.0 { mode sparse; } interface ge-0/0/4.0 { mode sparse; } } lldp { interface all; } }
BGP
We are using eBGP in this setup. The BGP will distribute the routes between all the routers.
Since we are using eBgp, we have to indicate this in the configuration with the “type external” command.
The multipath mutiple-as command allows the router to install multiple AS paths in the routing tables. This way you can load balance traffic between different links.
Juniper article about bgp multipath
Multipath example
root@vmx5> show route 172.16.100.16 inet.0: 18 destinations, 30 routes (18 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.16.100.16/32 *[BGP/170] 00:08:13, localpref 100, from 172.16.1.2 AS path: 64512 64516 I, validation-state: unverified to 172.16.1.2 via ge-0/0/0.0 > to 172.16.2.2 via ge-0/0/1.0 [BGP/170] 00:08:13, localpref 100 AS path: 64513 64516 I, validation-state: unverified > to 172.16.2.2 via ge-0/0/1.0 root@vmx5> show route forwarding-table destination 172.16.100.16 Routing table: default.inet Internet: Enabled protocols: Bridging, Destination Type RtRef Next hop Type Index NhRef Netif 172.16.100.16/32 user 0 ulst 1048574 3 172.16.1.2 ucst 595 8 ge-0/0/0.0 172.16.2.2 ucst 599 7 ge-0/0/1.0 root@vmx5> configure Entering configuration mode [edit] root@vmx5# deactivate protocols bgp group underlay multipath [edit] root@vmx5# commit and-quit commit complete Exiting configuration mode root@vmx5> show route 172.16.100.16 inet.0: 18 destinations, 30 routes (18 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 172.16.100.16/32 *[BGP/170] 00:00:03, localpref 100 AS path: 64512 64516 I, validation-state: unverified > to 172.16.1.2 via ge-0/0/0.0 [BGP/170] 00:08:50, localpref 100 AS path: 64513 64516 I, validation-state: unverified > to 172.16.2.2 via ge-0/0/1.0 root@vmx5> show route forwarding-table destination 172.16.100.16 Routing table: default.inet Internet: Enabled protocols: Bridging, Destination Type RtRef Next hop Type Index NhRef Netif 172.16.100.16/32 user 0 172.16.1.2 ucst 595 9 ge-0/0/0.0
PIM
We need to configure a rendezvous point. The rendezvous point (RP) router is the root of this shared tree and receives the multicast traffic from the source. All receivers will send a PIM join message to the RP. This will build a tree (rendezvous point tree, RPT). The RPT remains in effect, even if no active source generate traffic.
Juniper article about PIM Sparse Mode
You need to configure all interfaces in the sparse mode who have to transfer vxlan traffic. The fxp interface has to be disabled cause it cannot exchange traffic.
PIM Join when no mac addressses are learned
root@vmx5> show pim join Instance: PIM.master Family: INET R = Rendezvous Point Tree, S = Sparse, W = Wildcard Group: 239.0.0.1 Source: * RP: 172.16.100.12 Flags: sparse,rptree,wildcard Upstream interface: ge-0/0/0.0
PIM Join after mac addresses are learned
root@vmx5> show pim join Instance: PIM.master Family: INET R = Rendezvous Point Tree, S = Sparse, W = Wildcard Group: 239.0.0.1 Source: * RP: 172.16.100.12 Flags: sparse,rptree,wildcard Upstream interface: ge-0/0/0.0 Group: 239.0.0.1 Source: 172.16.100.14 Flags: sparse,spt Upstream interface: Local Group: 239.0.0.1 Source: 172.16.100.16 Flags: sparse,spt Upstream interface: ge-0/0/0.0
root@vmx5> show bridge mac-table MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC) Routing instance : default-switch Bridging domain : v100, VLAN : 100 MAC MAC Logical NH MAC address flags interface Index property 00:0c:29:47:6c:20 D ge-0/0/2.0 00:0c:29:47:6c:f8 D vtep.32769
Policy options
policy-options { policy-statement direct { term t1 { from protocol direct; then accept; } } policy-statement loadbalance { then { load-balance per-packet; } } }
Policy statement direct will be used to redistribute the directly connected networks into BGP.
Policy statement loadbalance is required to make the routers loadbalance traffic between the available links.
Switch options
switch-options { vtep-source-interface lo0.0; }
Configure a source interface for a Virtual Extensible LAN (VXLAN) tunnel. You must provide the name of a logical interface configured on the loopback interface.
If you don’t configure this your vtep interface won’t work:
root@vmx1> show interfaces vtep Physical interface: vtep, Enabled, Physical link is Up Interface index: 133, SNMP ifIndex: 510 Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited Device flags : Present Running Link type : Full-Duplex Link flags : None Last flapped : Never Input packets : 0 Output packets: 0
Your vtep interface will actually become active when you connect it to a source interface in the switch options:
root@vmx1# activate switch-options [edit] root@vmx1# show | compare [edit] ! active: switch-options { ... } [edit] root@vmx1# commit and-quit shcommit complete Exiting configuration mode root@vmx1> show interfaces vtep Physical interface: vtep, Enabled, Physical link is Up Interface index: 133, SNMP ifIndex: 510 Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited Device flags : Present Running Link type : Full-Duplex Link flags : None Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vtep.32768 (Index 328) (SNMP ifIndex 541) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 Ethernet segment value: 00:00:00:00:00:00:00:00:00:00, Mode: single-homed, Multi-homed status: Blocking VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 172.16.100.12, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 0
Leaf config
Here is the config of a leaf switch. In our example router vmx3. Here is also a test host connected.
I will try to explain the differences.
system { host-name vmx3; root-authentication { encrypted-password ""; ## SECRET-DATA } services { ssh; } } chassis { fpc 0 { pic 0 { tunnel-services { bandwidth 1g; } } lite-mode; } } interfaces { ge-0/0/0 { unit 0 { family inet { address 172.16.5.1/24; } } } ge-0/0/1 { unit 0 { family inet { address 172.16.6.1/24; } } } ge-0/0/2 { description "to testhost 1"; encapsulation ethernet-bridge; unit 0 { family bridge; } } fxp0 { unit 0 { family inet { address 192.168.1.13/24; } } } lo0 { unit 0 { family inet { address 172.16.100.16/32; } } } } routing-options { autonomous-system 64516; forwarding-table { export loadbalance; } } protocols { bgp { group underlay { type external; export direct; multipath multiple-as; neighbor 172.16.5.2 { peer-as 64512; } neighbor 172.16.6.2 { peer-as 64513; } } } pim { rp { static { address 172.16.100.12 { group-ranges { 224.0.0.0/4; } } } } interface fxp0.0 { disable; } interface lo0.0 { mode sparse; } interface ge-0/0/0.0 { mode sparse; } interface ge-0/0/1.0 { mode sparse; } } lldp { interface all; } } policy-options { policy-statement direct { term 10 { from protocol direct; then accept; } } policy-statement loadbalance { then { load-balance per-packet; } } } bridge-domains { v100 { domain-type bridge; vlan-id 100; interface ge-0/0/2.0; vxlan { vni 1000; multicast-group 239.0.0.1; } } } switch-options { vtep-source-interface lo0.0; }
Protocols
PIM
pim { rp { static { address 172.16.100.12 { group-ranges { 224.0.0.0/4; } } } } interface fxp0.0 { disable; } interface lo0.0 { mode sparse; } interface ge-0/0/0.0 { mode sparse; } interface ge-0/0/1.0 { mode sparse; } }
So in the PIM we need to join the rendezvous point for the used multicast groups.
Bridge domains
bridge-domains { v100 { domain-type bridge; vlan-id 100; interface ge-0/0/2.0; vxlan { vni 1000; multicast-group 239.0.0.1; } } }
So in the bridge domain configuration we actually connect interfaces with vlans and vxlans. Vlan id’s are only significant to their bridge domains. You can bind a vlan 100 to vni 1000 on vmx3 and connect a completely different vlan. The vni needs to be binded on a multicast group. This needs to be the same multicast group on all routers for the specific vni.
Untagged interface
Had to look for this, but apparently this i show you make an untagged interface on MX. Actually pretty obvious 🙂
ge-0/0/2 { description "to testhost1"; encapsulation ethernet-bridge; unit 0 { family bridge; } }
Conclusion
So the idea was to share you my labconfig on vmx. You can use this to get a better understanding of vxlan or as a study tool.
I’ll try to share a config for vxlan with evpn signaling later.