https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js

vxlan / multicast on vmx

This is an explanation how to configure vMX routers for VXLAN with Multicast control plane. The configuration can be used on MX routers with maybe some small adaptations. I didn’t try to run this config on a hardware MX router.

For people interested in getting the JNCIP-DC certification. This is one of the topics required for the exam. However the exam focusses on QFX switches and the configuration is a bit different.

I’m using the vmx virtual router which you can just download from the Juniper website.

It might sound obvious, but also use the trial license on your vmx. Some things work without and somethings don’t. Don’t waste your time figuring out what doesn’t and just register the trial license.

Scheme

This will be our setup. The top routers fuction as spine devices. The routers down are functioning as leaf routers. We are using manual vxlan, so we need to configure multicast on this network too. Vmx1 will act as the RP for our multicast group.

Config

Here is the config in full. I will try to explain block by block what is done. This configuration is from vmx1. This is a spine device. This router also functions as the Rendezvous Point of our multicast tree. I tried to keep this config to the very minimum. This is just for lab and testing purposes. Please include necessary security and monitoring configuration to production configurations.

system {
    host-name vmx1;
    root-authentication {
        encrypted-password ""; ## SECRET-DATA
    }
    services {
        ssh;
    }
}
chassis {
    fpc 0 {
        pic 0 {
            tunnel-services {
                bandwidth 1g;
            }
        }
        lite-mode;
    }
}
interfaces {
    ge-0/0/0 {
        disable;
    }
    ge-0/0/1 {
        unit 0 {
            family inet {
                address 172.16.5.2/24;
            }
        }
    }
    ge-0/0/3 {
        description "to vmx4 - 172.16.3.0/24";
        unit 0 {
            family inet {
                address 172.16.3.2/24;
            }
        }
    }
    ge-0/0/4 {
        unit 0 {
            family inet {
                address 172.16.1.2/24;
            }
        }
    }
    fxp0 {
        unit 0 {
            family inet {
                address 192.168.1.11/24;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 172.16.100.12/32;
            }
        }
    }
}
routing-options {
    autonomous-system 64512;
    forwarding-table {
        export loadbalance;
    }
}
protocols {
    bgp {
        group underlay {
            type external;
            export direct;
            multipath multiple-as;
            neighbor 172.16.5.1 {
                peer-as 64516;
            }
            neighbor 172.16.3.1 {
                peer-as 64515;
            }
            neighbor 172.16.1.1 {
                peer-as 64514;
            }
        }
    }
    pim {
        rp {
            local {
                address 172.16.100.12;
                group-ranges {
                    224.0.0.0/4;
                }
            }
        }
        interface fxp0.0 {
            disable;
        }
        interface ge-0/0/1.0 {
            mode sparse;
        }
        interface lo0.0 {
            mode sparse;
        }
        interface ge-0/0/3.0 {
            mode sparse;
        }
        interface ge-0/0/4.0 {
            mode sparse;
        }
    }
    lldp {
        interface all;
    }
}
policy-options {
    policy-statement direct {
        term t1 {
            from protocol direct;
            then accept;
        }
    }
    policy-statement loadbalance {
        then {
            load-balance per-packet;
        }
    }
}
switch-options {
    vtep-source-interface lo0.0;
}

System

Nothing special here. We put in a hostname and a root password. For remote management I enabled ssh.

system {
    host-name vmx1;
    root-authentication {
        encrypted-password ""; ## SECRET-DATA
    }
    services {
        ssh;
    }
}

Chassis

Tunnel services need to be enabled to provide encapsulation capabilities. It will enable the vtep interface.

And the lite-mode is something specific for vMX. It’s to make the vMX use less resources. You can read about it here.

chassis {
    fpc 0 {
        pic 0 {
            tunnel-services {
                bandwidth 1g;
            }
        }
        lite-mode;
    }
}

Interfaces

Configuration of the interfaces. Nothing special here either. We configure an ipv4 address to the neighbors and for the lo0 and fxp0 interface.

interfaces {
    ge-0/0/0 {
        disable;
    }
    ge-0/0/1 {
        unit 0 {
            family inet {
                address 172.16.5.2/24;
            }
        }
    }
    ge-0/0/3 {
        description "to vmx4 - 172.16.3.0/24";
        unit 0 {
            family inet {
                address 172.16.3.2/24;
            }
        }
    }
    ge-0/0/4 {
        unit 0 {
            family inet {
                address 172.16.1.2/24;
            }
        }
    }
    fxp0 {
        unit 0 {
            family inet {
                address 192.168.1.11/24;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 172.16.100.12/32;
            }
        }
    }
}

Routing options

Because we are using eBGP routing the AS numbers are important. Every router in the network should have it’s own AS number.

The forwarding-table statement will allow us to loadbalance traffic in the network (when multiple paths are available).

routing-options {
    autonomous-system 64512;
    forwarding-table {
        export loadbalance;
    }
}

Protocols

protocols {
    bgp {
        group underlay {
            type external;
            export direct;
            multipath multiple-as;
            neighbor 172.16.5.1 {
                peer-as 64516;
            }
            neighbor 172.16.3.1 {
                peer-as 64515;
            }
            neighbor 172.16.1.1 {
                peer-as 64514;
            }
        }
    }
    pim {
        rp {
            local {
                address 172.16.100.12;
                group-ranges {
                    224.0.0.0/4;
                }
            }
        }
        interface fxp0.0 {
            disable;
        }
        interface ge-0/0/1.0 {
            mode sparse;
        }
        interface lo0.0 {
            mode sparse;
        }
        interface ge-0/0/3.0 {
            mode sparse;
        }
        interface ge-0/0/4.0 {
            mode sparse;
        }
    }
    lldp {
        interface all;
    }
}

BGP

We are using eBGP in this setup. The BGP will distribute the routes between all the routers.

Since we are using eBgp, we have to indicate this in the configuration with the “type external” command.

The multipath mutiple-as command allows the router to install multiple AS paths in the routing tables. This way you can load balance traffic between different links.

Juniper article about bgp multipath

Multipath example
root@vmx5> show route 172.16.100.16

inet.0: 18 destinations, 30 routes (18 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.100.16/32   *[BGP/170] 00:08:13, localpref 100, from 172.16.1.2
                      AS path: 64512 64516 I, validation-state: unverified
                      to 172.16.1.2 via ge-0/0/0.0
                    > to 172.16.2.2 via ge-0/0/1.0
                    [BGP/170] 00:08:13, localpref 100
                      AS path: 64513 64516 I, validation-state: unverified
                    > to 172.16.2.2 via ge-0/0/1.0

root@vmx5> show route forwarding-table destination 172.16.100.16
Routing table: default.inet
Internet:
Enabled protocols: Bridging,
Destination        Type RtRef Next hop           Type Index    NhRef Netif
172.16.100.16/32   user     0                    ulst  1048574     3
                              172.16.1.2         ucst      595     8 ge-0/0/0.0
                              172.16.2.2         ucst      599     7 ge-0/0/1.0


root@vmx5> configure
Entering configuration mode

[edit]
root@vmx5# deactivate protocols bgp group underlay multipath

[edit]
root@vmx5# commit and-quit
commit complete
Exiting configuration mode

root@vmx5> show route 172.16.100.16

inet.0: 18 destinations, 30 routes (18 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.100.16/32   *[BGP/170] 00:00:03, localpref 100
                      AS path: 64512 64516 I, validation-state: unverified
                    > to 172.16.1.2 via ge-0/0/0.0
                    [BGP/170] 00:08:50, localpref 100
                      AS path: 64513 64516 I, validation-state: unverified
                    > to 172.16.2.2 via ge-0/0/1.0

root@vmx5> show route forwarding-table destination 172.16.100.16
Routing table: default.inet
Internet:
Enabled protocols: Bridging,
Destination        Type RtRef Next hop           Type Index    NhRef Netif
172.16.100.16/32   user     0 172.16.1.2         ucst      595     9 ge-0/0/0.0

PIM

We need to configure a rendezvous point. The rendezvous point (RP) router is the root of this shared tree and receives the multicast traffic from the source. All receivers will send a PIM join message to the RP. This will build a tree (rendezvous point tree, RPT). The RPT remains in effect, even if no active source generate traffic.

Juniper article about PIM Sparse Mode

You need to configure all interfaces in the sparse mode who have to transfer vxlan traffic. The fxp interface has to be disabled cause it cannot exchange traffic.

PIM Join when no mac addressses are learned
root@vmx5> show pim join
Instance: PIM.master Family: INET
R = Rendezvous Point Tree, S = Sparse, W = Wildcard

Group: 239.0.0.1
    Source: *
    RP: 172.16.100.12
    Flags: sparse,rptree,wildcard
    Upstream interface: ge-0/0/0.0
PIM Join after mac addresses are learned
root@vmx5> show pim join
Instance: PIM.master Family: INET
R = Rendezvous Point Tree, S = Sparse, W = Wildcard

Group: 239.0.0.1
Source: *
RP: 172.16.100.12
Flags: sparse,rptree,wildcard
Upstream interface: ge-0/0/0.0

Group: 239.0.0.1
Source: 172.16.100.14
Flags: sparse,spt
Upstream interface: Local

Group: 239.0.0.1
Source: 172.16.100.16
Flags: sparse,spt
Upstream interface: ge-0/0/0.0
root@vmx5> show bridge mac-table

MAC flags       (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
    O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)

Routing instance : default-switch
 Bridging domain : v100, VLAN : 100
   MAC                 MAC      Logical          NH     MAC
   address             flags    interface        Index  property
   00:0c:29:47:6c:20   D        ge-0/0/2.0
   00:0c:29:47:6c:f8   D        vtep.32769

Policy options

policy-options {
    policy-statement direct {
        term t1 {
            from protocol direct;
            then accept;
        }
    }
    policy-statement loadbalance {
        then {
            load-balance per-packet;
        }
    }
}

Policy statement direct will be used to redistribute the directly connected networks into BGP.

Policy statement loadbalance is required to make the routers loadbalance traffic between the available links.

Switch options

switch-options {
    vtep-source-interface lo0.0;
}

Configure a source interface for a Virtual Extensible LAN (VXLAN) tunnel. You must provide the name of a logical interface configured on the loopback interface.

If you don’t configure this your vtep interface won’t work:

root@vmx1> show interfaces vtep
Physical interface: vtep, Enabled, Physical link is Up
  Interface index: 133, SNMP ifIndex: 510
  Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited
  Device flags   : Present Running
  Link type      : Full-Duplex
  Link flags     : None
  Last flapped   : Never
    Input packets : 0
    Output packets: 0

Your vtep interface will actually become active when you connect it to a source interface in the switch options:

root@vmx1# activate switch-options

[edit]
root@vmx1# show | compare
[edit]
!   active: switch-options { ... }

[edit]
root@vmx1# commit and-quit
shcommit complete
Exiting configuration mode

root@vmx1> show interfaces vtep
Physical interface: vtep, Enabled, Physical link is Up
  Interface index: 133, SNMP ifIndex: 510
  Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited
  Device flags   : Present Running
  Link type      : Full-Duplex
  Link flags     : None
  Last flapped   : Never
    Input packets : 0
    Output packets: 0

  Logical interface vtep.32768 (Index 328) (SNMP ifIndex 541)
    Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2
    Ethernet segment value: 00:00:00:00:00:00:00:00:00:00, Mode: single-homed, Multi-homed status: Blocking
    VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 172.16.100.12, L2 Routing Instance: default-switch, L3 Routing Instance: default
    Input packets : 0
    Output packets: 0

Leaf config

Here is the config of a leaf switch. In our example router vmx3. Here is also a test host connected.

I will try to explain the differences.

system {
    host-name vmx3;
    root-authentication {
        encrypted-password ""; ## SECRET-DATA
    }
    services {
        ssh;
    }
}
chassis {
    fpc 0 {
        pic 0 {
            tunnel-services {
                bandwidth 1g;
            }
        }
        lite-mode;
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            family inet {
                address 172.16.5.1/24;
            }
        }
    }
    ge-0/0/1 {
        unit 0 {
            family inet {
                address 172.16.6.1/24;
            }
        }
    }
    ge-0/0/2 {
        description "to testhost 1";
        encapsulation ethernet-bridge;
        unit 0 {
            family bridge;
        }
    }
    fxp0 {
        unit 0 {
            family inet {
                address 192.168.1.13/24;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 172.16.100.16/32;
            }
        }
    }
}
routing-options {
    autonomous-system 64516;
    forwarding-table {
        export loadbalance;
    }
}
protocols {
    bgp {
        group underlay {
            type external;
            export direct;
            multipath multiple-as;
            neighbor 172.16.5.2 {
                peer-as 64512;
            }
            neighbor 172.16.6.2 {
                peer-as 64513;
            }
        }
    }
    pim {
        rp {
            static {
                address 172.16.100.12 {
                    group-ranges {
                        224.0.0.0/4;
                    }
                }
            }
        }
        interface fxp0.0 {
            disable;
        }
        interface lo0.0 {
            mode sparse;
        }
        interface ge-0/0/0.0 {
            mode sparse;
        }
        interface ge-0/0/1.0 {
            mode sparse;
        }
    }
    lldp {
        interface all;
    }
}
policy-options {
    policy-statement direct {
        term 10 {
            from protocol direct;
            then accept;
        }
    }
    policy-statement loadbalance {
        then {
            load-balance per-packet;
        }
    }
}
bridge-domains {
    v100 {
        domain-type bridge;
        vlan-id 100;
        interface ge-0/0/2.0;
        vxlan {
            vni 1000;
            multicast-group 239.0.0.1;
        }
    }
}
switch-options {
    vtep-source-interface lo0.0;
}

Protocols

PIM
    pim {
        rp {
            static {
                address 172.16.100.12 {
                    group-ranges {
                        224.0.0.0/4;
                    }
                }
            }
        }
        interface fxp0.0 {
            disable;
        }
        interface lo0.0 {
            mode sparse;
        }
        interface ge-0/0/0.0 {
            mode sparse;
        }
        interface ge-0/0/1.0 {
            mode sparse;
        }
    }

So in the PIM we need to join the rendezvous point for the used multicast groups.

Bridge domains

bridge-domains {
    v100 {
        domain-type bridge;
        vlan-id 100;
        interface ge-0/0/2.0;
        vxlan {
            vni 1000;
            multicast-group 239.0.0.1;
        }
    }
}

So in the bridge domain configuration we actually connect interfaces with vlans and vxlans. Vlan id’s are only significant to their bridge domains. You can bind a vlan 100 to vni 1000 on vmx3 and connect a completely different vlan. The vni needs to be binded on a multicast group. This needs to be the same multicast group on all routers for the specific vni.

Untagged interface

Had to look for this, but apparently this i show you make an untagged interface on MX. Actually pretty obvious 🙂

    ge-0/0/2 {
        description "to testhost1";
        encapsulation ethernet-bridge;
        unit 0 {
            family bridge;
        }
    }

Conclusion

So the idea was to share you my labconfig on vmx. You can use this to get a better understanding of vxlan or as a study tool.

I’ll try to share a config for vxlan with evpn signaling later.

Leave a Reply