Tweet
Tweet

Slide 1

Slide 1 text

Running an EVPN endpoint In a Kubernetes cluster (on my laptop!) 1 Federico Paolinelli - Red Hat

Slide 2

Slide 2 text

● Openshift Telco 5G Network team ● Contributed to: ○ KubeVirt ○ SR-IOV Network Operator ○ OVN-Kubernetes ○ CNI plugins ○ Kubernetes ○ MetalLB ○ FRR-K8s @fedepaol hachyderm.io/@fedepaol [email protected] About me 2

Slide 3

Slide 3 text

This talk is about my journey into EVPN 3 https://flic.kr/p/2kp1AH6

Slide 4

Slide 4 text

flic.kr/p/ooqPHW EVPN Primer 4

Slide 5

Slide 5 text

Let’s start with VXLan

Slide 6

Slide 6 text

6 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2

Slide 7

Slide 7 text

7 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2

Slide 8

Slide 8 text

8 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2

Slide 9

Slide 9 text

9 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2

Slide 10

Slide 10 text

Who is telling us what VTEP to target?

Slide 11

Slide 11 text

Who is telling us what VTEP to target? EVPN!

Slide 12

Slide 12 text

12 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 - Mac xxxx VLan2 VTep1 VTep2 BGP: VNI100 - Mac XXX ->VTEP2

Slide 13

Slide 13 text

13 Spine Leaf 1 Host1 Host2 Leaf 2 Host2 VLan1 VLan2 VLan1 VLan2 - IP XXXX VTep1 VTep2 BGP: VNI100 - 10.0.1.0/24 ->VTEP2

Slide 14

Slide 14 text

14 Spine Leaf 1 Host1 Host2 Leaf 2 Host2 VLan1 VLan2 VLan1 VLan2 - IP XXXX VTep1 VTep2 BGP: VNI200 - IPXXX

Slide 15

Slide 15 text

Our destination 15

Slide 16

Slide 16 text

16 Spine Leaf 1 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 VNI100 Node

Slide 17

Slide 17 text

17 Spine Leaf 1 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 VNI100 Node BGP Routes EVPN Routes

Slide 18

Slide 18 text

18 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green

Slide 19

Slide 19 text

19 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green The interface is moved inside the namespace!

Slide 20

Slide 20 text

20 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green BGP Routes EVPN Routes EVPN Routes

Slide 21

Slide 21 text

21 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green TCP Traffic VXLan VXLan

Slide 22

Slide 22 text

22 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green ONE SINGLE BGP SESSION, NO NEED TO RECONFIGURE THE FABRIC

Slide 23

Slide 23 text

On my Laptop! 23 https://flic.kr/p/2ixV7ag

Slide 24

Slide 24 text

My Diary 24 https://flic.kr/p/ffFhz5 https://github.com/fedepaol/evpnlab github.fedepaol.io

Slide 25

Slide 25 text

The shopping List 25 https://flic.kr/p/6pKba9

Slide 26

Slide 26 text

Routers! 26 https://flic.kr/p/tp1xAn

Slide 27

Slide 27 text

FRRouting 27 FRRouting (FRR) is a free and open source Internet routing protocol suite for Linux and Unix platforms. It implements BGP, OSPF, RIP, IS-IS, PIM, LDP, BFD, Babel, PBR, OpenFabric and VRRP, with alpha support for EIGRP and NHRP [...] FRR has its roots in the Quagga project.

Slide 28

Slide 28 text

28 BGP-EVPN is the control plane for the transport of Ethernet frames, regardless of whether those frames are bridged or routed [...] FRR learns about the system’s Linux network interface configuration from the kernel via Netlink, however it does not manage network interfaces directly. https://docs.frrouting.org/en/latest/evpn.html#evpn

Slide 29

Slide 29 text

29 lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 Vxlan Vni 200 br200 ● A Linux VRF for each VNI ● An SVI (Linux Bridge) ● A VXLan interface enslaved to the bridge For each (L3)VNI FRR Needs:

Slide 30

Slide 30 text

30 lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 Vxlan Vni 200 br200 router bgp 4200000000 neighbor 192.168.122.12 remote-as internal ! address-family ipv4 unicast network 100.64.0.1/32 exit-address-family ! address-family l2vpn evpn neighbor 192.168.122.12 activate advertise-all-vni advertise-svi-ip exit-address-family exit

Slide 31

Slide 31 text

31 lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 Vxlan Vni 200 br200 router bgp 4200000000 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast advertise ipv6 unicast exit-address-family exit ! router bgp 4200000000 neighbor 192.168.122.12 remote-as internal ! address-family ipv4 unicast network 100.64.0.1/32 exit-address-family ! address-family l2vpn evpn neighbor 192.168.122.12 activate advertise-all-vni advertise-svi-ip exit-address-family exit

Slide 32

Slide 32 text

A Network Lab! 32 https://flic.kr/p/ydnEvw

Slide 33

Slide 33 text

ContainerLab 33 Containerlab provides a CLI for orchestrating and managing container-based networking labs. It starts the containers, builds a virtual wiring between them to create lab topologies of users choice and manages labs lifecycle. https://containerlab.dev/

Slide 34

Slide 34 text

34 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 https://github.com/fedepaol/evpnlab/tree/main/01_clab_l3 fedepaol.github.io/blog/2024/05/09/l3evpn-using-frr-and-linux-vxlans/

Slide 35

Slide 35 text

35 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"]

Slide 36

Slide 36 text

36 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"] Node

Slide 37

Slide 37 text

37 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"] Links

Slide 38

Slide 38 text

38 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"] name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 binds: - leaf1/:/etc/frr/ - leaf1/setup.sh:/setup.sh

Slide 39

Slide 39 text

40 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh

Slide 40

Slide 40 text

41 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2

Slide 41

Slide 41 text

42 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2 Create the VRF

Slide 42

Slide 42 text

43 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2 Enslave the interface to the host In the linux VRF

Slide 43

Slide 43 text

44 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2 FRR VXLan setup: Bridge, vxlan

Slide 44

Slide 44 text

45 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1

Slide 45

Slide 45 text

46 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 router bgp 4200000000 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast advertise ipv6 unicast exit-address-family exit !

Slide 46

Slide 46 text

47 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 Type 5 Routes Type 5 Routes

Slide 47

Slide 47 text

48 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 Type 5 Routes Type 5 Routes leaf2# show bgp l2vpn evpn BGP table version is 1, local router ID is 100.65.0.2 Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.10.2:2 *> [5]:[0]:[24]:[192.168.10.0] 100.64.0.1 0 64612 64512 ? RT:64512:100 ET:8 Rmac:aa:bb:cc:00:00:65 Route Distinguisher: 192.168.11.2:2 *> [5]:[0]:[24]:[192.168.11.0] 100.65.0.2 0 32768 ? ET:8 RT:64512:100 Rmac:aa:bb:cc:00:00:64

Slide 48

Slide 48 text

49 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 VXLAN - VNI100 ICMP ICMP

Slide 49

Slide 49 text

50 github.com/fedepaol/evpnlab/tree/main/01_clab_l3 fedepaol.github.io/blog/2024/05/09/l3evpn-using-frr-and-linux-vxlans/ REFERENCES youtu.be/gBgdv61EaX8

Slide 50

Slide 50 text

A Kubernetes Cluster 53 github.com/fedepaol/evpnlab/tree/main/04_from_kind

Slide 51

Slide 51 text

54 kind is a tool for running local Kubernetes clusters using Docker container “nodes”.

Slide 52

Slide 52 text

55 name: kind topology: nodes: leaf2: kind: linux image: quay.io/frrouting/frr:10.0.2 k0: kind: k8s-kind k0-control-plane: kind: ext-container binds: - kind/setup.sh:/setup.sh links: - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["k0-control-plane:eth1", "leaf2:eth2"] Spine Leaf 2 Kind Node

Slide 53

Slide 53 text

56 name: kind topology: nodes: leaf2: kind: linux image: quay.io/frrouting/frr:10.0.2 k0: kind: k8s-kind k0-control-plane: kind: ext-container binds: - kind/setup.sh:/setup.sh links: - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["k0-control-plane:eth1", "leaf2:eth2"] Spine Leaf 2 Kind Node

Slide 54

Slide 54 text

57 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green

Slide 55

Slide 55 text

58 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ...

Slide 56

Slide 56 text

59 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh

Slide 57

Slide 57 text

60 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh FRR As a container

Slide 58

Slide 58 text

61 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh Veth Pair, one leg in the container

Slide 59

Slide 59 text

62 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh Interface for underlay

Slide 60

Slide 60 text

63 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh Interface setup required by FRR (bridge, vxlan, etc)

Slide 61

Slide 61 text

64 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green

Slide 62

Slide 62 text

65 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s

Slide 63

Slide 63 text

66 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s b6f5fc828810# show bgp vrf red ipv4 BGP table version is 2, local router ID is 192.169.10.1, vrf id 2 Network Next Hop *> 192.168.8.0/32 192.169.10.0 *> 192.168.10.0/24 100.64.0.1<

Slide 64

Slide 64 text

67 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s b6f5fc828810# show bgp l2vpn evpn BGP table version is 1, local router ID is 100.65.0.2 Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.10.2:2 *> [5]:[0]:[24]:[192.168.10.0] 100.64.0.1 0 64513 64612 64512 ? RT:64512:100 ET:8 Rmac:aa:bb:cc:00:00:65 Route Distinguisher: 192.169.10.1:2 *> [5]:[0]:[32]:[192.168.8.0] 100.65.0.2 0 0 64512 64515 i ET:8 RT:64512:100 Rmac:aa:bb:cc:00:00:66 Displayed 2 out of 2 total prefixes

Slide 65

Slide 65 text

68 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s leaf1# show bgp l2vpn evpn BGP table version is 2, local router ID is 100.64.0.1 Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.10.2:2 *> [5]:[0]:[24]:[192.168.10.0] 100.64.0.1 0 32768 ? ET:8 RT:64512:100 Rmac:aa:bb:cc:00:00:65 Route Distinguisher: 192.169.10.1:2 *> [5]:[0]:[32]:[192.168.8.0] 100.65.0.2 0 RT:64512:100 ET:8 Rmac:aa:bb:cc:00:00:66 Displayed 2 out of 2 total prefixes

Slide 66

Slide 66 text

69 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green TCP VXLAN VXLAN VXLAN TCP

Slide 67

Slide 67 text

70 github.com/fedepaol/evpnlab/tree/main/04_from_kind fedepaol.github.io/blog/2024/06/14/extending-the-kubernetes-host-network-with-evpn-tunnels-and-frr/ REFERENCES youtu.be/c6dqEwLVWA0

Slide 68

Slide 68 text

FRR as a Pod 73 github.com/fedepaol/evpnlab/tree/main/06_from_kind_with_pods

Slide 69

Slide 69 text

74 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green

Slide 70

Slide 70 text

Leaf 2 75 eth1 vxlan vrf FRR Pod Controller Node

Slide 71

Slide 71 text

Leaf 2 76 eth1 vxlan vrf FRR Pod Moves from host into ns Controller Node

Slide 72

Slide 72 text

Leaf 2 77 eth1 vxlan veth pair vrf FRR Pod Moves from host into ns Creates the veth and moves it into the network ns Controller Node

Slide 73

Slide 73 text

FRR 78 ● Same as previous examples: ○ FRR configuration ○ Script to setup the interfaces ○ Assumes the controller moved the interfaces The controller ● Uses crictl to find the target namespace ● Creates and moves the Veth leg dynamically ● Similar to what Multus Dynamic Controller does

Slide 74

Slide 74 text

FRR 79 ● Same as previous examples: ○ FRR configuration ○ Script to setup the interfaces ○ Assumes the controller moved the interfaces The controller ● Uses crictl to find the target namespace ● Creates and moves the Veth leg dynamically ● Similar to what Multus Dynamic Controller does CONTAINERD_SOCK="var/run/containerd/containerd.sock" POD_ID=$(crictl -r ${CONTAINERD_SOCK} pods --name=frr --namespace=frrtest -q --no-trunc) NSPATH=$(crictl -r ${CONTAINERD_SOCK} inspectp ${POD_ID} | \ jq -r '.info.runtimeSpec.linux.namespaces[] |select(.type=="network") | .path') NETNS=$(basename $NSPATH) ip link add frr0 type veth peer name frr1 ip link set frr0 up ip link set dev eth1 netns $NETNS ip link set frr1 netns $NETNS ip addr add dev frr0 192.169.10.0/24 Target NS

Slide 75

Slide 75 text

80 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red

Slide 76

Slide 76 text

88 REFERENCES github.com/fedepaol/evpnlab/tree/main/06_from_kind_with_pods fedepaol.github.io/blog/2024/11/10/running-evpn-termination-inside-a-kubernetes-pod/

Slide 77

Slide 77 text

A cloud native PE Router (WIP) 89 github.com/openperouter/openperouter https://flic.kr/p/gLaqBE

Slide 78

Slide 78 text

90 eth1 FRR Pod Reloader Container Controller

Slide 79

Slide 79 text

91 eth1 FRR Pod Reloader Container Controller CRD Based API

Slide 80

Slide 80 text

92 eth1 FRR Pod 1 - Moves from host into ns Reloader Container Controller CRD Based API

Slide 81

Slide 81 text

93 eth1 vxlan vrf FRR Pod 1 - Moves from host into ns 2 - Creates: - vrfs - bridges - vxlan interfaces Reloader Container Controller CRD Based API

Slide 82

Slide 82 text

94 eth1 vxlan vrf FRR Pod 1 - Moves from host into ns 2 - Creates: - vrfs - bridges - vxlan interfaces Reloader Container Controller CRD Based API veth pair 3 - Creates the veth and moves it into the network ns

Slide 83

Slide 83 text

95 eth1 vxlan vrf FRR Pod 1 - Moves from host into ns 2 - Creates: - vrfs - bridges - vxlan interfaces Reloader Container Controller CRD Based API veth pair 3 - Creates the veth and moves it into the network ns 4 - provides an FRR configuration and signals the reloader frr.conf

Slide 84

Slide 84 text

96 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red CRD Based API

Slide 85

Slide 85 text

The API: underlay 97 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2

Slide 86

Slide 86 text

The API: underlay 98 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2 CIDR to be used for the VTEP IP on each node

Slide 87

Slide 87 text

The API: underlay 99 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2 Interface to be moved under the FRR namespace

Slide 88

Slide 88 text

The API: underlay 100 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2 Session with the TOR

Slide 89

Slide 89 text

The API: VNI 101 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515

Slide 90

Slide 90 text

The API: VNI 102 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The ASN for the local session Towards the node

Slide 91

Slide 91 text

The API: VNI 103 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The VRF

Slide 92

Slide 92 text

The API: VNI 104 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The VXLan ID

Slide 93

Slide 93 text

The API: VNI 105 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The CIDR to be used to assign The IP to the VETH

Slide 94

Slide 94 text

106 ● Same logic as the previous examples, but in a Kubernetes reconcile loop ● Can interact with any BGP enabled component running on the host ● The IP of the PE Router side of the Veth pair is the same for all the nodes

Slide 95

Slide 95 text

107 ● Same logic as the previous examples, but in a Kubernetes reconcile loop ● Can interact with any BGP enabled component running on the host ● The IP of the PE Router side of the Veth pair is the same for all the nodes STILL VERY WIP!

Slide 96

Slide 96 text

108 SEEING IT IN ACTION

Slide 97

Slide 97 text

109 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red

Slide 98

Slide 98 text

110 WHY CALICO?

Slide 99

Slide 99 text

111 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red Node POD CIDR Via BGP Control Plane

Slide 100

Slide 100 text

112 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red Node POD CIDR as Type 5 routes across the fabric Control Plane

Slide 101

Slide 101 text

113 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red Node POD CIDR BGP routes Sent to the node Control Plane

Slide 102

Slide 102 text

114 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red ICMP VXLan VXLan Control Plane

Slide 103

Slide 103 text

115 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red ICMP VXLan VXLan Data Path

Slide 104

Slide 104 text

116 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red ICMP ICMP VXLan VXLan

Slide 105

Slide 105 text

Demo! 117 https://flic.kr/p/p51z9r

Slide 106

Slide 106 text

118 https://flic.kr/p/p51z9r Demo available at https://youtu.be/jetqMYnWjm8

Slide 107

Slide 107 text

Where to go from here

Slide 108

Slide 108 text

120 MAKING IT WORK!

Slide 109

Slide 109 text

121 ENABLE L2 EVPN

Slide 110

Slide 110 text

122 ● The architecture shown has one big limitation: we need to sacrifice one interface ● Can overcome by using a Vlan interface ● What if we want EVPN connectivity at day 0?

Slide 111

Slide 111 text

Running Podman Pods as systemd units 123 ● The same smart-controller / dumb-FRR architecture can be started as Podman pods running as systemd units ● It allows to have an EVPN-based primary interface ● Still to be consolidated github.com/fedepaol/evpnlab/tree/main/08_from_kind_with_systemdunits fedepaol.github.io/blog/2025/01/06/enabling-evpn-termination-with-podman-pods-as-systemd-units/

Slide 112

Slide 112 text

A faster datapath? 124 https://flic.kr/p/p51z9r

Slide 113

Slide 113 text

Wrapping Up 125

Slide 114

Slide 114 text

Resources 126 ● FRR Routing docs at frrouting.org ● ContainerLab containerlab.dev ● My EVPN Lab repo github.com/fedepaol/evpnlab/ ● My personal Blog fedepaol.github.io/posts/ ● OpenPERouter repo github.com/openperouter/openperouter ● Das Schiff Network Operator github.com/telekom/das-schiff-network-operator

Slide 115

Slide 115 text

Thanks! Any questions? fedepaol.bsky.social hachyderm.io/@fedepaol Slides at: speakerdeck.com/fedepaol [email protected]