Running an EVPN Endpoint in a Kubernetes Cluster—On My Laptop!

In this session, I will share my journey in prototyping and implementing an EVPN termination solution using Kubernetes, all on a single laptop. Leveraging tools like Kind and Containerlab, we’ll start with a basic EVPN example and progressively build toward a more complex spine-leaf topology integrated with a Kubernetes cluster. Finally, I will present a live demo, showcasing the solution in action and proving how easy it is to test and implement complex network topologies with Containerlab and Kind.

Transcript

1. Running an EVPN endpoint In a Kubernetes cluster (on my laptop!) 1 Federico Paolinelli - Red Hat
2. • Openshift Telco 5G Network team • Contributed to: ◦ KubeVirt ◦ SR-IOV Network Operator ◦ OVN-Kubernetes ◦ CNI plugins ◦ Kubernetes ◦ MetalLB ◦ FRR-K8s @fedepaol hachyderm.io/@fedepaol fedepaol@gmail.com About me 2
3. This talk is about my journey into EVPN 3 https://flic.kr/p/2kp1AH6
4. flic.kr/p/ooqPHW EVPN Primer 4
5. Let’s start with VXLan
6. 6 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2
7. 7 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2
8. 8 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2
9. 9 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 - To Vtep 2 VTep1 VTep2 VNI100 - To Vtep 2
10. Who is telling us what VTEP to target?
11. Who is telling us what VTEP to target? EVPN!
12. 12 Spine Leaf 1 Host1 Host2 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 - Mac xxxx VLan2 VTep1 VTep2 BGP: VNI100 - Mac XXX ->VTEP2
13. 13 Spine Leaf 1 Host1 Host2 Leaf 2 Host2 VLan1 VLan2 VLan1 VLan2 - IP XXXX VTep1 VTep2 BGP: VNI100 - 10.0.1.0/24 ->VTEP2
14. 14 Spine Leaf 1 Host1 Host2 Leaf 2 Host2 VLan1 VLan2 VLan1 VLan2 - IP XXXX VTep1 VTep2 BGP: VNI200 - IPXXX
15. Our destination 15
16. 16 Spine Leaf 1 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 VNI100 Node
17. 17 Spine Leaf 1 Leaf 2 Host1 Host2 VLan1 VLan2 VLan1 VLan2 VNI100 VNI100 Node BGP Routes EVPN Routes
18. 18 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green
19. 19 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green The interface is moved inside the namespace!
20. 20 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green BGP Routes EVPN Routes EVPN Routes
21. 21 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green TCP Traffic VXLan VXLan
22. 22 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green ONE SINGLE BGP SESSION, NO NEED TO RECONFIGURE THE FABRIC
23. On my Laptop! 23 https://flic.kr/p/2ixV7ag
24. My Diary 24 https://flic.kr/p/ffFhz5 https://github.com/fedepaol/evpnlab github.fedepaol.io
25. The shopping List 25 https://flic.kr/p/6pKba9
26. Routers! 26 https://flic.kr/p/tp1xAn
27. FRRouting 27 FRRouting (FRR) is a free and open source Internet routing protocol suite for Linux and Unix platforms. It implements BGP, OSPF, RIP, IS-IS, PIM, LDP, BFD, Babel, PBR, OpenFabric and VRRP, with alpha support for EIGRP and NHRP [...] FRR has its roots in the Quagga project.
28. 28 BGP-EVPN is the control plane for the transport of Ethernet frames, regardless of whether those frames are bridged or routed [...] FRR learns about the system’s Linux network interface configuration from the kernel via Netlink, however it does not manage network interfaces directly. https://docs.frrouting.org/en/latest/evpn.html#evpn
29. 29 lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 Vxlan Vni 200 br200 • A Linux VRF for each VNI • An SVI (Linux Bridge) • A VXLan interface enslaved to the bridge For each (L3)VNI FRR Needs:
30. 30 lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 Vxlan Vni 200 br200 router bgp 4200000000 neighbor 192.168.122.12 remote-as internal ! address-family ipv4 unicast network 100.64.0.1/32 exit-address-family ! address-family l2vpn evpn neighbor 192.168.122.12 activate advertise-all-vni advertise-svi-ip exit-address-family exit
31. 31 lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 Vxlan Vni 200 br200 router bgp 4200000000 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast advertise ipv6 unicast exit-address-family exit ! router bgp 4200000000 neighbor 192.168.122.12 remote-as internal ! address-family ipv4 unicast network 100.64.0.1/32 exit-address-family ! address-family l2vpn evpn neighbor 192.168.122.12 activate advertise-all-vni advertise-svi-ip exit-address-family exit
32. A Network Lab! 32 https://flic.kr/p/ydnEvw
33. ContainerLab 33 Containerlab provides a CLI for orchestrating and managing container-based networking labs. It starts the containers, builds a virtual wiring between them to create lab topologies of users choice and manages labs lifecycle. https://containerlab.dev/
34. 34 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 https://github.com/fedepaol/evpnlab/tree/main/01_clab_l3 fedepaol.github.io/blog/2024/05/09/l3evpn-using-frr-and-linux-vxlans/
35. 35 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"]
36. 36 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"] Node
37. 37 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"] Links
38. 38 name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 leaf2: kind: linux image: quay.io/frrouting/frr:10.2.1 spine: kind: linux image: quay.io/frrouting/frr:10.2.1 HOST1: kind: linux image: praqma/network-multitool:latest HOST2: kind: linux image: praqma/network-multitool:latest links: - endpoints: ["leaf1:eth1", "spine:eth1"] - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["HOST1:eth1", "leaf1:eth2"] - endpoints: ["HOST2:eth1", "leaf2:eth2"] name: evpnl3 topology: nodes: leaf1: kind: linux image: quay.io/frrouting/frr:10.2.1 binds: - leaf1/:/etc/frr/ - leaf1/setup.sh:/setup.sh
39. 40 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh
40. 41 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2
41. 42 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2 Create the VRF
42. 43 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2 Enslave the interface to the host In the linux VRF
43. 44 sudo clab deploy --reconfigure --topo direct.clab.yml docker exec clab-evpnl3-leaf1 /setup.sh docker exec clab-evpnl3-leaf2 /setup.sh docker exec clab-evpnl3-spine /setup.sh #!/bin/bash # # VTEP IP ip addr add 100.64.0.1/32 dev lo # Leaf - spine leg ip addr add 192.168.1.1/24 dev eth1 # L3 VRF ip link add red type vrf table 1100 # Leaf - host leg ip link set eth2 master red ip addr add 192.168.10.2/24 dev eth2 ip link set red up ip link add br100 type bridge ip link set br100 master red addrgenmode none ip link set br100 addr aa:bb:cc:00:00:65 ip link add vni100 type vxlan local 100.64.0.1 dstport 4789 id 100 nolearning ip link set vni100 master br100 addrgenmode none ip link set vni100 type bridge_slave neigh_suppress on learning off ip link set vni100 up ip link set br100 up lo 100.65.0.2/32 Vxlan Vni 100 eth1 br100 eth2 FRR VXLan setup: Bridge, vxlan
44. 45 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1
45. 46 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 router bgp 4200000000 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast advertise ipv6 unicast exit-address-family exit !
46. 47 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 Type 5 Routes Type 5 Routes
47. 48 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 Type 5 Routes Type 5 Routes leaf2# show bgp l2vpn evpn BGP table version is 1, local router ID is 100.65.0.2 Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.10.2:2 *> [5]:[0]:[24]:[192.168.10.0] 100.64.0.1 0 64612 64512 ? RT:64512:100 ET:8 Rmac:aa:bb:cc:00:00:65 Route Distinguisher: 192.168.11.2:2 *> [5]:[0]:[24]:[192.168.11.0] 100.65.0.2 0 32768 ? ET:8 RT:64512:100 Rmac:aa:bb:cc:00:00:64
48. 49 Spine Leaf 1 Host1 Leaf 2 Host1 VLan1 VLan1 VXLAN - VNI100 ICMP ICMP
49. 50 github.com/fedepaol/evpnlab/tree/main/01_clab_l3 fedepaol.github.io/blog/2024/05/09/l3evpn-using-frr-and-linux-vxlans/ REFERENCES youtu.be/gBgdv61EaX8
50. A Kubernetes Cluster 53 github.com/fedepaol/evpnlab/tree/main/04_from_kind
51. 54 kind is a tool for running local Kubernetes clusters using Docker container “nodes”.
52. 55 name: kind topology: nodes: leaf2: kind: linux image: quay.io/frrouting/frr:10.0.2 k0: kind: k8s-kind k0-control-plane: kind: ext-container binds: - kind/setup.sh:/setup.sh links: - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["k0-control-plane:eth1", "leaf2:eth2"] Spine Leaf 2 Kind Node
53. 56 name: kind topology: nodes: leaf2: kind: linux image: quay.io/frrouting/frr:10.0.2 k0: kind: k8s-kind k0-control-plane: kind: ext-container binds: - kind/setup.sh:/setup.sh links: - endpoints: ["leaf2:eth1", "spine:eth2"] - endpoints: ["k0-control-plane:eth1", "leaf2:eth2"] Spine Leaf 2 Kind Node
54. 57 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green
55. 58 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ...
56. 59 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh
57. 60 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh FRR As a container
58. 61 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh Veth Pair, one leg in the container
59. 62 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh Interface for underlay
60. 63 sudo clab deploy --reconfigure --topo kind.clab.yml ... docker cp kind/setup.sh k0-control-plane:/setup.sh docker cp kind/frr k0-control-plane:/frr ... #!/bin/bash # KIND NODE SETUP ip addr add dev eth1 192.168.11.3/24 systemctl start docker docker run --name frr --privileged -v /frr:/etc/frr -d quay.io/frrouting/frr:10.2.0 NAMESPACE=$(docker inspect -f '{{.NetworkSettings.SandboxKey}}' frr) ip link add frrhost type veth peer name frrns ip link set frrhost up ip link set dev eth1 netns $NAMESPACE ip link set frrns netns $NAMESPACE ip addr add dev frrhost 192.169.10.0/24 docker exec frr /etc/frr/setup.sh Interface setup required by FRR (bridge, vxlan, etc)
61. 64 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green
62. 65 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s
63. 66 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s b6f5fc828810# show bgp vrf red ipv4 BGP table version is 2, local router ID is 192.169.10.1, vrf id 2 Network Next Hop *> 192.168.8.0/32 192.169.10.0 *> 192.168.10.0/24 100.64.0.1<
64. 67 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s b6f5fc828810# show bgp l2vpn evpn BGP table version is 1, local router ID is 100.65.0.2 Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.10.2:2 *> [5]:[0]:[24]:[192.168.10.0] 100.64.0.1 0 64513 64612 64512 ? RT:64512:100 ET:8 Rmac:aa:bb:cc:00:00:65 Route Distinguisher: 192.169.10.1:2 *> [5]:[0]:[32]:[192.168.8.0] 100.65.0.2 0 0 64512 64515 i ET:8 RT:64512:100 Rmac:aa:bb:cc:00:00:66 Displayed 2 out of 2 total prefixes
65. 68 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE httpd LoadBalancer 10.96.217.3 192.168.8.0 80:31047/TCP 3m16s leaf1# show bgp l2vpn evpn BGP table version is 2, local router ID is 100.64.0.1 Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 192.168.10.2:2 *> [5]:[0]:[24]:[192.168.10.0] 100.64.0.1 0 32768 ? ET:8 RT:64512:100 Rmac:aa:bb:cc:00:00:65 Route Distinguisher: 192.169.10.1:2 *> [5]:[0]:[32]:[192.168.8.0] 100.65.0.2 0 RT:64512:100 ET:8 Rmac:aa:bb:cc:00:00:66 Displayed 2 out of 2 total prefixes
66. 69 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red Veth green TCP VXLAN VXLAN VXLAN TCP
67. 70 github.com/fedepaol/evpnlab/tree/main/04_from_kind fedepaol.github.io/blog/2024/06/14/extending-the-kubernetes-host-network-with-evpn-tunnels-and-frr/ REFERENCES youtu.be/c6dqEwLVWA0
68. FRR as a Pod 73 github.com/fedepaol/evpnlab/tree/main/06_from_kind_with_pods
69. 74 Spine Leaf 1 Host1 Host2 VLan1 VLan2 VNI100 VNI100 Node Leaf 2 Veth red Veth green
70. Leaf 2 75 eth1 vxlan vrf FRR Pod Controller Node
71. Leaf 2 76 eth1 vxlan vrf FRR Pod Moves from host into ns Controller Node
72. Leaf 2 77 eth1 vxlan veth pair vrf FRR Pod Moves from host into ns Creates the veth and moves it into the network ns Controller Node
73. FRR 78 • Same as previous examples: ◦ FRR configuration ◦ Script to setup the interfaces ◦ Assumes the controller moved the interfaces The controller • Uses crictl to find the target namespace • Creates and moves the Veth leg dynamically • Similar to what Multus Dynamic Controller does
74. FRR 79 • Same as previous examples: ◦ FRR configuration ◦ Script to setup the interfaces ◦ Assumes the controller moved the interfaces The controller • Uses crictl to find the target namespace • Creates and moves the Veth leg dynamically • Similar to what Multus Dynamic Controller does CONTAINERD_SOCK="var/run/containerd/containerd.sock" POD_ID=$(crictl -r ${CONTAINERD_SOCK} pods --name=frr --namespace=frrtest -q --no-trunc) NSPATH=$(crictl -r ${CONTAINERD_SOCK} inspectp ${POD_ID} | \ jq -r '.info.runtimeSpec.linux.namespaces[] |select(.type=="network") | .path') NETNS=$(basename $NSPATH) ip link add frr0 type veth peer name frr1 ip link set frr0 up ip link set dev eth1 netns $NETNS ip link set frr1 netns $NETNS ip addr add dev frr0 192.169.10.0/24 Target NS
75. 80 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red
76. 88 REFERENCES github.com/fedepaol/evpnlab/tree/main/06_from_kind_with_pods fedepaol.github.io/blog/2024/11/10/running-evpn-termination-inside-a-kubernetes-pod/
77. A cloud native PE Router (WIP) 89 github.com/openperouter/openperouter https://flic.kr/p/gLaqBE
78. 90 eth1 FRR Pod Reloader Container Controller
79. 91 eth1 FRR Pod Reloader Container Controller CRD Based API
80. 92 eth1 FRR Pod 1 - Moves from host into ns Reloader Container Controller CRD Based API
81. 93 eth1 vxlan vrf FRR Pod 1 - Moves from host into ns 2 - Creates: - vrfs - bridges - vxlan interfaces Reloader Container Controller CRD Based API
82. 94 eth1 vxlan vrf FRR Pod 1 - Moves from host into ns 2 - Creates: - vrfs - bridges - vxlan interfaces Reloader Container Controller CRD Based API veth pair 3 - Creates the veth and moves it into the network ns
83. 95 eth1 vxlan vrf FRR Pod 1 - Moves from host into ns 2 - Creates: - vrfs - bridges - vxlan interfaces Reloader Container Controller CRD Based API veth pair 3 - Creates the veth and moves it into the network ns 4 - provides an FRR configuration and signals the reloader frr.conf
84. 96 Spine Leaf 1 Host1 Host2 VLan1 VLan2 Node Leaf 2 Veth red CRD Based API
85. The API: underlay 97 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2
86. The API: underlay 98 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2 CIDR to be used for the VTEP IP on each node
87. The API: underlay 99 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2 Interface to be moved under the FRR namespace
88. The API: underlay 100 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: Underlay metadata: name: underlay namespace: openperouter-system spec: asn: 64514 vtepcidr: 100.65.0.0/24 nic: cleth1 neighbors: - asn: 64512 address: 192.168.11.2 Session with the TOR
89. The API: VNI 101 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515
90. The API: VNI 102 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The ASN for the local session Towards the node
91. The API: VNI 103 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The VRF
92. The API: VNI 104 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The VXLan ID
93. The API: VNI 105 apiVersion: per.io.openperouter.github.io/v1alpha1 kind: VNI metadata: name: vni-sample namespace: openperouter-system spec: asn: 64514 vrf: red vni: 100 localcidr: 192.169.10.0/24 localasn: 64515 The CIDR to be used to assign The IP to the VETH
94. 106 • Same logic as the previous examples, but in a Kubernetes reconcile loop • Can interact with any BGP enabled component running on the host • The IP of the PE Router side of the Veth pair is the same for all the nodes
95. 107 • Same logic as the previous examples, but in a Kubernetes reconcile loop • Can interact with any BGP enabled component running on the host • The IP of the PE Router side of the Veth pair is the same for all the nodes STILL VERY WIP!
96. 108 SEEING IT IN ACTION
97. 109 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red
98. 110 WHY CALICO?
99. 111 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red Node POD CIDR Via BGP Control Plane
100. 112 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red Node POD CIDR as Type 5 routes across the fabric Control Plane
101. 113 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red Node POD CIDR BGP routes Sent to the node Control Plane
102. 114 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red ICMP VXLan VXLan Control Plane
103. 115 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red ICMP VXLan VXLan Data Path
104. 116 Spine Leaf 1 Host1 VLan1 Node Leaf 2 Veth red Node Veth red ICMP ICMP VXLan VXLan
105. Demo! 117 https://flic.kr/p/p51z9r
106. 118 https://flic.kr/p/p51z9r Demo available at https://youtu.be/jetqMYnWjm8
107. Where to go from here
108. 120 MAKING IT WORK!
109. 121 ENABLE L2 EVPN
110. 122 • The architecture shown has one big limitation: we need to sacrifice one interface • Can overcome by using a Vlan interface • What if we want EVPN connectivity at day 0?
111. Running Podman Pods as systemd units 123 • The same smart-controller / dumb-FRR architecture can be started as Podman pods running as systemd units • It allows to have an EVPN-based primary interface • Still to be consolidated github.com/fedepaol/evpnlab/tree/main/08_from_kind_with_systemdunits fedepaol.github.io/blog/2025/01/06/enabling-evpn-termination-with-podman-pods-as-systemd-units/
112. A faster datapath? 124 https://flic.kr/p/p51z9r
113. Wrapping Up 125
114. Resources 126 • FRR Routing docs at frrouting.org • ContainerLab containerlab.dev • My EVPN Lab repo github.com/fedepaol/evpnlab/ • My personal Blog fedepaol.github.io/posts/ • OpenPERouter repo github.com/openperouter/openperouter • Das Schiff Network Operator github.com/telekom/das-schiff-network-operator
115. Thanks! Any questions? fedepaol.bsky.social hachyderm.io/@fedepaol Slides at: speakerdeck.com/fedepaol fedepaol@gmail.com