Writing a CNI plugin from scratch

Writing a CNI plugin from scratch

CNI (Container Network Interface) plugins are the cornerstone of Kubernetes networking.
CNI is the standardized way used by Kubernetes to expose network devices to pods,
responsible for pod to pod communication across physical nodes in your cluster.

During this talk we will:
- Explore the details of the CNI plugin interface
- Understand how it is used with Kubernetes
- Provide a detailed walkthrough of a simple CNI plugin from scratch

Attendees in this talk will gain insight into the process of creating a CNI plugin and get familiar with networking decisions required for having their pods connected and reachable from within the cluster and the internet.

All the examples in the slides are also available here: https://github.com/eranyanay/cni-from-scratch

6ff6d45c6cd8804aeba61a9262579ab2?s=128

Eran Yanay

May 16, 2019
Tweet

Transcript

  1. Writing a CNI plugin from scratch Eran Yanay

  2. Eran Yanay Twistlock R&D Team Lead eran@twistlock.com Who am I

  3. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version =$(kubectl version | base64 | tr

    -d '\n')"
  4. Writing a CNI (Container Network Interface) plugin from scratch, using

    only bash • What is CNI? • How CNI plugin works? • What a CNI plugin is made of? • How a CNI plugin is being used in K8s? • How a CNI plugin is executed? • Anatomy of pod networking • Live demo Agenda
  5. What is CNI? • CNI stands for Container Networking Interface

    • An interface between container runtime and the network implementation • Configures the network interfaces and routes • Concerns itself only with network connectivity • https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md
  6. How CNI plugin works (in k8s) • A CNI binary

    Handles connectivity - configures the network interface of the pod • A daemon Handles reachability - manages routings across the cluster
  7. # cat /etc/cni/net.d/10-my-cni-demo.conf { "cniVersion": "0.3.1", "name": "my-cni-demo", "type": "my-cni-demo",

    "podcidr": "10.240.0.0/24", } What a CNI plugin is made of? # cat /opt/cni/bin/my-cni-demo case $CNI_COMMAND in ADD) # Configure networking for a new container ;; DEL) # Cleanup when container is stopped ;; GET) ;; VERSION) # Get the plugin version ;; esac
  8. The weave example.. kind: DaemonSet spec: containers: - name: weave

    command: - /home/weave/launch.sh image: 'docker.io/weaveworks/weave-kube:2.5.1' volumeMounts: - name: weavedb mountPath: /weavedb - name: cni-bin mountPath: /host/opt - name: cni-bin2 mountPath: /host/home - name: cni-conf mountPath: /host/etc hostNetwork: true
  9. The weave example.. $ cat /home/weave/launch.sh # ... previous non

    related code ... # Install CNI plugin binary to typical CNI bin location # with fall-back to CNI directory used by kube-up on GCI OS if ! mkdir -p $HOST_ROOT/opt/cni/bin ; then if mkdir -p $HOST_ROOT/home/kubernetes/bin ; then export WEAVE_CNI_PLUGIN_DIR=$HOST_ROOT/home/kubernetes/bin else echo "Failed to install the Weave CNI plugin" >&2 exit 1 fi fi mkdir -p $HOST_ROOT/etc/cni/net.d export HOST_ROOT /home/weave/weave --local setup-cni
  10. How a CNI plugin is being used in K8s? API

    Server Kubelet Schedule a pod Host network namespace br
  11. How a CNI plugin is being used in K8s? Host

    network namespace API Server Kubelet Create pod network ns Pod network namespace br lo
  12. How a CNI plugin is being used in K8s? Host

    network namespace API Server Kubelet CNI_COMMAND: ADD Pod network namespace CNI br lo
  13. How a CNI plugin is being used in K8s? Host

    network namespace API Server Kubelet Pod network namespace CNI eth0 • Create eth0 • Allocate IP • Define routes br lo
  14. How a CNI plugin is executed? Container info CNI config

    CNI env vars stdin
  15. How a CNI plugin is executed? Container info CNI config

    CNI env vars stdin CNI_COMMAND=ADD CNI_CONTAINERID=b784318.. CNI_NETNS=/proc/12345/ns/net CNI_IFNAME=eth0
  16. How a CNI plugin is executed? Container info CNI config

    CNI env vars stdin { "cniVersion" : "0.3.1", "name": "my-cni-demo" , "type": "my-cni-demo" , "podcidr" : "10.240.0.0/24" } CNI_COMMAND=ADD CNI_CONTAINERID=b784318.. CNI_NETNS=/proc/12345/ns/net CNI_IFNAME=eth0
  17. { "cniVersion" : "0.3.1", "interfaces" : [ { "name": "eth0",

    "mac": "ce:60:4c:b9:3a:06" , "sandbox" : "/proc/15116/ns/net" } ], "ips": [ { "version" : "4", "address" : "10.240.0.6/24" , "gateway" : "10.240.0.1" , "interface" : 0 } ] } How a CNI plugin is executed? Container info CNI config CNI env vars stdin
  18. Anatomy of pod networking Linux host pod1

  19. Anatomy of pod networking Linux host bridge pod1

  20. Anatomy of pod networking Linux host pod1 veth bridge veth

  21. Anatomy of pod networking Linux host bridge pod1 eth0 veth

  22. Anatomy of pod networking Linux host pod1 eth0 eth0 veth

    pod2 pod3 pod4 veth veth veth bridge eth0 eth0 eth0
  23. enp0s9 enp0s9 node1 node2 # cat /etc/cni/net.d/10-my-cni-demo.conf { "cniVersion": "0.3.1",

    "name": "my-cni-demo", "type": "my-cni-demo", "podcidr": "10.240.0.0/24" , } # cat /opt/cni/bin/my-cni-demo case $CNI_COMMAND in ADD) ;; # Configure networking DEL) ;; # Cleanup GET) ;; VERSION) ;; # Print plugin version esac 10.10.10.10/24 10.240.0.0/24 10.10.10.11/24 10.240.1.0/24 pod
  24. CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin

    | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 ;; enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge Container info
  25. CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin

    | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 ;; enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge Container info
  26. CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge

    eth0 veth1 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ;; Container info
  27. CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge

    eth0 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ip link set $host_ifname master cni0 # connect veth1 to bridge ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns ;; Container info veth1
  28. CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge

    10.240.0.2 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ip link set $host_ifname master cni0 # connect veth1 to bridge ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns # calculate $ip ip netns exec $CNI_CONTAINERID ip link set $CNI_IFNAME up ip netns exec $CNI_CONTAINERID ip addr add $ip/24 dev $CNI_IFNAME ip netns exec $CNI_CONTAINERID ip route add default via $podcidr_gw dev $CNI_IFNAME Container info eth0 veth1
  29. CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge

    10.240.0.2 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ip link set $host_ifname master cni0 # connect veth1 to bridge ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns # calculate $ip ip netns exec $CNI_CONTAINERID ip link set $CNI_IFNAME up ip netns exec $CNI_CONTAINERID ip addr add $ip/24 dev $CNI_IFNAME ip netns exec $CNI_CONTAINERID ip route add default via $podcidr_gw dev $CNI_IFNAME Container info eth0 veth1 if [ -f /tmp/last_allocated_ip ]; then n=`cat /tmp/last_allocated_ip` else n=1 fi ip=$(echo $podcidr | sed "s:0/24:$(( $n+1)):g") echo $(($n+1)) > /tmp/last_allocated_ip
  30. Lets see a demo!

  31. Thank you! Full implementation is available at https://github.com/eranyanay/cni-from-scratch