Slide 1

Slide 1 text

Writing a CNI plugin from scratch Eran Yanay

Slide 2

Slide 2 text

Eran Yanay Twistlock R&D Team Lead [email protected] Who am I

Slide 3

Slide 3 text

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version =$(kubectl version | base64 | tr -d '\n')"

Slide 4

Slide 4 text

Writing a CNI (Container Network Interface) plugin from scratch, using only bash ● What is CNI? ● How CNI plugin works? ● What a CNI plugin is made of? ● How a CNI plugin is being used in K8s? ● How a CNI plugin is executed? ● Anatomy of pod networking ● Live demo Agenda

Slide 5

Slide 5 text

What is CNI? ● CNI stands for Container Networking Interface ● An interface between container runtime and the network implementation ● Configures the network interfaces and routes ● Concerns itself only with network connectivity ● https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md

Slide 6

Slide 6 text

How CNI plugin works (in k8s) ● A CNI binary Handles connectivity - configures the network interface of the pod ● A daemon Handles reachability - manages routings across the cluster

Slide 7

Slide 7 text

# cat /etc/cni/net.d/10-my-cni-demo.conf { "cniVersion": "0.3.1", "name": "my-cni-demo", "type": "my-cni-demo", "podcidr": "10.240.0.0/24", } What a CNI plugin is made of? # cat /opt/cni/bin/my-cni-demo case $CNI_COMMAND in ADD) # Configure networking for a new container ;; DEL) # Cleanup when container is stopped ;; GET) ;; VERSION) # Get the plugin version ;; esac

Slide 8

Slide 8 text

The weave example.. kind: DaemonSet spec: containers: - name: weave command: - /home/weave/launch.sh image: 'docker.io/weaveworks/weave-kube:2.5.1' volumeMounts: - name: weavedb mountPath: /weavedb - name: cni-bin mountPath: /host/opt - name: cni-bin2 mountPath: /host/home - name: cni-conf mountPath: /host/etc hostNetwork: true

Slide 9

Slide 9 text

The weave example.. $ cat /home/weave/launch.sh # ... previous non related code ... # Install CNI plugin binary to typical CNI bin location # with fall-back to CNI directory used by kube-up on GCI OS if ! mkdir -p $HOST_ROOT/opt/cni/bin ; then if mkdir -p $HOST_ROOT/home/kubernetes/bin ; then export WEAVE_CNI_PLUGIN_DIR=$HOST_ROOT/home/kubernetes/bin else echo "Failed to install the Weave CNI plugin" >&2 exit 1 fi fi mkdir -p $HOST_ROOT/etc/cni/net.d export HOST_ROOT /home/weave/weave --local setup-cni

Slide 10

Slide 10 text

How a CNI plugin is being used in K8s? API Server Kubelet Schedule a pod Host network namespace br

Slide 11

Slide 11 text

How a CNI plugin is being used in K8s? Host network namespace API Server Kubelet Create pod network ns Pod network namespace br lo

Slide 12

Slide 12 text

How a CNI plugin is being used in K8s? Host network namespace API Server Kubelet CNI_COMMAND: ADD Pod network namespace CNI br lo

Slide 13

Slide 13 text

How a CNI plugin is being used in K8s? Host network namespace API Server Kubelet Pod network namespace CNI eth0 ● Create eth0 ● Allocate IP ● Define routes br lo

Slide 14

Slide 14 text

How a CNI plugin is executed? Container info CNI config CNI env vars stdin

Slide 15

Slide 15 text

How a CNI plugin is executed? Container info CNI config CNI env vars stdin CNI_COMMAND=ADD CNI_CONTAINERID=b784318.. CNI_NETNS=/proc/12345/ns/net CNI_IFNAME=eth0

Slide 16

Slide 16 text

How a CNI plugin is executed? Container info CNI config CNI env vars stdin { "cniVersion" : "0.3.1", "name": "my-cni-demo" , "type": "my-cni-demo" , "podcidr" : "10.240.0.0/24" } CNI_COMMAND=ADD CNI_CONTAINERID=b784318.. CNI_NETNS=/proc/12345/ns/net CNI_IFNAME=eth0

Slide 17

Slide 17 text

{ "cniVersion" : "0.3.1", "interfaces" : [ { "name": "eth0", "mac": "ce:60:4c:b9:3a:06" , "sandbox" : "/proc/15116/ns/net" } ], "ips": [ { "version" : "4", "address" : "10.240.0.6/24" , "gateway" : "10.240.0.1" , "interface" : 0 } ] } How a CNI plugin is executed? Container info CNI config CNI env vars stdin

Slide 18

Slide 18 text

Anatomy of pod networking Linux host pod1

Slide 19

Slide 19 text

Anatomy of pod networking Linux host bridge pod1

Slide 20

Slide 20 text

Anatomy of pod networking Linux host pod1 veth bridge veth

Slide 21

Slide 21 text

Anatomy of pod networking Linux host bridge pod1 eth0 veth

Slide 22

Slide 22 text

Anatomy of pod networking Linux host pod1 eth0 eth0 veth pod2 pod3 pod4 veth veth veth bridge eth0 eth0 eth0

Slide 23

Slide 23 text

enp0s9 enp0s9 node1 node2 # cat /etc/cni/net.d/10-my-cni-demo.conf { "cniVersion": "0.3.1", "name": "my-cni-demo", "type": "my-cni-demo", "podcidr": "10.240.0.0/24" , } # cat /opt/cni/bin/my-cni-demo case $CNI_COMMAND in ADD) ;; # Configure networking DEL) ;; # Cleanup GET) ;; VERSION) ;; # Print plugin version esac 10.10.10.10/24 10.240.0.0/24 10.10.10.11/24 10.240.1.0/24 pod

Slide 24

Slide 24 text

CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 ;; enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge Container info

Slide 25

Slide 25 text

CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 ;; enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge Container info

Slide 26

Slide 26 text

CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge eth0 veth1 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ;; Container info

Slide 27

Slide 27 text

CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge eth0 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ip link set $host_ifname master cni0 # connect veth1 to bridge ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns ;; Container info veth1

Slide 28

Slide 28 text

CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge 10.240.0.2 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ip link set $host_ifname master cni0 # connect veth1 to bridge ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns # calculate $ip ip netns exec $CNI_CONTAINERID ip link set $CNI_IFNAME up ip netns exec $CNI_CONTAINERID ip addr add $ip/24 dev $CNI_IFNAME ip netns exec $CNI_CONTAINERID ip route add default via $podcidr_gw dev $CNI_IFNAME Container info eth0 veth1

Slide 29

Slide 29 text

CNI_CONTAINERID=b552f9… CNI_IFNAME=eth0 CNI_COMMAND=ADD CNI_NETNS=/proc/6137/ns/net enp0s9 node1 10.10.10.10/24 10.240.0.0/24 pod bridge 10.240.0.2 case $CNI_COMMAND in ADD) podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24 podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1 brctl addbr cni0 # create a new bridge (if doesnt exist), cni0 ip link set cni0 up ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0 host_ifname="veth$n" # n=1,2,3... ip link add $CNI_IFNAME type veth peer name $host_ifname ip link set $host_ifname up ip link set $host_ifname master cni0 # connect veth1 to bridge ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns # calculate $ip ip netns exec $CNI_CONTAINERID ip link set $CNI_IFNAME up ip netns exec $CNI_CONTAINERID ip addr add $ip/24 dev $CNI_IFNAME ip netns exec $CNI_CONTAINERID ip route add default via $podcidr_gw dev $CNI_IFNAME Container info eth0 veth1 if [ -f /tmp/last_allocated_ip ]; then n=`cat /tmp/last_allocated_ip` else n=1 fi ip=$(echo $podcidr | sed "s:0/24:$(( $n+1)):g") echo $(($n+1)) > /tmp/last_allocated_ip

Slide 30

Slide 30 text

Lets see a demo!

Slide 31

Slide 31 text

Thank you! Full implementation is available at https://github.com/eranyanay/cni-from-scratch