Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing a CNI plugin from scratch

Writing a CNI plugin from scratch

CNI (Container Network Interface) plugins are the cornerstone of Kubernetes networking.
CNI is the standardized way used by Kubernetes to expose network devices to pods,
responsible for pod to pod communication across physical nodes in your cluster.

During this talk we will:
- Explore the details of the CNI plugin interface
- Understand how it is used with Kubernetes
- Provide a detailed walkthrough of a simple CNI plugin from scratch

Attendees in this talk will gain insight into the process of creating a CNI plugin and get familiar with networking decisions required for having their pods connected and reachable from within the cluster and the internet.

All the examples in the slides are also available here: https://github.com/eranyanay/cni-from-scratch

Eran Yanay

May 16, 2019
Tweet

More Decks by Eran Yanay

Other Decks in Programming

Transcript

  1. Writing a CNI plugin from
    scratch
    Eran Yanay

    View Slide

  2. Eran Yanay
    Twistlock
    R&D Team Lead
    [email protected]
    Who am I

    View Slide

  3. kubectl apply -f
    "https://cloud.weave.works/k8s/net?k8s-version
    =$(kubectl version | base64 | tr -d '\n')"

    View Slide

  4. Writing a CNI (Container Network Interface) plugin from scratch, using only bash
    ● What is CNI?
    ● How CNI plugin works?
    ● What a CNI plugin is made of?
    ● How a CNI plugin is being used in K8s?
    ● How a CNI plugin is executed?
    ● Anatomy of pod networking
    ● Live demo
    Agenda

    View Slide

  5. What is CNI?
    ● CNI stands for Container Networking Interface
    ● An interface between container runtime and the network implementation
    ● Configures the network interfaces and routes
    ● Concerns itself only with network connectivity
    ● https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md

    View Slide

  6. How CNI plugin works (in k8s)
    ● A CNI binary
    Handles connectivity - configures the network interface of the pod
    ● A daemon
    Handles reachability - manages routings across the cluster

    View Slide

  7. # cat /etc/cni/net.d/10-my-cni-demo.conf
    {
    "cniVersion": "0.3.1",
    "name": "my-cni-demo",
    "type": "my-cni-demo",
    "podcidr": "10.240.0.0/24",
    }
    What a CNI plugin is made of?
    # cat /opt/cni/bin/my-cni-demo
    case $CNI_COMMAND in
    ADD)
    # Configure networking for a new container
    ;;
    DEL)
    # Cleanup when container is stopped
    ;;
    GET)
    ;;
    VERSION)
    # Get the plugin version
    ;;
    esac

    View Slide

  8. The weave example..
    kind: DaemonSet
    spec:
    containers:
    - name: weave
    command:
    - /home/weave/launch.sh
    image: 'docker.io/weaveworks/weave-kube:2.5.1'
    volumeMounts:
    - name: weavedb
    mountPath: /weavedb
    - name: cni-bin
    mountPath: /host/opt
    - name: cni-bin2
    mountPath: /host/home
    - name: cni-conf
    mountPath: /host/etc
    hostNetwork: true

    View Slide

  9. The weave example..
    $ cat /home/weave/launch.sh
    # ... previous non related code ...
    # Install CNI plugin binary to typical CNI bin location
    # with fall-back to CNI directory used by kube-up on GCI OS
    if ! mkdir -p $HOST_ROOT/opt/cni/bin ; then
    if mkdir -p $HOST_ROOT/home/kubernetes/bin ; then
    export WEAVE_CNI_PLUGIN_DIR=$HOST_ROOT/home/kubernetes/bin
    else
    echo "Failed to install the Weave CNI plugin" >&2
    exit 1
    fi
    fi
    mkdir -p $HOST_ROOT/etc/cni/net.d
    export HOST_ROOT
    /home/weave/weave --local setup-cni

    View Slide

  10. How a CNI plugin is being used in K8s?
    API Server Kubelet
    Schedule a pod
    Host network
    namespace
    br

    View Slide

  11. How a CNI plugin is being used in K8s?
    Host network
    namespace
    API Server Kubelet
    Create pod network ns
    Pod network
    namespace br
    lo

    View Slide

  12. How a CNI plugin is being used in K8s?
    Host network
    namespace
    API Server Kubelet
    CNI_COMMAND: ADD
    Pod network
    namespace
    CNI
    br
    lo

    View Slide

  13. How a CNI plugin is being used in K8s?
    Host network
    namespace
    API Server Kubelet
    Pod network
    namespace
    CNI
    eth0
    ● Create eth0
    ● Allocate IP
    ● Define routes
    br
    lo

    View Slide

  14. How a CNI plugin is executed?
    Container
    info
    CNI
    config
    CNI
    env vars
    stdin

    View Slide

  15. How a CNI plugin is executed?
    Container
    info
    CNI
    config
    CNI
    env vars
    stdin
    CNI_COMMAND=ADD
    CNI_CONTAINERID=b784318..
    CNI_NETNS=/proc/12345/ns/net
    CNI_IFNAME=eth0

    View Slide

  16. How a CNI plugin is executed?
    Container
    info
    CNI
    config
    CNI
    env vars
    stdin
    {
    "cniVersion" : "0.3.1",
    "name": "my-cni-demo" ,
    "type": "my-cni-demo" ,
    "podcidr" : "10.240.0.0/24"
    }
    CNI_COMMAND=ADD
    CNI_CONTAINERID=b784318..
    CNI_NETNS=/proc/12345/ns/net
    CNI_IFNAME=eth0

    View Slide

  17. {
    "cniVersion" : "0.3.1",
    "interfaces" : [
    {
    "name": "eth0",
    "mac": "ce:60:4c:b9:3a:06" ,
    "sandbox" : "/proc/15116/ns/net"
    }
    ],
    "ips": [
    {
    "version" : "4",
    "address" : "10.240.0.6/24" ,
    "gateway" : "10.240.0.1" ,
    "interface" : 0
    }
    ]
    }
    How a CNI plugin is executed?
    Container
    info
    CNI
    config
    CNI
    env vars
    stdin

    View Slide

  18. Anatomy of pod networking
    Linux host
    pod1

    View Slide

  19. Anatomy of pod networking
    Linux host
    bridge
    pod1

    View Slide

  20. Anatomy of pod networking
    Linux host
    pod1
    veth
    bridge
    veth

    View Slide

  21. Anatomy of pod networking
    Linux host
    bridge
    pod1
    eth0
    veth

    View Slide

  22. Anatomy of pod networking
    Linux host
    pod1
    eth0
    eth0
    veth
    pod2 pod3 pod4
    veth
    veth
    veth
    bridge
    eth0 eth0 eth0

    View Slide

  23. enp0s9
    enp0s9
    node1
    node2
    # cat /etc/cni/net.d/10-my-cni-demo.conf
    {
    "cniVersion": "0.3.1",
    "name": "my-cni-demo",
    "type": "my-cni-demo",
    "podcidr": "10.240.0.0/24"
    ,
    }
    # cat /opt/cni/bin/my-cni-demo
    case $CNI_COMMAND in
    ADD)
    ;; # Configure networking
    DEL)
    ;; # Cleanup
    GET)
    ;;
    VERSION)
    ;; # Print plugin version
    esac
    10.10.10.10/24
    10.240.0.0/24
    10.10.10.11/24
    10.240.1.0/24
    pod

    View Slide

  24. CNI_CONTAINERID=b552f9…
    CNI_IFNAME=eth0
    CNI_COMMAND=ADD
    CNI_NETNS=/proc/6137/ns/net
    case $CNI_COMMAND in
    ADD)
    podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24
    podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1
    ;;
    enp0s9
    node1
    10.10.10.10/24
    10.240.0.0/24
    pod
    bridge
    Container info

    View Slide

  25. CNI_CONTAINERID=b552f9…
    CNI_IFNAME=eth0
    CNI_COMMAND=ADD
    CNI_NETNS=/proc/6137/ns/net
    case $CNI_COMMAND in
    ADD)
    podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24
    podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1
    brctl addbr cni0 # create a new bridge (if doesnt exist), cni0
    ip link set cni0 up
    ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0
    ;; enp0s9
    node1
    10.10.10.10/24
    10.240.0.0/24
    pod
    bridge
    Container info

    View Slide

  26. CNI_CONTAINERID=b552f9…
    CNI_IFNAME=eth0
    CNI_COMMAND=ADD
    CNI_NETNS=/proc/6137/ns/net
    enp0s9
    node1
    10.10.10.10/24
    10.240.0.0/24
    pod
    bridge
    eth0
    veth1
    case $CNI_COMMAND in
    ADD)
    podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24
    podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1
    brctl addbr cni0 # create a new bridge (if doesnt exist), cni0
    ip link set cni0 up
    ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0
    host_ifname="veth$n" # n=1,2,3...
    ip link add $CNI_IFNAME type veth peer name $host_ifname
    ip link set $host_ifname up
    ;;
    Container info

    View Slide

  27. CNI_CONTAINERID=b552f9…
    CNI_IFNAME=eth0
    CNI_COMMAND=ADD
    CNI_NETNS=/proc/6137/ns/net
    enp0s9
    node1
    10.10.10.10/24
    10.240.0.0/24
    pod
    bridge
    eth0
    case $CNI_COMMAND in
    ADD)
    podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24
    podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1
    brctl addbr cni0 # create a new bridge (if doesnt exist), cni0
    ip link set cni0 up
    ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0
    host_ifname="veth$n" # n=1,2,3...
    ip link add $CNI_IFNAME type veth peer name $host_ifname
    ip link set $host_ifname up
    ip link set $host_ifname master cni0 # connect veth1 to bridge
    ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID
    ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns
    ;;
    Container info
    veth1

    View Slide

  28. CNI_CONTAINERID=b552f9…
    CNI_IFNAME=eth0
    CNI_COMMAND=ADD
    CNI_NETNS=/proc/6137/ns/net
    enp0s9
    node1
    10.10.10.10/24
    10.240.0.0/24
    pod
    bridge
    10.240.0.2
    case $CNI_COMMAND in
    ADD)
    podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24
    podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1
    brctl addbr cni0 # create a new bridge (if doesnt exist), cni0
    ip link set cni0 up
    ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0
    host_ifname="veth$n" # n=1,2,3...
    ip link add $CNI_IFNAME type veth peer name $host_ifname
    ip link set $host_ifname up
    ip link set $host_ifname master cni0 # connect veth1 to bridge
    ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID
    ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns
    # calculate $ip
    ip netns exec $CNI_CONTAINERID ip link set $CNI_IFNAME up
    ip netns exec $CNI_CONTAINERID ip addr add $ip/24 dev $CNI_IFNAME
    ip netns exec $CNI_CONTAINERID ip route add default via $podcidr_gw
    dev $CNI_IFNAME
    Container info
    eth0
    veth1

    View Slide

  29. CNI_CONTAINERID=b552f9…
    CNI_IFNAME=eth0
    CNI_COMMAND=ADD
    CNI_NETNS=/proc/6137/ns/net
    enp0s9
    node1
    10.10.10.10/24
    10.240.0.0/24
    pod
    bridge
    10.240.0.2
    case $CNI_COMMAND in
    ADD)
    podcidr=$(cat /dev/stdin | jq -r ".podcidr") # 10.240.0.0/24
    podcidr_gw=$(echo $podcidr | sed "s:0/24:1:g") # 10.240.0.1
    brctl addbr cni0 # create a new bridge (if doesnt exist), cni0
    ip link set cni0 up
    ip addr add "${podcidr_gw}/24" dev cni0 # assign 10.240.0.1/24 to cni0
    host_ifname="veth$n" # n=1,2,3...
    ip link add $CNI_IFNAME type veth peer name $host_ifname
    ip link set $host_ifname up
    ip link set $host_ifname master cni0 # connect veth1 to bridge
    ln -sfT $CNI_NETNS /var/run/netns/$CNI_CONTAINERID
    ip link set $CNI_IFNAME netns $CNI_CONTAINERID # move eth0 to pod ns
    # calculate $ip
    ip netns exec $CNI_CONTAINERID ip link set $CNI_IFNAME up
    ip netns exec $CNI_CONTAINERID ip addr add $ip/24 dev $CNI_IFNAME
    ip netns exec $CNI_CONTAINERID ip route add default via $podcidr_gw
    dev $CNI_IFNAME
    Container info
    eth0
    veth1
    if [ -f /tmp/last_allocated_ip ]; then
    n=`cat /tmp/last_allocated_ip`
    else
    n=1
    fi
    ip=$(echo $podcidr | sed "s:0/24:$((
    $n+1)):g")
    echo $(($n+1)) > /tmp/last_allocated_ip

    View Slide

  30. Lets see a demo!

    View Slide

  31. Thank you!
    Full implementation is available at https://github.com/eranyanay/cni-from-scratch

    View Slide