CNI Introduction

- CNI Overview
　- Preface
　- What is CNI
　- CNI Introduction
- CNI Plugins
　- network plugins of containernetworking
　- 3rd Party Plugins
- Hands-on: CNI Plugins with Network Namespace
- CNI Network Controller Intro
- Deep Dive Into K8s Pod & CNI
　- Install pod network method
　- About CNI path of kubernetes
　- K8s pod with CNI
- Hands-on: Playing with POD Networking

1. CNI Introduction Date: 2018/12/06 Place: ITRI Presenter: Samina (Shan-Jung Fu) Prepare Hands-On Environment: http://bit.ly/2QtUMyZ
2. Outline • CNI Overview • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking 2
3. • CNI Overview ◦ Preface ◦ What is CNI ◦ CNI Introduction • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking Outline 3
4. • Preface • What is CNI • CNI Introduction Preface How the container setup network? Let’s take a look at Docker with default network mode - bridge. 4
5. • Preface • What is CNI • CNI Introduction 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 5 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 Create a container & setup network ns1 veth0 veth1
6. 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 6 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
7. 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 7 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
8. 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 8 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
9. 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 9 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
10. 10 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables • Preface • What is CNI • CNI Introduction Create a container & setup network
11. Preface (Cont.) Problem: • Many container technical (Docker, rkt, LXC...) • Many container cluster orchestration (K8s, Mesos...) • Many network setup ways (OVS, layer3 routing, overlay network...) Solve: • Provide an interface • Let everyone can write its own network service 11 Container Network Interface(CNI) • Preface • What is CNI • CNI Introduction
12. • Container Network Interface • The simplest possible interface between container runtime & network implementation • CNCF (Cloud Native Computing Foundation) Project What is CNI 12 • Preface • What is CNI • CNI Introduction
13. What is CNI (Cont.) 13 • Preface • What is CNI • CNI Introduction Two main Projects: • containernetworking/cni Libraries for writing plugins to configure network interfaces • containernetworking/plugins Additional CNI network plugins
14. What is CNI (Cont.) • CNI Specification ◦ The API between runtimes and network plugins • Conventions ◦ Extensions to the API that are not required for all plugins • Library ◦ A Go implementation of the CNI specification that plugins and runtimes can use 14 • Preface • What is CNI • CNI Introduction
15. Container Runtime Net Plugin Config CNI commands & network config Configures Network Ref: https://github.com/sufuf3/network-study-notes/blob/master/Container-Network/cni.md CNI Introduction 15 • Preface • What is CNI • CNI Introduction
16. CNI Introduction (Cont.) 16 • Preface • What is CNI • CNI Introduction Container runtime • A software • Executes containers • Manages container images on a node • Ecosystem ◦ Docker ◦ Rkt ◦ Containerd ◦ lxd
17. CNI Introduction (Cont.) 17 • Preface • What is CNI • CNI Introduction { "cniVersion": "0.6.0", "name": "mynet", "type": "my-plugin", "some-parameter": "foo", "ipam": { "type": "host-local", "subnet": "10.42.0.0/24", } } A json document defining the network my-config
18. CNI Introduction (Cont.) 18 • Preface • What is CNI • CNI Introduction CNI_COMMAND=ADD CNI_CONTAINERID=$id \ CNI_NETNS=/proc/$pid/ns/net \ CNI_PATH=/opt/cni/bin \ CNI_IFNAME=net0 \ my-plugin < my-config CNI_COMMAND=ADD \ CNI_CONTAINERID=ns1 \ CNI_NETNS=/var/run/netns/ns1 \ CNI_IFNAME=eth2 \ bridge <mybridge.conf Example
19. CNI_COMMAND=ADD CNI_CONTAINERID=$id \ CNI_NETNS=/proc/$pid/ns/net \ CNI_PATH=/opt/cni/bin \ CNI_IFNAME=net0 \ my-plugin < my-config CNI Introduction (Cont.) 19 • Preface • What is CNI • CNI Introduction https://github.com/containernetworking/cni/blob/maste r/SPEC.md#parameters • ADD: Add container to network • DEL: Delete container from network • VERSION: Report version
20. CNI Introduction (Cont.) 20 • Preface • What is CNI • CNI Introduction CNI_COMMAND=ADD \ CNI_CONTAINERID=ns1 \ CNI_NETNS=/var/run/netns/ns1 \ CNI_IFNAME=eth2 \ bridge <mybridge.conf Example CNI_COMMAND=ADD CNI_CONTAINERID=$id \ CNI_NETNS=/proc/$pid/ns/net \ CNI_PATH=/opt/cni/bin \ CNI_IFNAME=net0 \ my-plugin < my-config
21. Container Runtime Net Plugin Config CNI commands & network config Configures Network Ref: https://github.com/sufuf3/network-study-notes/blob/master/Container-Network/cni.md CNI Introduction (Cont.) 21 • Preface • What is CNI • CNI Introduction
22. Outline 22 • CNI Overview • CNI Plugins ◦ network plugins of containernetworking ◦ 3rd Party Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking
23. CNI Plugins 23 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
24. Network Plugins of containernetworking • Main: interface-creating • IPAM: IP address allocation • Meta: other plugins • Sample 24 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
25. • Main: interface-creating ◦ bridge: Creates a bridge, adds the host & the container to it. ◦ ipvlan: Adds an ipvlan interface in the container. ◦ loopback: Set the state of loopback interface to up. ◦ macvlan: Creates a new MAC address, forwards all traffic to that to the container. ◦ ptp: Creates a veth pair. ◦ vlan: Allocates a vlan device. ◦ host-device: Move an already-existing device into a container. Network Plugins of containernetworking (Cont.) 25 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
26. • Main: interface-creating • IPAM: IP address allocation ◦ dhcp: Runs a daemon on the host to make DHCP requests on behalf of the container ◦ host-local: Maintains a local database of allocated IPs ◦ static: Allocate a static IPv4/IPv6 addresses to container and it's useful in debugging purpose. • Meta: other plugins • Sample Network Plugins of containernetworking (Cont.) 26 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
27. • Main: interface-creating • IPAM: IP address allocation • Meta: other plugins ◦ flannel: Generates an interface corresponding to a flannel config file ◦ tuning: Tweaks sysctl parameters of an existing interface ◦ portmap: An iptables-based port mapping plugin. Maps ports from the host's address space to the container. ◦ bandwidth: Allows bandwidth-limiting through use of traffic control tbf (ingress/egress). • Sample Network Plugins of containernetworking (Cont.) 27 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
28. • Main: interface-creating • IPAM: IP address allocation • Meta: other plugins • Sample ◦ Provides an example for building your own plugin Network Plugins of containernetworking (Cont.) 28 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
29. Calico • Provides secure network connectivity workloads • Creates and manages a flat layer 3 network • Assigning each workload a fully routable IP address More 3rd party plugins: https://github.com/containernetworking/cni#3rd-party-plugins 29 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins 3rd Party Plugins
30. Outline • CNI Overview • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking 30
31. Host Hands-on Using bridge CNI Plugin with Network Namespaces 31 Network Namespace 0 (net0) Network Namespace 1 (net1) Veth0 Veth1 Bridge cni0 10.23.0.X/24 10.23.0.Y/24 Network config file cd ~/cni-hands-on/ mybridge.conf.tmp
32. Answer of Hands-on 32 sudo ip netns add ns1 sudo ip netns add ns2 cat > mybridge.conf <<"EOF" { "cniVersion": "0.2.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.23.0.0/24", "routes": [ { "dst": "0.0.0.0/0" } ] } } EOF cat > loopback.conf <<"EOF" { "cniVersion": "0.2.0", "type": "loopback" } EOF
33. sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns1 CNI_NETNS=/var/run/netns/ns1 CNI_IFNAME=lo CNI_PATH=`pwd` ./loopback < loopback.conf sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns2 CNI_NETNS=/var/run/netns/ns2 CNI_IFNAME=lo CNI_PATH=`pwd` ./loopback < loopback.conf sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns1 CNI_NETNS=/var/run/netns/ns1 CNI_IFNAME=net0 CNI_PATH=`pwd` ./bridge <mybridge.conf sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns2 CNI_NETNS=/var/run/netns/ns2 CNI_IFNAME=net0 CNI_PATH=`pwd` ./bridge <mybridge.conf Answer of Hands-on (Cont.) 33
34. Answer of Hands-on (Cont.) 34 sudo ip --all netns exec ip a sudo ip netns exec ns2 ping 10.23.0.2
35. Outline 35 • CNI Overview • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Introduction • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking
36. CNI Network Controller Introduction • Create multiple network interface of pod at init-container 36
37. CNI Network Controller Introduction 37
39. • CNI Overview • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI ◦ Install pod network method ◦ About CNI path of kubernetes ◦ K8s pod with CNI • Hands-on: Playing with POD Networking Outline 39
40. • Install pod network • CNI path • pod with CNI Install pod network method 40
41. • Install pod network • CNI path • pod with CNI About CNI path of K8s • /etc/cni/net.d ◦ netconf file • /opt/cni/bin ◦ CNI plugins binary file These files are on every K8s nodes. 41
42. • Install pod network • CNI path • pod with CNI • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 42
43. • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 43 Linux Host • Install pod network • CNI path • pod with CNI
44. • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 44 Linux Host Pause container • Install pod network • CNI path • pod with CNI
45. • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 45 Linux Host Pause container /etc/cni/net.d/ xxx.config Call /opt/cni/bin/CNI • Install pod network • CNI path • pod with CNI
46. • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 46 Linux Host Pause container Network Connect • Install pod network • CNI path • pod with CNI
47. K8s pod with CNI 47 Linux Host Pause container Network Connect • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container container1 container2 container3 • Install pod network • CNI path • pod with CNI
48. K8s pod with CNI 48 Linux Host Pause container Network Connect container1 container2 container3 • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container • Install pod network • CNI path • pod with CNI
49. K8s pod with CNI • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container 49 container1 Linux Host Pause container Network Connect container1 container2 container3 Pod • Install pod network • CNI path • pod with CNI
50. • CNI Overview • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking Outline 50
51. Network namespace path: /proc/<PID>/ns/net 1. Find containerID via kubectl 2. Find PID via docker command 3. monitor POD network Playing with POD Networking 51 kubectl get po <POD-NAME> -o jsonpath='{.status.containerStatuses[0].containerID} ' | cut -c 10-21 PID=$(sudo docker inspect c94df80d080e --format '{{ .State.Pid }}') sudo nsenter -t ${PID} -n ip addr
52. Playing with POD Networking (Cont.) 52 $ $ $ $ $ kubectl create -f https://k8s.io/examples/application/shell-demo.yaml kubectl get po -o wide kubectl get po shell-demo -o jsonpath='{.status.containerStatuses[0].containerID }' | cut -c 10-21 c94df80d080e PID=$(sudo docker inspect c94df80d080e --format '{{ .State.Pid }}') sudo nsenter -t ${PID} -n ip addr
53. References • https://github.com/containernetworking/cni • https://www.slideshare.net/hongweiqiu/introduction -to-cni-container-network-interface 53
54. Thanks for your attention. Slides: http://bit.ly/cni1206 GitHub, Telegram: @sufuf3 Twitter: @sufuf3149 54