Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CNI Introduction

CNI Introduction

- CNI Overview
 - Preface
 - What is CNI
 - CNI Introduction
- CNI Plugins
 - network plugins of containernetworking
 - 3rd Party Plugins
- Hands-on: CNI Plugins with Network Namespace
- CNI Network Controller Intro
- Deep Dive Into K8s Pod & CNI
 - Install pod network method
 - About CNI path of kubernetes
 - K8s pod with CNI
- Hands-on: Playing with POD Networking

Samina (Shan Jung Fu)

December 06, 2018
Tweet

More Decks by Samina (Shan Jung Fu)

Other Decks in Technology

Transcript

  1. CNI Introduction Date: 2018/12/06 Place: ITRI Presenter: Samina (Shan-Jung Fu)

    Prepare Hands-On Environment: http://bit.ly/2QtUMyZ
  2. Outline • CNI Overview • CNI Plugins • Hands-on: CNI

    Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking 2
  3. • CNI Overview ◦ Preface ◦ What is CNI ◦

    CNI Introduction • CNI Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking Outline 3
  4. • Preface • What is CNI • CNI Introduction Preface

    How the container setup network? Let’s take a look at Docker with default network mode - bridge. 4
  5. • Preface • What is CNI • CNI Introduction 1.

    Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 5 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 Create a container & setup network ns1 veth0 veth1
  6. 1. Create a linux bridge 2. Create a linux network

    namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 6 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
  7. 1. Create a linux bridge 2. Create a linux network

    namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 7 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
  8. 1. Create a linux bridge 2. Create a linux network

    namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 8 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
  9. 1. Create a linux bridge 2. Create a linux network

    namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables 9 Linux Host Linux Host br0 Linux Host br0 ns1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 • Preface • What is CNI • CNI Introduction Create a container & setup network
  10. 10 Linux Host Linux Host br0 Linux Host br0 ns1

    Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 Linux Host br0 ns1 veth0 veth1 1. Create a linux bridge 2. Create a linux network namespace 3. Create a veth pair 4. Attach veth into namespace 5. Attach veth into the linux bridge 6. Setup IP address 7. Setup route rules 8. Setup iptables • Preface • What is CNI • CNI Introduction Create a container & setup network
  11. Preface (Cont.) Problem: • Many container technical (Docker, rkt, LXC...)

    • Many container cluster orchestration (K8s, Mesos...) • Many network setup ways (OVS, layer3 routing, overlay network...) Solve: • Provide an interface • Let everyone can write its own network service 11 Container Network Interface(CNI) • Preface • What is CNI • CNI Introduction
  12. • Container Network Interface • The simplest possible interface between

    container runtime & network implementation • CNCF (Cloud Native Computing Foundation) Project What is CNI 12 • Preface • What is CNI • CNI Introduction
  13. What is CNI (Cont.) 13 • Preface • What is

    CNI • CNI Introduction Two main Projects: • containernetworking/cni Libraries for writing plugins to configure network interfaces • containernetworking/plugins Additional CNI network plugins
  14. What is CNI (Cont.) • CNI Specification ◦ The API

    between runtimes and network plugins • Conventions ◦ Extensions to the API that are not required for all plugins • Library ◦ A Go implementation of the CNI specification that plugins and runtimes can use 14 • Preface • What is CNI • CNI Introduction
  15. Container Runtime Net Plugin Config CNI commands & network config

    Configures Network Ref: https://github.com/sufuf3/network-study-notes/blob/master/Container-Network/cni.md CNI Introduction 15 • Preface • What is CNI • CNI Introduction
  16. CNI Introduction (Cont.) 16 • Preface • What is CNI

    • CNI Introduction Container runtime • A software • Executes containers • Manages container images on a node • Ecosystem ◦ Docker ◦ Rkt ◦ Containerd ◦ lxd
  17. CNI Introduction (Cont.) 17 • Preface • What is CNI

    • CNI Introduction { "cniVersion": "0.6.0", "name": "mynet", "type": "my-plugin", "some-parameter": "foo", "ipam": { "type": "host-local", "subnet": "10.42.0.0/24", } } A json document defining the network my-config
  18. CNI Introduction (Cont.) 18 • Preface • What is CNI

    • CNI Introduction CNI_COMMAND=ADD CNI_CONTAINERID=$id \ CNI_NETNS=/proc/$pid/ns/net \ CNI_PATH=/opt/cni/bin \ CNI_IFNAME=net0 \ my-plugin < my-config CNI_COMMAND=ADD \ CNI_CONTAINERID=ns1 \ CNI_NETNS=/var/run/netns/ns1 \ CNI_IFNAME=eth2 \ bridge <mybridge.conf Example
  19. CNI_COMMAND=ADD CNI_CONTAINERID=$id \ CNI_NETNS=/proc/$pid/ns/net \ CNI_PATH=/opt/cni/bin \ CNI_IFNAME=net0 \ my-plugin

    < my-config CNI Introduction (Cont.) 19 • Preface • What is CNI • CNI Introduction https://github.com/containernetworking/cni/blob/maste r/SPEC.md#parameters • ADD: Add container to network • DEL: Delete container from network • VERSION: Report version
  20. CNI Introduction (Cont.) 20 • Preface • What is CNI

    • CNI Introduction CNI_COMMAND=ADD \ CNI_CONTAINERID=ns1 \ CNI_NETNS=/var/run/netns/ns1 \ CNI_IFNAME=eth2 \ bridge <mybridge.conf Example CNI_COMMAND=ADD CNI_CONTAINERID=$id \ CNI_NETNS=/proc/$pid/ns/net \ CNI_PATH=/opt/cni/bin \ CNI_IFNAME=net0 \ my-plugin < my-config
  21. Container Runtime Net Plugin Config CNI commands & network config

    Configures Network Ref: https://github.com/sufuf3/network-study-notes/blob/master/Container-Network/cni.md CNI Introduction (Cont.) 21 • Preface • What is CNI • CNI Introduction
  22. Outline 22 • CNI Overview • CNI Plugins ◦ network

    plugins of containernetworking ◦ 3rd Party Plugins • Hands-on: CNI Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking
  23. Network Plugins of containernetworking • Main: interface-creating • IPAM: IP

    address allocation • Meta: other plugins • Sample 24 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
  24. • Main: interface-creating ◦ bridge: Creates a bridge, adds the

    host & the container to it. ◦ ipvlan: Adds an ipvlan interface in the container. ◦ loopback: Set the state of loopback interface to up. ◦ macvlan: Creates a new MAC address, forwards all traffic to that to the container. ◦ ptp: Creates a veth pair. ◦ vlan: Allocates a vlan device. ◦ host-device: Move an already-existing device into a container. Network Plugins of containernetworking (Cont.) 25 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
  25. • Main: interface-creating • IPAM: IP address allocation ◦ dhcp:

    Runs a daemon on the host to make DHCP requests on behalf of the container ◦ host-local: Maintains a local database of allocated IPs ◦ static: Allocate a static IPv4/IPv6 addresses to container and it's useful in debugging purpose. • Meta: other plugins • Sample Network Plugins of containernetworking (Cont.) 26 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
  26. • Main: interface-creating • IPAM: IP address allocation • Meta:

    other plugins ◦ flannel: Generates an interface corresponding to a flannel config file ◦ tuning: Tweaks sysctl parameters of an existing interface ◦ portmap: An iptables-based port mapping plugin. Maps ports from the host's address space to the container. ◦ bandwidth: Allows bandwidth-limiting through use of traffic control tbf (ingress/egress). • Sample Network Plugins of containernetworking (Cont.) 27 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
  27. • Main: interface-creating • IPAM: IP address allocation • Meta:

    other plugins • Sample ◦ Provides an example for building your own plugin Network Plugins of containernetworking (Cont.) 28 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins
  28. Calico • Provides secure network connectivity workloads • Creates and

    manages a flat layer 3 network • Assigning each workload a fully routable IP address More 3rd party plugins: https://github.com/containernetworking/cni#3rd-party-plugins 29 • CNI Plugins ◦ Containernetworking ◦ 3rd Party Plugins 3rd Party Plugins
  29. Outline • CNI Overview • CNI Plugins • Hands-on: CNI

    Plugins with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking 30
  30. Host Hands-on Using bridge CNI Plugin with Network Namespaces 31

    Network Namespace 0 (net0) Network Namespace 1 (net1) Veth0 Veth1 Bridge cni0 10.23.0.X/24 10.23.0.Y/24 Network config file cd ~/cni-hands-on/ mybridge.conf.tmp
  31. Answer of Hands-on 32 sudo ip netns add ns1 sudo

    ip netns add ns2 cat > mybridge.conf <<"EOF" { "cniVersion": "0.2.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.23.0.0/24", "routes": [ { "dst": "0.0.0.0/0" } ] } } EOF cat > loopback.conf <<"EOF" { "cniVersion": "0.2.0", "type": "loopback" } EOF
  32. sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns1 CNI_NETNS=/var/run/netns/ns1 CNI_IFNAME=lo CNI_PATH=`pwd` ./loopback < loopback.conf sudo

    CNI_COMMAND=ADD CNI_CONTAINERID=ns2 CNI_NETNS=/var/run/netns/ns2 CNI_IFNAME=lo CNI_PATH=`pwd` ./loopback < loopback.conf sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns1 CNI_NETNS=/var/run/netns/ns1 CNI_IFNAME=net0 CNI_PATH=`pwd` ./bridge <mybridge.conf sudo CNI_COMMAND=ADD CNI_CONTAINERID=ns2 CNI_NETNS=/var/run/netns/ns2 CNI_IFNAME=net0 CNI_PATH=`pwd` ./bridge <mybridge.conf Answer of Hands-on (Cont.) 33
  33. Answer of Hands-on (Cont.) 34 sudo ip --all netns exec

    ip a sudo ip netns exec ns2 ping 10.23.0.2
  34. Outline 35 • CNI Overview • CNI Plugins • Hands-on:

    CNI Plugins with Network Namespace • CNI Network Controller Introduction • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking
  35. 38

  36. • CNI Overview • CNI Plugins • Hands-on: CNI Plugins

    with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI ◦ Install pod network method ◦ About CNI path of kubernetes ◦ K8s pod with CNI • Hands-on: Playing with POD Networking Outline 39
  37. • Install pod network • CNI path • pod with

    CNI Install pod network method 40
  38. • Install pod network • CNI path • pod with

    CNI About CNI path of K8s • /etc/cni/net.d ◦ netconf file • /opt/cni/bin ◦ CNI plugins binary file These files are on every K8s nodes. 41
  39. • Install pod network • CNI path • pod with

    CNI • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 42
  40. • User post Pod config to master • Master find

    a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 43 Linux Host • Install pod network • CNI path • pod with CNI
  41. • User post Pod config to master • Master find

    a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 44 Linux Host Pause container • Install pod network • CNI path • pod with CNI
  42. • User post Pod config to master • Master find

    a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 45 Linux Host Pause container /etc/cni/net.d/ xxx.config Call /opt/cni/bin/CNI • Install pod network • CNI path • pod with CNI
  43. • User post Pod config to master • Master find

    a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container K8s pod with CNI 46 Linux Host Pause container Network Connect • Install pod network • CNI path • pod with CNI
  44. K8s pod with CNI 47 Linux Host Pause container Network

    Connect • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container container1 container2 container3 • Install pod network • CNI path • pod with CNI
  45. K8s pod with CNI 48 Linux Host Pause container Network

    Connect container1 container2 container3 • User post Pod config to master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container • Install pod network • CNI path • pod with CNI
  46. K8s pod with CNI • User post Pod config to

    master • Master find a node to deploy the pod • Create a Pause container • Load CNI network config • Execute the CNI plugin • Create target containers • Attach to Pause container 49 container1 Linux Host Pause container Network Connect container1 container2 container3 Pod • Install pod network • CNI path • pod with CNI
  47. • CNI Overview • CNI Plugins • Hands-on: CNI Plugins

    with Network Namespace • CNI Network Controller Intro • Deep Dive Into K8s Pod & CNI • Hands-on: Playing with POD Networking Outline 50
  48. Network namespace path: /proc/<PID>/ns/net 1. Find containerID via kubectl 2.

    Find PID via docker command 3. monitor POD network Playing with POD Networking 51 kubectl get po <POD-NAME> -o jsonpath='{.status.containerStatuses[0].containerID} ' | cut -c 10-21 PID=$(sudo docker inspect c94df80d080e --format '{{ .State.Pid }}') sudo nsenter -t ${PID} -n ip addr
  49. Playing with POD Networking (Cont.) 52 $ $ $ $

    $ kubectl create -f https://k8s.io/examples/application/shell-demo.yaml kubectl get po -o wide kubectl get po shell-demo -o jsonpath='{.status.containerStatuses[0].containerID }' | cut -c 10-21 c94df80d080e PID=$(sudo docker inspect c94df80d080e --format '{{ .State.Pid }}') sudo nsenter -t ${PID} -n ip addr