Bringing Security and Multi- tenancy to Kubernetes Lei (Harry) Zhang

About Me • Lei (Harry) Zhang • #Microsoft MVP in cloud and datacenter management • though I’m a Linux guy :/ • Previous: VMware, Baidu • Feature maintainer of Kubernetes • HyperCrew: https://hyper.sh • Publications: Docker & Kubernetes Under the Hood • PhD candidate @ZJU: Large-scale cluster management and scheduling

A survey about “boundary” • Are you comfortable with Linux containers as an effective boundary? • Yes, I use containers in my private/safe environment • No, I use containers to serve the public cloud

As long as we care security… • We have to wrap containers inside full-blown virtual machines • But we lose cloud-native deployment • Slow startup time • Huge resources wasting • Memory tax for every container • … dream reality

Revisit container • Container Runtime • The dynamic view and boundary of your running process • Container Image • The static view of your program, data, dependencies, files and directories namespace cgroups FROM busybox ADD temp.txt / VOLUME /data CMD [“echo hello"] Read-Write Layer & /data “echo hello” read-only layer /bin /dev /etc /home /lib / lib64 /media /mnt /opt /proc / root /run /sbin /sys /tmp / usr /var /data /temp.txt /etc/hosts /etc/hostname /etc/resolv.conf read-write layer /tem p.txt json json init layer FROM busybox ADD temp.txt / VOLUME /data CMD [“echo hello"] Docker Container

HyperContainer Secure Kubernetes from runtime level

HyperContainer • Container Runtime • RunV • https://github.com/hyperhq/runv • The OCI compatible hypervisor based runtime implementation • Widely adopted by companies like Huawei etc • Control daemon • https://github.com/hyperhq/hyperd • Container Image • Docker Image Spec

Combine the best parts • Portable and behaves like a Linux container • $ hyperctl run -t busybox echo helloworld • sub-second startup time*, ~12MB memory cost • Fully isolated sandbox with an independent guest kernel • $ hyperctl exec -t busybox uname -r • 4.4.12-hyper (or your provided kernel) • security, backward compatibility, maturity See: http://hypercontainer.io/why-hyper.html

HyperContainer is a Pod • That’s how HyperContainer fits into the Kubernetes philosophy • Wait, why Pod is so important?

Pod: lesson learned from Borg • Should sample.war be packaged with Tomcat?

Pod: lesson learned from Borg • InitContainers: one or more containers started in sequence before the pod's normal containers are started. • Share volumes, perform network operations, and perform computation prior to the app containers.

So, Pod is • The group of super-affinity containers • The atomic scheduling unit • The process group in container cloud • Do right things • without modifying your container image • Kubernetes = Spring Framework • Pod = IoC Pod log app infra container volume init container

Pod is not easy to simulate • log super affinity app • Requirement: • app: 1G, log: 0.5G • Available: • Node_A: 1.25G, Node_B: 2G • What happens if app scheduled to Node_A?

HyperContainer is a Pod • Linux container based runtimes • wraps and encapsulates several app containers into a logical group • Hypervisor container based runtime • hypervisor serves as a natural boundary of Pod

HyperContainer is a Pod • kubelet Container Runtime Interface • create sandbox Foo --> create container C --> start container C • stop container C --> remove container C --> delete sandbox Foo • Sandbox • Normally: the infra container • HyperContainer: hypervisor • with HyperKernel • a HyperStart process as PID 1 • setup mnt namespace, launch apps from the images etc

Hypernetes Kubernetes with HyperContainer Runtime

Hypernetes • Also: h8s 1. Kubernetes + HyperContainer runtime • officially supported by using kubernetes/frakti 2. Multi-tenant network and persistent volumes • battle tested Neutron + Cinder plugin

Multi-tenant Network

Multi-tenant Network • Goal: • leveraging tenant-aware neutron network for Kubernetes • following the network plugin workflow • Non-goal: • break k8s network model or hack k8s code

Define the Network • Network • a top class api object • each tenant (created by Keystone) has its own Network • Network mapping to Neutron “net” • a Network Controller is responsible to manage Network lifecycle

Example kubelet SyncLoop controller-manager ControlLoop kubelet SyncLoop proxy proxy network pod replica namespace service job deployment volume petset … etcd scheduler api-server Desired World Real World Call Neutron to create/delete network

Kubernetes Network Model • Container reach container • all containers can communicate with all other containers without NAT • Node reach container • all nodes can communicate with all containers (and vice-versa) without NAT • IP addressing • Pod in cluster can be addressed by its IP

How h8s fits that? • Network can be assigned to one or more Namespaces • Pods belonging to the same Network can reach each other directly through IP • a Pod’s network mapping to Neutron “port” • kubelet is responsible for Pod network setup • let’s see how kubelet works

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 1 Pod created etcd scheduler api-server

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 2 Pod object added etcd scheduler api-server

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 3.1 New pod object detected 3.2 Bind pod with node etcd scheduler api-server

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 4.1 Detected pod bind with me 4.2 Start containers in pod etcd scheduler api-server

Design of kubelet InitNetworkPlugin Choose Runtime ҁdocker, rkt, hyper/remote҂ InitNetworkPlugin HandlePods {Add, Update, Remove, Delete, …} NodeStatus Network Status status Manager PLEG SyncLoop Pod Update Worker (e.g.ADD) • generale Pod status • check volume status (talk later) • call runtime to start containers • set up Pod network (see next slide) volume Manager PodUpdate image Manager

Set Up Pod Network

kubestack A standalone gRPC daemon 1. to “translate” the SetUpPod request to the Neutron network API 2. handling multi-tenant Service proxy

Service $ iptables-save | grep my-service -A KUBE-SERVICES -d 10.0.0.116/32 -p tcp -m comment --comment "default/my-service: cluster IP" -m tcp --dport 8001 -j KUBE-SVC-KEAUNL7HVWWSEZA6 -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-6XXFWO3KTRMPKCHZ -A KUBE-SVC-KEAUNL7HVWWSEZA6 -m comment --comment "default/my-service:" --mode random -j KUBE-SEP-57KPRZ3JQVENLNBRZ -A KUBE-SEP-6XXFWO3KTRMPKCHZ -p tcp -m comment --comment "default/my-service:" -m tcp -j DNAT --to-destination 172.17.0.2:80 -A KUBE-SEP-57KPRZ3JQVENLNBRZ -p tcp -m comment --comment "default/my-service:" -m tcp -j DNAT --to-destination 172.17.0.3:80 portal 10.10.0.116:8001 random mode rules backend rule_1 backend rule_2 172.17.0.2.:80 172.17.0.3.:80 OnServiceUpdate OnEndpointsUpdate

Multi-tenant Service • Default iptables-based kube-proxy is not tenant aware • Endpoint Pods and Nodes with iptables rules are isolated into different networks • Hypernetes uses a built-in HAproxy as the Service portal • to handle all Service instances within same namespace • the same OnServiceUpdate and OnEndpointsUpdate workflow • ExternalProvider • a OpenStack LB will be created as Service • e.g. curl 58.215.33.98:8078

Persistent Volume

Kubernetes Persistent Volume Host path Cinder volume plugin Pod Pod mountPath mountPath attach mount Volume Manager desired World reconcile • Get mountedVolume from actualStateOfWorld • Unmount volumes in mountedVolume but not in desiredStateOfWorld • AttachVolume() if vol in desiredStateOfWorld and not attached • MountVolume() if vol in desiredStateOfWorld and not in mountedVolume • Verify devices that should be detached/unmounted are detached/unmounted • Tips: 1. -v host:path 2. attach VS mount 3. Totally independent from container management

Persistent Volume with HyperContainer • Enhanced Cinder volume plugin • Linux container: 1. full OpenStack cluster 2. query Nova to find node 3. attach Cinder volume to host path 4. bind mount host path to Pod containers • HyperContainer: • directly attach block devices to Pod • thanks to the hypervisor based Pod boundary • eliminates extra time to query Nova Host vol Enhanced Cinder volume plugin Pod Pod mountPath mountPath attach vol desired World reconcile Volume Manager

PV Example • Create a Cinder volume • Claim volume by reference its volumeID

Container Runtime Interface

Future of CRI • Keep Docker as the only one default container runtime • oci-runtime, rktlet, hyperd • Frakti: the Remote Container Runtime Kit • https://github.com/kubernetes/frakti • welcome to tryout, star and fork

“if image becomes non-standard” • e.g. Docker image becomes somehow Docker specific • Don’t worry, kubelet.imageManager is moving to runtime specific • but then k8s will probably choose • NO DEFAULT runtime

Node Node Full Topology Node kubestack Neutron L2 Agent kube-proxy kubelet Cinder Plugin Pod Pod Pod Pod KeyStone Neutron Cinder Master Object: Network Ceph Object: Pod Object: …

Summary • A new way to build secure and multi-tenant Kubernetes • Kubernetes + HyperContainer + Neutron Plugin + Cinder Plugin + Keystone • Project URL: https://github.com/hyperhq/hypernetes • Roadmap • Graduate HyperContainer runtime on k8s upstream • see HyperContainer in official k8s release • Neutron CNI plugin • Tip: https://hyper.sh is totally built on Hypernetes, try it out :)

END Lei (Harry) Zhang @resouer