Go Programming Pattern in Kubernetes Philosophy Harry Zhang @resouer

Contents • What I will talk? • Kubernetes basic in 1 min • For Kubernetes developers: • The Golang programming patterns of Kubernetes (Controller, codegen etc) • Write your own Controller • gPRC based interface design in Kubernetes (CRI as example) • For Kubernetes users: • Effective pattern of programming based on Kubernetes • ⼴广告（Don’t worry, it’s not that kind of AD） • What I will not talk? • Kubernetes usage and under the hood • Internal systems or commercial software

Kubernetes • The container orchestration and management project created by Google • Successor of Google Borg/Omega system • One of the most popular open source projects in this world • Written by, and heavily depends on Golang

Again: Why Go? • All about community • A sad story: • https://github.com/google/lmctfy • Now think about a C/C++ based Kubernetes? • And, well designed programming patterns with powerful extensibility

Understand Kubernetes in 1 min Container Pod Deployment I need co-scheduling I have many Pod replicas Service I want to proxy my Pod replicas Ingress I want to expose my Services DaemonSet I run as daemon StatefulSet I am stateful Job I only run for once CronJob I run periodically ConfigMap I read configure file Secret I need confidential data HPA I need auto-scaling

Understand Kubernetes in 2 min • kubectl run nginx —image=nginx:1.7.9 —replicas=3 apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 • kubectl create -f deployment.yaml • kubectl create -f hpa.yaml apiVersion: autoscaling/v2alpha1 kind: HorizontalPodAutoscaler metadata: name: php-apache namespace: default spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: nginx-deployment minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 • API Object Oriented Programming

Core of API “OO” 1.API objects stores in etcd 2.Control loops (Sync Loop) to reconcile API objects

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 1 Pod created etcd scheduler api-server

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 2 Object added etcd scheduler api-server

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 3.1 New Pod detected 3.2 Bind Pod to a node etcd scheduler api-server

Example kubelet SyncLoop kubelet SyncLoop proxy proxy 4.1 Detected bind operation 4.2 Start Pod on this machine etcd scheduler api-server

Pattern 1: Controller • Control everything by Controller • Level driven, not edge driven edge level Image: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic

Controller • The heart of Kubernetes orchestrator • drives the cluster state based on the changes to the API objects for { desired := getDesiredState() current := getCurrentState() makeChanges(desired, current) } • Write your own controller!

Why DIY? Container Pod Deployment I need co-scheduling I have many Pod replicas Service I want to proxy my Pod replicas Ingress I want to expose my Services DaemonSet I run as daemon StatefulSet I am stateful Job I only run for once CronJob I run periodically ConfigMap I read configure file Secret I need confidential data HPA I need auto-scaling My Awesome Object I have my own special case

Demo • I want to have a Asta Xie object into k8s API • I want a controller to handle add/update/delete of all Asta Xie instances $ kubectl get astaxie NAME KIND astaxie1 AstaXie.v1.cr.client-go.k8s.io

My AstaXie Object & Controller etcd kube-apiserver types register AstaXie Controller astaxie1 OnDelete OnUpdate OnAdd Kubernetes Custom Controller User operation

A Real World Example • I want to have a Network object into k8s API • I want a controller to handle add/update/delete of all Network instances • onAdd: create Neutron network • onDelete: delete Neutron network • onUpdate: update Network object status • https://github.com/openstack/stackube/blob/master/pkg/network- controller/network_controller.go

Pattern 2: Gode Generator • client-gen: generate typed Kubernetes AP client for type • client.Pod.Get().Resource(…).Do() • conversion-gen: seamless upgrades between API versions • apiVersion: k8s.io/v1alpha1 -> apiVersion: k8s.io/v1beta1 • deepcopy-gen: deepcopy • go get k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/deepcopy-gen • deepcopy-gen -i ./pkg/apis/v1 • defaulter-gen: set default values for fields • go-to-protobuf: generate protobuf messages for your types • informer-gen: generate informers that can be used to watch for updates to your types • openapi-gen: generate openapi compatible API documentation

More Reference • github.com/kubernetes/gengo • github.com/kubernetes/kubernetes/tree/master/cmd/libs/go2idl

Pattern 3: gRPC based Interface • Decouple Kubernetes from external dependencies • kubelet -> gRPC -> dockershim -> dockerd • go2idl: gogoprotobuf based protobuf gen

CRI Management kubelet Workloads Orchestration kubelet SyncLoop Scheduling api-server Etcd bind pod, node list pod GenericRuntime SyncPod CRI grpc dockershim remote (no-op) Sandbox Create Delete List Container Create Start Exec Image Pull List shim client api dockerd runtime pod CRI Spec

Deployment • yum install -y kubelet kubeadm kubectl • sed -i '2 i\Environment="KUBELET_EXTRA_ARGS=--container- runtime=remote --container-runtime-endpoint=/var/run/xxx.sock -- feature-gates=AllAlpha=true"' /etc/systemd/system/kubelet.service.d/10- kubeadm.conf • kubeadm init • kubeadm join --token $token ${master_ip:port} • Done!

But that’s only one part … • Kubernetes is also about design pattern in container world • decoupling containers • re-use images • well-designed architecture for your container workloads • “How can I build distributed micro-services with container?”

Programming Pattern • Sidecar apiVersion: v1 kind: Pod metadata: name: test-app spec: containers: - name: app-container image: gcr.io/google_containers/testapp:v1 volumeMounts: - name: varlog mountPath: /var/log - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: {} - name: config-volume configMap: name: fluentd-config

Programming Pattern • InitContainer apiVersion: v1 kind: Pod metadata: name: init-demo spec: containers: - name: nginx image: nginx ports: - containerPort: 80 volumeMounts: - name: workdir mountPath: /usr/share/nginx/html # These containers are run during pod initialization initContainers: - name: install image: busybox command: - wget - "-O" - "/work-dir/index.html" - http://kubernetes.io volumeMounts: - name: workdir mountPath: "/work-dir" dnsPolicy: Default volumes: - name: workdir emptyDir: {}

Programming Pattern • Initializer apiVersion: apps/v1beta1 kind: Deployment metadata: annotations: "initializer.kubernetes.io/logging-agent": "true" name: helloworld-with-annotation spec: replicas: 1 template: metadata: name: helloworld-with-annotation spec: containers: - name: helloworld image: gcr.io/hightowerlabs/helloworld:0.0.1 apiVersion: v1 kind: ConfigMap metadata: name: logging-agent-initializer data: config: | - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: {} - name: config-volume configMap: name: fluentd-config Automatically Inject

Summary 1.How Kubernetes is using Golang? 1.What is Kubernetes and how it works (1 mins) 2.The heart of Kubernetes orchestration: Controller 3.Write your own Controller with CRD 4.code gen for deep copy, API conversion, API doc, encoding/decoding etc 5.gRPC based interface (e.g. CRI) 2.How we can do better to use Kubernetes? 1.Programming Patterns in Kubernetes 1.this is the main difference of Kubernetes with others 2.think about why everyone loves Borg

Come and Join, Gophers! ⼴广告时间

END Harry Zhang @resouer