Extending Kubernetes with Custom Resources and Operator Frameworks

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Ian Lewis ● @IanMLewis ● ●  Tokyo, Japan ● #kubernetes, #go, #python

Slide 3

Slide 3 text

The Problem

Slide 4

Slide 4 text

Problems ● Dynamic, self-healing environment ● Kubernetes provides building blocks, not complete solutions ● New API and constructs

Slide 5

Slide 5 text

Memcached

Slide 6

Slide 6 text

Problems w/ Deploying Memcached ● Memcached needs client side load balancing ● Needs some kind of service discovery ● Don't want to update application code ● Want to support replication + sharding topologies

Slide 7

Slide 7 text

Deploy memcached ● Deploy a proxy using a Deployment ● Configure proxy using a ConfigMap ● When backends change create new ConfigMap and trigger a rolling-update for the proxy memcached memcached memcached client proxy

Slide 8

Slide 8 text

Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec: template: … spec: volumes: - name: proxy-conf configMap: name: conf-v1 memcached memcached memcached Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 10.0.2.17 10.0.2.18 10.0.2.19 proxy

Slide 9

Slide 9 text

Deploy memcached ● Backend Pod endpoints change memcached memcached 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy

Slide 10

Slide 10 text

Deploy memcached kind: ConfigMap apiVersion: v1 metadata: name: conf-v2 data: ... "10.0.2.17", "10.0.2.18" ... memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy

Slide 11

Slide 11 text

Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec: template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

How do we support an application like memcached?

Slide 15

Slide 15 text

github.com/ianlewis/memcached-operator

Slide 16

Slide 16 text

A Quick Kubernetes API Primer

Slide 17

Slide 17 text

API Objects ● API Version ● Kind ● Metadata ○ Name ○ Labels ○ Owner References

Slide 18

Slide 18 text

API Objects ● API Version ● Kind ● Metadata ○ Name ○ Labels ○ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...

Slide 19

Slide 19 text

API Objects ● API Version ● Kind ● Metadata ○ Name ○ Labels ○ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...

Slide 20

Slide 20 text

API Objects ● API Version ● Kind ● Metadata ○ Name ○ Labels ○ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...

Slide 21

Slide 21 text

API Objects ● API Version ● Kind ● Metadata ○ Name ○ Labels ○ Owner References Deployment ReplicaSet ReplicaSet OwnerRef ReplicaSet OwnerRef OwnerRef

Slide 22

Slide 22 text

API Objects ● API Version ● Kind ● Metadata ○ Name ○ Labels ○ Owner References apiVersion: v1 kind: ReplicaSet metadata: ... ownerReferences: - apiVersion: apps/v1 controller: true blockOwnerDeletion: true kind: Deployment name: nginx-deployment uid: d9607e19-f88f-11e6-a518-42010a800195 ...

Slide 23

Slide 23 text

The Spoke and the Wheel

Slide 24

Slide 24 text

Spoke & Wheel

Slide 25

Slide 25 text

Spoke & Wheel API Server Client Client Client Client Client Client Client

Slide 26

Slide 26 text

Controllers observe diff act

Slide 27

Slide 27 text

Controllers watch diff update API Server

Slide 28

Slide 28 text

Built-in Clients kube- apiserver kubelet kube- proxy kube- proxy kube- controller- manager kube- scheduler kubelet kubelet ● kubelet ● kube-proxy ● kube-controller-manager ● kube-scheduler

Slide 29

Slide 29 text

Kubernetes Controllers

Slide 30

Slide 30 text

controller-manager Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

Slide 31

Slide 31 text

Google Cloud Platform controller-manager Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

Slide 32

Slide 32 text

Google Cloud Platform controller-manager ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

Slide 33

Slide 33 text

Google Cloud Platform controller-manager ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

Slide 34

Slide 34 text

Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

Slide 37

Slide 37 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: Pending node1 kubelet docker

Slide 38

Slide 38 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker

Slide 39

Slide 39 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker

Slide 40

Slide 40 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker Docker Hub / GCR

Slide 41

Slide 41 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker nginx-xxxx-x xxx

Slide 42

Slide 42 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker nginx-xxxx-x xxx

Slide 43

Slide 43 text

Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: Running node1 kubelet docker nginx-xxxx-x xxx

Slide 44

Slide 44 text

Extending Kubernetes

Slide 45

Slide 45 text

Extending Kubernetes ● Need a place to store state - Data ● Need to do something - Logic

Slide 46

Slide 46 text

Custom Resource Definition (Data) ● Type definition for a custom type ● Allows the same CRUD + WATCH ● Can describe higher level constructs apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: foo.example.com spec: group: example.com version: v1 names: kind: Foo plural: foos scope: Namespaced

Slide 47

Slide 47 text

Controllers (Logic) ● Typically runs in the cluster ● Uses the Kubernetes API ● One idiomatic client ○ client-go ● Many generated client libraries ○ Go ○ Python ○ Java

Slide 48

Slide 48 text

client-go ● Most featureful ● Used by Kubernetes built-in components ● More like a controller framework than a client library

Slide 49

Slide 49 text

Operator Frameworks

Slide 50

Slide 50 text

Operator Frameworks ● Provide a simplified controller API ● Rely on code generation to provide API clients for CRDs

Slide 51

Slide 51 text

Frequent Requests ● Controllers can potentially run often ● Easy to overload the API server ● Some GETS could be X00MB of data Object

Slide 52

Slide 52 text

Concurrent Updates ● Overwriting object state ● The API Server isn't a database ● No transactions Object

Slide 53

Slide 53 text

operator-sdk ● Built by former CoreOS developers at Red Hat ● Quick and dirty ● Provides support for one controller per process ● Caching of watched objects w/ client-go ● Serial updates per CRD object

Slide 54

Slide 54 text

Architecture Patterns

Slide 55

Slide 55 text

Reuse Built-in Objects ● Services, Deployments, ConfigMaps, Secrets ● Built on the logic of other controllers ● Architect based on what you would do manually API

Slide 56

Slide 56 text

Use Multiple Controllers ● Multiple controllers per process ● Reuse caches, informers etc. ● Keep controllers simple ● Each controller manages/writes to one object type ● All message passing is done through the API API

Slide 57

Slide 57 text

kubebuilder ● Built by Kubernetes developers at Google ● More robust ● Helps manage lifecycle of generated code ● Supports controllers for built-in objects ● Supports multiple controllers in single operator binary

Slide 58

Slide 58 text

Thank you! ianlewis@google.com IanMLewis@