Extending Kubernetes with Custom Resources and Operator Frameworks

Kubernetes is a container orchestration system that provides a lot of built in functionality to deploy and manage applications using containers. But it’s applications that extend Kubernetes’ that is where the true power of Kubernetes lies. Supporting stateful applications, renewing certificates, managing distributed job pipelines can all be made easier by extending the Kubernetes API. Custom Resource Definitions (CRD) allow you to extend the API with new data types and Operators let you write automation logic based on your new data types.

Developing operators has traditionally been hard and new frameworks for writing operators like kubebuilder, operator-sdk, and Metacontroller, have made that job much easier, but it's hard to know which one to choose. In this session, attendees will learn about the Kubernetes API, it's architecture, and how to build operators. We will explore the key differences between the different operator frameworks so that attendees are empowered to choose the right framework for their use case.

Transcript

1. Extending Kubernetes with Custom Resources and Operator Frameworks
2. Ian Lewis • @IanMLewis • •
    Tokyo, Japan • #kubernetes, #go, #python
3. The Problem
4. Problems • Dynamic, self-healing environment • Kubernetes provides building blocks, not complete solutions • New API and constructs
5. Memcached
6. Problems w/ Deploying Memcached • Memcached needs client side load balancing • Needs some kind of service discovery • Don't want to update application code • Want to support replication + sharding topologies
7. Deploy memcached • Deploy a proxy using a Deployment • Configure proxy using a ConfigMap • When backends change create new ConfigMap and trigger a rolling-update for the proxy memcached memcached memcached client proxy
8. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec: template: … spec: volumes: - name: proxy-conf configMap: name: conf-v1 memcached memcached memcached Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 10.0.2.17 10.0.2.18 10.0.2.19 proxy
9. Deploy memcached • Backend Pod endpoints change memcached memcached 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
10. Deploy memcached kind: ConfigMap apiVersion: v1 metadata: name: conf-v2 data: ... "10.0.2.17", "10.0.2.18" ... memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
11. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec: template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
12. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec: template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy proxy
13. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec: template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
14. How do we support an application like memcached?
15. github.com/ianlewis/memcached-operator
16. A Quick Kubernetes API Primer
17. API Objects • API Version • Kind • Metadata ◦ Name ◦ Labels ◦ Owner References
18. API Objects • API Version • Kind • Metadata ◦ Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
19. API Objects • API Version • Kind • Metadata ◦ Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
20. API Objects • API Version • Kind • Metadata ◦ Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
21. API Objects • API Version • Kind • Metadata ◦ Name ◦ Labels ◦ Owner References Deployment ReplicaSet ReplicaSet OwnerRef ReplicaSet OwnerRef OwnerRef
22. API Objects • API Version • Kind • Metadata ◦ Name ◦ Labels ◦ Owner References apiVersion: v1 kind: ReplicaSet metadata: ... ownerReferences: - apiVersion: apps/v1 controller: true blockOwnerDeletion: true kind: Deployment name: nginx-deployment uid: d9607e19-f88f-11e6-a518-42010a800195 ...
23. The Spoke and the Wheel
24. Spoke & Wheel
25. Spoke & Wheel API Server Client Client Client Client Client Client Client
26. Controllers observe diff act
27. Controllers watch diff update API Server
28. Built-in Clients kube- apiserver kubelet kube- proxy kube- proxy kube- controller- manager kube- scheduler kubelet kubelet • kubelet • kube-proxy • kube-controller-manager • kube-scheduler
29. Kubernetes Controllers
30. controller-manager Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
31. Google Cloud Platform controller-manager Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
32. Google Cloud Platform controller-manager ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
33. Google Cloud Platform controller-manager ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
34. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: <null> ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
35. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: <null> ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
36. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
37. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: Pending node1 kubelet docker
38. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker
39. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker
40. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker Docker Hub / GCR
41. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker nginx-xxxx-x xxx
42. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating node1 kubelet docker nginx-xxxx-x xxx
43. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: Running node1 kubelet docker nginx-xxxx-x xxx
44. Extending Kubernetes
45. Extending Kubernetes • Need a place to store state - Data • Need to do something - Logic
46. Custom Resource Definition (Data) • Type definition for a custom type • Allows the same CRUD + WATCH • Can describe higher level constructs apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: foo.example.com spec: group: example.com version: v1 names: kind: Foo plural: foos scope: Namespaced
47. Controllers (Logic) • Typically runs in the cluster • Uses the Kubernetes API • One idiomatic client ◦ client-go • Many generated client libraries ◦ Go ◦ Python ◦ Java
48. client-go • Most featureful • Used by Kubernetes built-in components • More like a controller framework than a client library
49. Operator Frameworks
50. Operator Frameworks • Provide a simplified controller API • Rely on code generation to provide API clients for CRDs
51. Frequent Requests • Controllers can potentially run often • Easy to overload the API server • Some GETS could be X00MB of data Object
52. Concurrent Updates • Overwriting object state • The API Server isn't a database • No transactions Object
53. operator-sdk • Built by former CoreOS developers at Red Hat • Quick and dirty • Provides support for one controller per process • Caching of watched objects w/ client-go • Serial updates per CRD object
54. Architecture Patterns
55. Reuse Built-in Objects • Services, Deployments, ConfigMaps, Secrets • Built on the logic of other controllers • Architect based on what you would do manually API
56. Use Multiple Controllers • Multiple controllers per process • Reuse caches, informers etc. • Keep controllers simple • Each controller manages/writes to one object type • All message passing is done through the API API
57. kubebuilder • Built by Kubernetes developers at Google • More robust • Helps manage lifecycle of generated code • Supports controllers for built-in objects • Supports multiple controllers in single operator binary
58. Thank you! ianlewis@google.com IanMLewis@