Extending Kubernetes with Custom Resources and Operator Frameworks

01dc8e954957a10b428aa60b28c89d52?s=47 Ian Lewis
September 29, 2018

Extending Kubernetes with Custom Resources and Operator Frameworks

Kubernetes is a container orchestration system that provides a lot of built in functionality to deploy and manage applications using containers. But it’s applications that extend Kubernetes’ that is where the true power of Kubernetes lies. Supporting stateful applications, renewing certificates, managing distributed job pipelines can all be made easier by extending the Kubernetes API. Custom Resource Definitions (CRD) allow you to extend the API with new data types and Operators let you write automation logic based on your new data types.

Developing operators has traditionally been hard and new frameworks for writing operators like kubebuilder, operator-sdk, and Metacontroller, have made that job much easier, but it's hard to know which one to choose. In this session, attendees will learn about the Kubernetes API, it's architecture, and how to build operators. We will explore the key differences between the different operator frameworks so that attendees are empowered to choose the right framework for their use case.

01dc8e954957a10b428aa60b28c89d52?s=128

Ian Lewis

September 29, 2018
Tweet

Transcript

  1. Extending Kubernetes with Custom Resources and Operator Frameworks

  2. Ian Lewis • @IanMLewis • •  Tokyo, Japan •

    #kubernetes, #go, #python
  3. The Problem

  4. Problems • Dynamic, self-healing environment • Kubernetes provides building blocks,

    not complete solutions • New API and constructs
  5. Memcached

  6. Problems w/ Deploying Memcached • Memcached needs client side load

    balancing • Needs some kind of service discovery • Don't want to update application code • Want to support replication + sharding topologies
  7. Deploy memcached • Deploy a proxy using a Deployment •

    Configure proxy using a ConfigMap • When backends change create new ConfigMap and trigger a rolling-update for the proxy memcached memcached memcached client proxy
  8. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v1 memcached memcached memcached Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  9. Deploy memcached • Backend Pod endpoints change memcached memcached 10.0.2.17

    10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  10. Deploy memcached kind: ConfigMap apiVersion: v1 metadata: name: conf-v2 data:

    ... "10.0.2.17", "10.0.2.18" ... memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  11. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  12. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy proxy
  13. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  14. How do we support an application like memcached?

  15. github.com/ianlewis/memcached-operator

  16. A Quick Kubernetes API Primer

  17. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References
  18. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
  19. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
  20. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
  21. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References Deployment ReplicaSet ReplicaSet OwnerRef ReplicaSet OwnerRef OwnerRef
  22. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: v1 kind: ReplicaSet metadata: ... ownerReferences: - apiVersion: apps/v1 controller: true blockOwnerDeletion: true kind: Deployment name: nginx-deployment uid: d9607e19-f88f-11e6-a518-42010a800195 ...
  23. The Spoke and the Wheel

  24. Spoke & Wheel

  25. Spoke & Wheel API Server Client Client Client Client Client

    Client Client
  26. Controllers observe diff act

  27. Controllers watch diff update API Server

  28. Built-in Clients kube- apiserver kubelet kube- proxy kube- proxy kube-

    controller- manager kube- scheduler kubelet kubelet • kubelet • kube-proxy • kube-controller-manager • kube-scheduler
  29. Kubernetes Controllers

  30. controller-manager Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler

  31. Google Cloud Platform controller-manager Deployment metadata.name: nginx Deployment Controller ReplicaSet

    Controller Scheduler
  32. Google Cloud Platform controller-manager ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx

    Deployment Controller ReplicaSet Controller Scheduler
  33. Google Cloud Platform controller-manager ReplicaSet metadata.name: nginx-xxxx Deployment metadata.name: nginx

    Deployment Controller ReplicaSet Controller Scheduler
  34. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: <null> ReplicaSet

    metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
  35. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: <null> ReplicaSet

    metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
  36. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 ReplicaSet

    metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
  37. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: Pending

    node1 kubelet docker
  38. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating

    node1 kubelet docker
  39. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating

    node1 kubelet docker
  40. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating

    node1 kubelet docker Docker Hub / GCR
  41. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating

    node1 kubelet docker nginx-xxxx-x xxx
  42. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: ContainerCreating

    node1 kubelet docker nginx-xxxx-x xxx
  43. Google Cloud Platform Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 status: Running

    node1 kubelet docker nginx-xxxx-x xxx
  44. Extending Kubernetes

  45. Extending Kubernetes • Need a place to store state -

    Data • Need to do something - Logic
  46. Custom Resource Definition (Data) • Type definition for a custom

    type • Allows the same CRUD + WATCH • Can describe higher level constructs apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: foo.example.com spec: group: example.com version: v1 names: kind: Foo plural: foos scope: Namespaced
  47. Controllers (Logic) • Typically runs in the cluster • Uses

    the Kubernetes API • One idiomatic client ◦ client-go • Many generated client libraries ◦ Go ◦ Python ◦ Java
  48. client-go • Most featureful • Used by Kubernetes built-in components

    • More like a controller framework than a client library
  49. Operator Frameworks

  50. Operator Frameworks • Provide a simplified controller API • Rely

    on code generation to provide API clients for CRDs
  51. Frequent Requests • Controllers can potentially run often • Easy

    to overload the API server • Some GETS could be X00MB of data Object
  52. Concurrent Updates • Overwriting object state • The API Server

    isn't a database • No transactions Object
  53. operator-sdk • Built by former CoreOS developers at Red Hat

    • Quick and dirty • Provides support for one controller per process • Caching of watched objects w/ client-go • Serial updates per CRD object
  54. Architecture Patterns

  55. Reuse Built-in Objects • Services, Deployments, ConfigMaps, Secrets • Built

    on the logic of other controllers • Architect based on what you would do manually API
  56. Use Multiple Controllers • Multiple controllers per process • Reuse

    caches, informers etc. • Keep controllers simple • Each controller manages/writes to one object type • All message passing is done through the API API
  57. kubebuilder • Built by Kubernetes developers at Google • More

    robust • Helps manage lifecycle of generated code • Supports controllers for built-in objects • Supports multiple controllers in single operator binary
  58. Thank you! ianlewis@google.com IanMLewis@