$30 off During Our Annual Pro Sale. View Details »

Extending Kubernetes with Custom Resources and Operator Frameworks

Ian Lewis
September 29, 2018

Extending Kubernetes with Custom Resources and Operator Frameworks

Kubernetes is a container orchestration system that provides a lot of built in functionality to deploy and manage applications using containers. But it’s applications that extend Kubernetes’ that is where the true power of Kubernetes lies. Supporting stateful applications, renewing certificates, managing distributed job pipelines can all be made easier by extending the Kubernetes API. Custom Resource Definitions (CRD) allow you to extend the API with new data types and Operators let you write automation logic based on your new data types.

Developing operators has traditionally been hard and new frameworks for writing operators like kubebuilder, operator-sdk, and Metacontroller, have made that job much easier, but it's hard to know which one to choose. In this session, attendees will learn about the Kubernetes API, it's architecture, and how to build operators. We will explore the key differences between the different operator frameworks so that attendees are empowered to choose the right framework for their use case.

Ian Lewis

September 29, 2018
Tweet

More Decks by Ian Lewis

Other Decks in Technology

Transcript

  1. Extending Kubernetes with
    Custom Resources and
    Operator Frameworks

    View Slide

  2. Ian Lewis
    ● @IanMLewis

    ●  Tokyo, Japan
    ● #kubernetes, #go, #python

    View Slide

  3. The Problem

    View Slide

  4. Problems
    ● Dynamic, self-healing environment
    ● Kubernetes provides building blocks, not complete solutions
    ● New API and constructs

    View Slide

  5. Memcached

    View Slide

  6. Problems w/ Deploying Memcached
    ● Memcached needs client side load balancing
    ● Needs some kind of service discovery
    ● Don't want to update application code
    ● Want to support replication + sharding topologies

    View Slide

  7. Deploy memcached
    ● Deploy a proxy using a
    Deployment
    ● Configure proxy using a
    ConfigMap
    ● When backends change
    create new ConfigMap and
    trigger a rolling-update for
    the proxy
    memcached
    memcached
    memcached
    client proxy

    View Slide

  8. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:

    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v1
    memcached
    memcached
    memcached
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy

    View Slide

  9. Deploy memcached
    ● Backend Pod endpoints
    change
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy

    View Slide

  10. Deploy memcached
    kind: ConfigMap
    apiVersion: v1
    metadata:
    name: conf-v2
    data:
    ...
    "10.0.2.17",
    "10.0.2.18"
    ...
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy

    View Slide

  11. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:

    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v2
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy

    View Slide

  12. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:

    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v2
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy
    proxy

    View Slide

  13. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:

    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v2
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy

    View Slide

  14. How do we support an
    application like memcached?

    View Slide

  15. github.com/ianlewis/memcached-operator

    View Slide

  16. A Quick Kubernetes API Primer

    View Slide

  17. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References

    View Slide

  18. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    ...

    View Slide

  19. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    ...

    View Slide

  20. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    ...

    View Slide

  21. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    Deployment
    ReplicaSet ReplicaSet
    OwnerRef
    ReplicaSet
    OwnerRef
    OwnerRef

    View Slide

  22. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: v1
    kind: ReplicaSet
    metadata:
    ...
    ownerReferences:
    - apiVersion: apps/v1
    controller: true
    blockOwnerDeletion: true
    kind: Deployment
    name: nginx-deployment
    uid: d9607e19-f88f-11e6-a518-42010a800195
    ...

    View Slide

  23. The Spoke and the Wheel

    View Slide

  24. Spoke & Wheel

    View Slide

  25. Spoke & Wheel
    API
    Server
    Client Client
    Client
    Client
    Client
    Client
    Client

    View Slide

  26. Controllers
    observe
    diff
    act

    View Slide

  27. Controllers
    watch
    diff
    update
    API
    Server

    View Slide

  28. Built-in Clients
    kube-
    apiserver
    kubelet kube-
    proxy
    kube-
    proxy
    kube-
    controller-
    manager
    kube-
    scheduler
    kubelet
    kubelet
    ● kubelet
    ● kube-proxy
    ● kube-controller-manager
    ● kube-scheduler

    View Slide

  29. Kubernetes Controllers

    View Slide

  30. controller-manager
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  31. Google Cloud Platform
    controller-manager
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  32. Google Cloud Platform
    controller-manager
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  33. Google Cloud Platform
    controller-manager
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  34. Google Cloud Platform
    controller-manager
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName:
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  35. Google Cloud Platform
    controller-manager
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName:
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  36. Google Cloud Platform
    controller-manager
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler

    View Slide

  37. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: Pending
    node1
    kubelet docker

    View Slide

  38. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker

    View Slide

  39. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker

    View Slide

  40. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    Docker Hub /
    GCR

    View Slide

  41. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    nginx-xxxx-x
    xxx

    View Slide

  42. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    nginx-xxxx-x
    xxx

    View Slide

  43. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: Running
    node1
    kubelet docker
    nginx-xxxx-x
    xxx

    View Slide

  44. Extending Kubernetes

    View Slide

  45. Extending Kubernetes
    ● Need a place to store state - Data
    ● Need to do something - Logic

    View Slide

  46. Custom Resource Definition (Data)
    ● Type definition for a custom
    type
    ● Allows the same CRUD +
    WATCH
    ● Can describe higher level
    constructs
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    name: foo.example.com
    spec:
    group: example.com
    version: v1
    names:
    kind: Foo
    plural: foos
    scope: Namespaced

    View Slide

  47. Controllers (Logic)
    ● Typically runs in the cluster
    ● Uses the Kubernetes API
    ● One idiomatic client
    ○ client-go
    ● Many generated client libraries
    ○ Go
    ○ Python
    ○ Java

    View Slide

  48. client-go
    ● Most featureful
    ● Used by Kubernetes built-in
    components
    ● More like a controller
    framework than a client library

    View Slide

  49. Operator Frameworks

    View Slide

  50. Operator Frameworks
    ● Provide a simplified controller
    API
    ● Rely on code generation to
    provide API clients for CRDs

    View Slide

  51. Frequent Requests
    ● Controllers can potentially run
    often
    ● Easy to overload the API server
    ● Some GETS could be X00MB of
    data
    Object

    View Slide

  52. Concurrent Updates
    ● Overwriting object state
    ● The API Server isn't a database
    ● No transactions
    Object

    View Slide

  53. operator-sdk
    ● Built by former CoreOS
    developers at
    Red Hat
    ● Quick and dirty
    ● Provides support for one
    controller per process
    ● Caching of watched objects w/
    client-go
    ● Serial updates per CRD object

    View Slide

  54. Architecture Patterns

    View Slide

  55. Reuse Built-in Objects
    ● Services, Deployments,
    ConfigMaps, Secrets
    ● Built on the logic of other
    controllers
    ● Architect based on what you
    would do manually
    API

    View Slide

  56. Use Multiple Controllers
    ● Multiple controllers per
    process
    ● Reuse caches, informers etc.
    ● Keep controllers simple
    ● Each controller
    manages/writes to one object
    type
    ● All message passing is done
    through the API
    API

    View Slide

  57. kubebuilder
    ● Built by Kubernetes developers at Google
    ● More robust
    ● Helps manage lifecycle of generated code
    ● Supports controllers for built-in objects
    ● Supports multiple controllers in single operator binary

    View Slide

  58. Thank you! [email protected]
    IanMLewis@

    View Slide