Extending Kubernetes with Custom Resources and Operator Frameworks

Kubernetes is a container orchestration system that provides a lot of built in functionality to deploy and manage applications using containers. But it’s applications that extend Kubernetes’ that is where the true power of Kubernetes lies. Supporting stateful applications, renewing certificates, managing distributed job pipelines can all be made easier by extending the Kubernetes API. Custom Resource Definitions (CRD) allow you to extend the API with new data types and Operators let you write automation logic based on your new data types.

Developing operators has traditionally been hard and new frameworks for writing operators like kubebuilder, operator-sdk, and Metacontroller, have made that job much easier, but it's hard to know which one to choose. In this session, attendees will learn about the Kubernetes API, it's architecture, and how to build operators. We will explore the key differences between the different operator frameworks so that attendees are empowered to choose the right framework for their use case.

Transcript

1. Extending Kubernetes with
   Custom Resources and
   Operator Frameworks
   

   View Slide

2. Ian Lewis
   ● @IanMLewis
   ●
   ●  Tokyo, Japan
   ● #kubernetes, #go, #python
   

   View Slide

3. The Problem
   

   View Slide

4. Problems
   ● Dynamic, self-healing environment
   ● Kubernetes provides building blocks, not complete solutions
   ● New API and constructs
   

   View Slide

5. Memcached
   

   View Slide

6. Problems w/ Deploying Memcached
   ● Memcached needs client side load balancing
   ● Needs some kind of service discovery
   ● Don't want to update application code
   ● Want to support replication + sharding topologies
   

   View Slide

7. Deploy memcached
   ● Deploy a proxy using a
   Deployment
   ● Configure proxy using a
   ConfigMap
   ● When backends change
   create new ConfigMap and
   trigger a rolling-update for
   the proxy
   memcached
   memcached
   memcached
   client proxy
   

   View Slide

8. Deploy memcached
   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
   name: proxy
   spec:
   template:
   …
   spec:
   volumes:
   - name: proxy-conf
   configMap:
   name: conf-v1
   memcached
   memcached
   memcached
   Deployment
   ConfigMap
   10.0.2.17
   10.0.2.18
   10.0.2.19
   10.0.2.17
   10.0.2.18
   10.0.2.19
   proxy
   

   View Slide

9. Deploy memcached
   ● Backend Pod endpoints
   change
   memcached
   memcached
   10.0.2.17
   10.0.2.18
   Deployment
   ConfigMap
   10.0.2.17
   10.0.2.18
   10.0.2.19
   proxy
   

   View Slide

10. Deploy memcached
    kind: ConfigMap
    apiVersion: v1
    metadata:
    name: conf-v2
    data:
    ...
    "10.0.2.17",
    "10.0.2.18"
    ...
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy
    

    View Slide

11. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:
    …
    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v2
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy
    

    View Slide

12. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:
    …
    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v2
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy
    proxy
    

    View Slide

13. Deploy memcached
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: proxy
    spec:
    template:
    …
    spec:
    volumes:
    - name: proxy-conf
    configMap:
    name: conf-v2
    memcached
    memcached
    10.0.2.17
    10.0.2.18
    ConfigMap
    10.0.2.17
    10.0.2.18
    Deployment
    ConfigMap
    10.0.2.17
    10.0.2.18
    10.0.2.19
    proxy
    

    View Slide

14. How do we support an
    application like memcached?
    

    View Slide

15. github.com/ianlewis/memcached-operator
    

    View Slide

16. A Quick Kubernetes API Primer
    

    View Slide

17. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    

    View Slide

18. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    ...
    

    View Slide

19. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    ...
    

    View Slide

20. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx-deployment
    labels:
    app: nginx
    spec:
    ...
    

    View Slide

21. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    Deployment
    ReplicaSet ReplicaSet
    OwnerRef
    ReplicaSet
    OwnerRef
    OwnerRef
    

    View Slide

22. API Objects
    ● API Version
    ● Kind
    ● Metadata
    ○ Name
    ○ Labels
    ○ Owner References
    apiVersion: v1
    kind: ReplicaSet
    metadata:
    ...
    ownerReferences:
    - apiVersion: apps/v1
    controller: true
    blockOwnerDeletion: true
    kind: Deployment
    name: nginx-deployment
    uid: d9607e19-f88f-11e6-a518-42010a800195
    ...
    

    View Slide

23. The Spoke and the Wheel
    

    View Slide

24. Spoke & Wheel
    

    View Slide

25. Spoke & Wheel
    API
    Server
    Client Client
    Client
    Client
    Client
    Client
    Client
    

    View Slide

26. Controllers
    observe
    diff
    act
    

    View Slide

27. Controllers
    watch
    diff
    update
    API
    Server
    

    View Slide

28. Built-in Clients
    kube-
    apiserver
    kubelet kube-
    proxy
    kube-
    proxy
    kube-
    controller-
    manager
    kube-
    scheduler
    kubelet
    kubelet
    ● kubelet
    ● kube-proxy
    ● kube-controller-manager
    ● kube-scheduler
    

    View Slide

29. Kubernetes Controllers
    

    View Slide

30. controller-manager
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

31. Google Cloud Platform
    controller-manager
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

32. Google Cloud Platform
    controller-manager
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

33. Google Cloud Platform
    controller-manager
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

34. Google Cloud Platform
    controller-manager
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName:
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

35. Google Cloud Platform
    controller-manager
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName:
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

36. Google Cloud Platform
    controller-manager
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    ReplicaSet
    metadata.name: nginx-xxxx
    Deployment
    metadata.name: nginx
    Deployment
    Controller
    ReplicaSet
    Controller
    Scheduler
    

    View Slide

37. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: Pending
    node1
    kubelet docker
    

    View Slide

38. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    

    View Slide

39. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    

    View Slide

40. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    Docker Hub /
    GCR
    

    View Slide

41. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    nginx-xxxx-x
    xxx
    

    View Slide

42. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: ContainerCreating
    node1
    kubelet docker
    nginx-xxxx-x
    xxx
    

    View Slide

43. Google Cloud Platform
    Pod
    metadata.name: nginx-xxxx-xxxx
    spec.nodeName: node1
    status: Running
    node1
    kubelet docker
    nginx-xxxx-x
    xxx
    

    View Slide

44. Extending Kubernetes
    

    View Slide

45. Extending Kubernetes
    ● Need a place to store state - Data
    ● Need to do something - Logic
    

    View Slide

46. Custom Resource Definition (Data)
    ● Type definition for a custom
    type
    ● Allows the same CRUD +
    WATCH
    ● Can describe higher level
    constructs
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    name: foo.example.com
    spec:
    group: example.com
    version: v1
    names:
    kind: Foo
    plural: foos
    scope: Namespaced
    

    View Slide

47. Controllers (Logic)
    ● Typically runs in the cluster
    ● Uses the Kubernetes API
    ● One idiomatic client
    ○ client-go
    ● Many generated client libraries
    ○ Go
    ○ Python
    ○ Java
    

    View Slide

48. client-go
    ● Most featureful
    ● Used by Kubernetes built-in
    components
    ● More like a controller
    framework than a client library
    

    View Slide

49. Operator Frameworks
    

    View Slide

50. Operator Frameworks
    ● Provide a simplified controller
    API
    ● Rely on code generation to
    provide API clients for CRDs
    

    View Slide

51. Frequent Requests
    ● Controllers can potentially run
    often
    ● Easy to overload the API server
    ● Some GETS could be X00MB of
    data
    Object
    

    View Slide

52. Concurrent Updates
    ● Overwriting object state
    ● The API Server isn't a database
    ● No transactions
    Object
    

    View Slide

53. operator-sdk
    ● Built by former CoreOS
    developers at
    Red Hat
    ● Quick and dirty
    ● Provides support for one
    controller per process
    ● Caching of watched objects w/
    client-go
    ● Serial updates per CRD object
    

    View Slide

54. Architecture Patterns
    

    View Slide

55. Reuse Built-in Objects
    ● Services, Deployments,
    ConfigMaps, Secrets
    ● Built on the logic of other
    controllers
    ● Architect based on what you
    would do manually
    API
    

    View Slide

56. Use Multiple Controllers
    ● Multiple controllers per
    process
    ● Reuse caches, informers etc.
    ● Keep controllers simple
    ● Each controller
    manages/writes to one object
    type
    ● All message passing is done
    through the API
    API
    

    View Slide

57. kubebuilder
    ● Built by Kubernetes developers at Google
    ● More robust
    ● Helps manage lifecycle of generated code
    ● Supports controllers for built-in objects
    ● Supports multiple controllers in single operator binary
    

    View Slide

58. Thank you! ianlewis@google.com
    IanMLewis@
    

    View Slide