Introduction to Custom Resources in Kubernetes

This talk is an introduction to custom resources in Kubernetes. Custom resources are a way to extend the Kubernetes API to create objects of a kind. Custom resources are implemented with Custom Resource Definitions (CRDs) and, when coupled with a custom controller, they offer a way to implement your own declarative API.

An sample custom resource of type `HelmRelease` is introduced along with an example of a custom controller used to handle Helm releases.

Transcript

1. Introduction to
   Custom Resources
   Pauline Lallinec
   Workday
   Dublin Docker Meetup, March 2018
   

   View Slide

2. Software Engineer II - DevOps
   N -
   @plallin
   Workday Public Cloud Engineering
   We’re hiring!
   Best logo ever (or so I tell my director) ->
   

   View Slide

3. ● Core concepts: Kubernetes, Helm, CRDs
   ● Overview of the unicorn app
   ● What are CRDs?
   ● What are custom controllers?
   ● Demo of CRDs + Custom controller using unicorns
   ● Implementation of the Helm Release controller
   ● More demo using unicorns
   No Agenda No Attenda
   

   View Slide

4. Before we start: Kubernetes
   “Kubernetes (K8s) is an
   open-source system for
   automating deployment,
   scaling, and management of
   containerized applications.”
   Long story short: it deploys
   and manages your (Docker)
   containers for you.
   

   View Slide

5. Before we start: Helm
   ● Package manager for Kubernetes
   ● Helm chart: a set of Kubernetes
   resources
   ● Helm release: a version of a Helm
   Chart
   ● 2 parts: The Helm client (Helm), and
   the Helm server (Tiller)
   

   View Slide

6. Custom Resources
   A way to create custom objects that
   live within your cluster, and are handled
   by a custom controller running a logic
   of your own.
   (Ideally) CRDs responds to CRUD
   events (Create, Read, Update, Delete)
   and allow you to implement your own
   declarative API.
   

   View Slide

7. Custom Resources
   Example CRDs in Workday
   S3Bucket
   a CRD to create, delete, update and
   retrieve data from S3 Buckets
   HelmRelease
   a CRD to install, upgrade, delete, and
   store information on Helm releases
   

   View Slide

8. I
   A very simple app!
   The app
   One single HTML page showing a unicorn,
   serviced by Python’s SimpleHTTPServer
   Kubernetes resources
   One deployment, with only 1 container
   containing the Unicorn app
   3 Helm charts
   - Pink unicorn
   - Blue unicorn
   - Green unicorn
   

   View Slide

9. I : D
   

   View Slide

10. CRDs: why the interest?
    “In the future there should be
    nothing we [Kubernetes
    Developers] can do that you
    [Kubernetes users] can’t”
    Tim Hockin (@thockin)
    Kubernetes co-founder / pioneer &
    Co-chair of Kubernetes Network SIG
    “CRDs aren’t just for add-ons anymore - painting a picture for the
    future”, Lightning Talk, Kubecon North America 2018
    

    View Slide

11. Standalone CRDs
    ● Custom object with their own
    API endpoint
    ● Store / retrieve structured data
    CRDs + Custom controllers
    ● Declarative API
    What are CRDs?
    

    View Slide

12. What are CRDs?
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    name: helmreleases.samplecontroller.k8s.io
    spec:
    group: samplecontroller.k8s.io
    version: v1alpha1
    names:
    kind: HelmRelease
    plural: helmreleases
    scope: Namespaced
    Custom object with their own API endpoint
    

    View Slide

13. $ kubectl create -f helm_release_crd.yaml
    customresourcedefinition.apiextensions.k8s.io/helmreleases.sample
    controller.k8s.io created
    $ kubectl get crd
    NAME CREATED AT
    helmreleases.samplecontroller.k8s.io 2019-03-23T05:21:43Z
    Custom object with their own API endpoint
    What are CRDs?
    

    View Slide

14. apiVersion: samplecontroller.k8s.io/v1alpha1
    kind: HelmRelease
    metadata:
    name: unicorn-release
    spec:
    releaseVersion: pink
    releaseName: unicorn
    Store / retrieve structured data
    What are CRDs?
    

    View Slide

15. $ kubectl create -f unicorn-release-pink.yaml
    helmrelease.samplecontroller.k8s.io/unicorn-release created
    $ kubectl get helmreleases
    NAME AGE
    unicorn-release 36s
    Store / retrieve structured data
    What are CRDs?
    

    View Slide

16. $ kubectl describe helmrelease unicorn-release
    Name: unicorn-release
    Namespace: default
    API Version: samplecontroller.k8s.io/v1alpha1
    Kind: HelmRelease
    Metadata:
    ...
    Spec:
    Release Name: unicorn
    Release Version: pink
    Events:
    Store / retrieve structured data
    What are CRDs?
    

    View Slide

17. ● Listen to any resource type
    ● Ensure desired state = existing state
    ● If desired state ≠ existing state, will
    take action to make existing state =
    desired state
    ● This is implemented using your own
    logic!
    Clone kubernetes/sample-controller from GitHub for an example of a
    sample controller
    Custom controllers
    

    View Slide

18. Helm Release Controller
    ● Listen to CRDs of type HelmRelease
    ● Ensures all desired Helm releases are
    installed / upgraded
    ● Will install / upgrade the Helm release if
    not already installed / not upgraded to
    desired version
    Custom controllers
    

    View Slide

19. (No) Helm Release Controller
    ● Current situation: run a script to
    handle installation / upgrade /
    rollback of Helm resources
    ● Need a way (i.e. Jenkins job) to run
    that script to install / upgrade /
    rollback Helm charts
    ● Imperative rather than declarative
    Custom controllers
    

    View Slide

20. Helm Release Controller
    ● Cluster logic remains within the
    cluster
    ● Declarative API: let the cluster
    manage itself
    ● No need for additional script /
    Jenkins job
    Custom controllers
    

    View Slide

21. Helm Release Controller
    ● Automated rollback according to a
    logic of our own
    ● Allow for custom business logic
    ● No need to install / maintain the
    Helm Client on different servers
    Custom controllers
    

    View Slide

22. U CRD’
    

    View Slide

23. (Unicorn) HelmRelease
    New Helm Release?
    Y: Install Helm Release
    N: Upgrade Helm Release
    U CRD’
    

    View Slide

24. U CRD’ :
    

    View Slide

25. Clone of the existing Sample
    Controller from Kubernetes
    No update done to listeners, informers,
    event handlers, etc.
    Focus on SyncHandlers() which is
    responsible for ensuring that
    desired state = existing state
    kubernetes/sample-controller: https://github.com/kubernetes/sample-controller
    Helm Release Controller: the implementation
    

    View Slide

26. Receive a CRD of type Helm Release
    Helm Release Controller: the implementation
    

    View Slide

27. Get information on CRD of type HelmRelease
    Helm Release Controller: the implementation
    

    View Slide

28. Get information about the current CRD (= desired state)
    Helm Release Controller: the implementation
    

    View Slide

29. Install Helm release if it doesn’t already exist (= match desired state)
    Helm Release Controller: the implementation
    

    View Slide

30. Check existing deployment (= existing state)
    Helm Release Controller: the implementation
    

    View Slide

31. Upgrade existing release if necessary (= match desired state)
    Helm Release Controller: the implementation
    

    View Slide

32. Update Helm Release status
    Helm Release Controller: the implementation
    

    View Slide

33. Update Helm Release status
    Helm Release Controller: the implementation
    

    View Slide

34. If an error happens, re-enqueue the event and retry later
    Helm Release Controller: the implementation
    

    View Slide

35. Finally, return successful sync event
    Helm Release Controller: the implementation
    

    View Slide

36. ● Choice of programming language
    ● Can enforce validation (example to
    follow)
    ● Can support /status and /scale
    subresources (and maybe /exec and
    /log in the future)
    CRDs + Custom controllers: Other benefits
    

    View Slide

37. apiVersion:
    apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    ...
    spec:
    ...
    validation:
    openAPIV3Schema:
    properties:
    spec:
    properties:
    releaseVersion:
    type: string
    pattern: "(pink|blue)"
    E
    

    View Slide

38. HelmRelease
    Y: New Helm Release?
    Y: Install Helm Release
    N: Upgrade Helm Release
    Is this unicorn release pink
    or blue?
    N: reject create / update
    event
    E
    

    View Slide

39. E :
    

    View Slide

40. Thank you!
    Join my team! Apply to “[Senior]
    Devops Engineer - DevOps in Cloud
    Orchestration” on Workday (Link:
    https://lnkd.in/gkMHwMC), or directly
    introduce yourself to Dave Doran
    (he’s here tonight!). You will be able
    to work with trendy technologies
    AND join the karaoke club. How cool
    is that?
    Follow me on Twitter at @plallin to
    hear me brag about how much I walk
    (a lot!), how much I lift (really not a
    lot), and laugh about my struggles
    with Arch Linux.
    

    View Slide

41. Pictures!
    All pictures downloaded from Pixabay https://pixabay.com/
    Ship of containers: https://pixabay.com/photos/hamburg-port-of-hamburg-3021820/
    Ending ship of containers: https://pixabay.com/photos/container-container-ship-port-1611490/
    Unicorns: https://pixabay.com/vectors/unicorn-unicorn-crown-flower-crown-3392560/
    Picture of a Helm on a blue background: https://pixabay.com/photos/steering-wheel-blue-twist-sea-981439/
    Agenda ship sailing in the sun: https://pixabay.com/photos/sea-sailing-vessel-boat-ship-701079/
    

    View Slide