Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Custom Resources in Kubernetes

Introduction to Custom Resources in Kubernetes

This talk is an introduction to custom resources in Kubernetes. Custom resources are a way to extend the Kubernetes API to create objects of a kind. Custom resources are implemented with Custom Resource Definitions (CRDs) and, when coupled with a custom controller, they offer a way to implement your own declarative API.

An sample custom resource of type `HelmRelease` is introduced along with an example of a custom controller used to handle Helm releases.

Pauline Lallinec

March 28, 2019
Tweet

More Decks by Pauline Lallinec

Other Decks in Technology

Transcript

  1. Introduction to
    Custom Resources
    Pauline Lallinec
    Workday
    Dublin Docker Meetup, March 2018

    View Slide

  2. Software Engineer II - DevOps
    N -
    @plallin
    Workday Public Cloud Engineering
    We’re hiring!
    Best logo ever (or so I tell my director) ->

    View Slide

  3. ● Core concepts: Kubernetes, Helm, CRDs
    ● Overview of the unicorn app
    ● What are CRDs?
    ● What are custom controllers?
    ● Demo of CRDs + Custom controller using unicorns
    ● Implementation of the Helm Release controller
    ● More demo using unicorns
    No Agenda No Attenda

    View Slide

  4. Before we start: Kubernetes
    “Kubernetes (K8s) is an
    open-source system for
    automating deployment,
    scaling, and management of
    containerized applications.”
    Long story short: it deploys
    and manages your (Docker)
    containers for you.

    View Slide

  5. Before we start: Helm
    ● Package manager for Kubernetes
    ● Helm chart: a set of Kubernetes
    resources
    ● Helm release: a version of a Helm
    Chart
    ● 2 parts: The Helm client (Helm), and
    the Helm server (Tiller)

    View Slide

  6. Custom Resources
    A way to create custom objects that
    live within your cluster, and are handled
    by a custom controller running a logic
    of your own.
    (Ideally) CRDs responds to CRUD
    events (Create, Read, Update, Delete)
    and allow you to implement your own
    declarative API.

    View Slide

  7. Custom Resources
    Example CRDs in Workday
    S3Bucket
    a CRD to create, delete, update and
    retrieve data from S3 Buckets
    HelmRelease
    a CRD to install, upgrade, delete, and
    store information on Helm releases

    View Slide

  8. I
    A very simple app!
    The app
    One single HTML page showing a unicorn,
    serviced by Python’s SimpleHTTPServer
    Kubernetes resources
    One deployment, with only 1 container
    containing the Unicorn app
    3 Helm charts
    - Pink unicorn
    - Blue unicorn
    - Green unicorn

    View Slide

  9. I : D

    View Slide

  10. CRDs: why the interest?
    “In the future there should be
    nothing we [Kubernetes
    Developers] can do that you
    [Kubernetes users] can’t”
    Tim Hockin (@thockin)
    Kubernetes co-founder / pioneer &
    Co-chair of Kubernetes Network SIG
    “CRDs aren’t just for add-ons anymore - painting a picture for the
    future”, Lightning Talk, Kubecon North America 2018

    View Slide

  11. Standalone CRDs
    ● Custom object with their own
    API endpoint
    ● Store / retrieve structured data
    CRDs + Custom controllers
    ● Declarative API
    What are CRDs?

    View Slide

  12. What are CRDs?
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    name: helmreleases.samplecontroller.k8s.io
    spec:
    group: samplecontroller.k8s.io
    version: v1alpha1
    names:
    kind: HelmRelease
    plural: helmreleases
    scope: Namespaced
    Custom object with their own API endpoint

    View Slide

  13. $ kubectl create -f helm_release_crd.yaml
    customresourcedefinition.apiextensions.k8s.io/helmreleases.sample
    controller.k8s.io created
    $ kubectl get crd
    NAME CREATED AT
    helmreleases.samplecontroller.k8s.io 2019-03-23T05:21:43Z
    Custom object with their own API endpoint
    What are CRDs?

    View Slide

  14. apiVersion: samplecontroller.k8s.io/v1alpha1
    kind: HelmRelease
    metadata:
    name: unicorn-release
    spec:
    releaseVersion: pink
    releaseName: unicorn
    Store / retrieve structured data
    What are CRDs?

    View Slide

  15. $ kubectl create -f unicorn-release-pink.yaml
    helmrelease.samplecontroller.k8s.io/unicorn-release created
    $ kubectl get helmreleases
    NAME AGE
    unicorn-release 36s
    Store / retrieve structured data
    What are CRDs?

    View Slide

  16. $ kubectl describe helmrelease unicorn-release
    Name: unicorn-release
    Namespace: default
    API Version: samplecontroller.k8s.io/v1alpha1
    Kind: HelmRelease
    Metadata:
    ...
    Spec:
    Release Name: unicorn
    Release Version: pink
    Events:
    Store / retrieve structured data
    What are CRDs?

    View Slide

  17. ● Listen to any resource type
    ● Ensure desired state = existing state
    ● If desired state ≠ existing state, will
    take action to make existing state =
    desired state
    ● This is implemented using your own
    logic!
    Clone kubernetes/sample-controller from GitHub for an example of a
    sample controller
    Custom controllers

    View Slide

  18. Helm Release Controller
    ● Listen to CRDs of type HelmRelease
    ● Ensures all desired Helm releases are
    installed / upgraded
    ● Will install / upgrade the Helm release if
    not already installed / not upgraded to
    desired version
    Custom controllers

    View Slide

  19. (No) Helm Release Controller
    ● Current situation: run a script to
    handle installation / upgrade /
    rollback of Helm resources
    ● Need a way (i.e. Jenkins job) to run
    that script to install / upgrade /
    rollback Helm charts
    ● Imperative rather than declarative
    Custom controllers

    View Slide

  20. Helm Release Controller
    ● Cluster logic remains within the
    cluster
    ● Declarative API: let the cluster
    manage itself
    ● No need for additional script /
    Jenkins job
    Custom controllers

    View Slide

  21. Helm Release Controller
    ● Automated rollback according to a
    logic of our own
    ● Allow for custom business logic
    ● No need to install / maintain the
    Helm Client on different servers
    Custom controllers

    View Slide

  22. U CRD’

    View Slide

  23. (Unicorn) HelmRelease
    New Helm Release?
    Y: Install Helm Release
    N: Upgrade Helm Release
    U CRD’

    View Slide

  24. U CRD’ :

    View Slide

  25. Clone of the existing Sample
    Controller from Kubernetes
    No update done to listeners, informers,
    event handlers, etc.
    Focus on SyncHandlers() which is
    responsible for ensuring that
    desired state = existing state
    kubernetes/sample-controller: https://github.com/kubernetes/sample-controller
    Helm Release Controller: the implementation

    View Slide

  26. Receive a CRD of type Helm Release
    Helm Release Controller: the implementation

    View Slide

  27. Get information on CRD of type HelmRelease
    Helm Release Controller: the implementation

    View Slide

  28. Get information about the current CRD (= desired state)
    Helm Release Controller: the implementation

    View Slide

  29. Install Helm release if it doesn’t already exist (= match desired state)
    Helm Release Controller: the implementation

    View Slide

  30. Check existing deployment (= existing state)
    Helm Release Controller: the implementation

    View Slide

  31. Upgrade existing release if necessary (= match desired state)
    Helm Release Controller: the implementation

    View Slide

  32. Update Helm Release status
    Helm Release Controller: the implementation

    View Slide

  33. Update Helm Release status
    Helm Release Controller: the implementation

    View Slide

  34. If an error happens, re-enqueue the event and retry later
    Helm Release Controller: the implementation

    View Slide

  35. Finally, return successful sync event
    Helm Release Controller: the implementation

    View Slide

  36. ● Choice of programming language
    ● Can enforce validation (example to
    follow)
    ● Can support /status and /scale
    subresources (and maybe /exec and
    /log in the future)
    CRDs + Custom controllers: Other benefits

    View Slide

  37. apiVersion:
    apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    ...
    spec:
    ...
    validation:
    openAPIV3Schema:
    properties:
    spec:
    properties:
    releaseVersion:
    type: string
    pattern: "(pink|blue)"
    E

    View Slide

  38. HelmRelease
    Y: New Helm Release?
    Y: Install Helm Release
    N: Upgrade Helm Release
    Is this unicorn release pink
    or blue?
    N: reject create / update
    event
    E

    View Slide

  39. E :

    View Slide

  40. Thank you!
    Join my team! Apply to “[Senior]
    Devops Engineer - DevOps in Cloud
    Orchestration” on Workday (Link:
    https://lnkd.in/gkMHwMC), or directly
    introduce yourself to Dave Doran
    (he’s here tonight!). You will be able
    to work with trendy technologies
    AND join the karaoke club. How cool
    is that?
    Follow me on Twitter at @plallin to
    hear me brag about how much I walk
    (a lot!), how much I lift (really not a
    lot), and laugh about my struggles
    with Arch Linux.

    View Slide

  41. Pictures!
    All pictures downloaded from Pixabay https://pixabay.com/
    Ship of containers: https://pixabay.com/photos/hamburg-port-of-hamburg-3021820/
    Ending ship of containers: https://pixabay.com/photos/container-container-ship-port-1611490/
    Unicorns: https://pixabay.com/vectors/unicorn-unicorn-crown-flower-crown-3392560/
    Picture of a Helm on a blue background: https://pixabay.com/photos/steering-wheel-blue-twist-sea-981439/
    Agenda ship sailing in the sun: https://pixabay.com/photos/sea-sailing-vessel-boat-ship-701079/

    View Slide