Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Custom Resources in Kubernetes

Introduction to Custom Resources in Kubernetes

This talk is an introduction to custom resources in Kubernetes. Custom resources are a way to extend the Kubernetes API to create objects of a kind. Custom resources are implemented with Custom Resource Definitions (CRDs) and, when coupled with a custom controller, they offer a way to implement your own declarative API.

An sample custom resource of type `HelmRelease` is introduced along with an example of a custom controller used to handle Helm releases.

Pauline Lallinec

March 28, 2019
Tweet

More Decks by Pauline Lallinec

Other Decks in Technology

Transcript

  1. Introduction to Custom Resources Pauline Lallinec Workday Dublin Docker Meetup,

    March 2018
  2. Software Engineer II - DevOps N - @plallin Workday Public

    Cloud Engineering We’re hiring! Best logo ever (or so I tell my director) ->
  3. • Core concepts: Kubernetes, Helm, CRDs • Overview of the

    unicorn app • What are CRDs? • What are custom controllers? • Demo of CRDs + Custom controller using unicorns • Implementation of the Helm Release controller • More demo using unicorns No Agenda No Attenda
  4. Before we start: Kubernetes “Kubernetes (K8s) is an open-source system

    for automating deployment, scaling, and management of containerized applications.” Long story short: it deploys and manages your (Docker) containers for you.
  5. Before we start: Helm • Package manager for Kubernetes •

    Helm chart: a set of Kubernetes resources • Helm release: a version of a Helm Chart • 2 parts: The Helm client (Helm), and the Helm server (Tiller)
  6. Custom Resources A way to create custom objects that live

    within your cluster, and are handled by a custom controller running a logic of your own. (Ideally) CRDs responds to CRUD events (Create, Read, Update, Delete) and allow you to implement your own declarative API.
  7. Custom Resources Example CRDs in Workday S3Bucket a CRD to

    create, delete, update and retrieve data from S3 Buckets HelmRelease a CRD to install, upgrade, delete, and store information on Helm releases
  8. I A very simple app! The app One single HTML

    page showing a unicorn, serviced by Python’s SimpleHTTPServer Kubernetes resources One deployment, with only 1 container containing the Unicorn app 3 Helm charts - Pink unicorn - Blue unicorn - Green unicorn
  9. I : D

  10. CRDs: why the interest? “In the future there should be

    nothing we [Kubernetes Developers] can do that you [Kubernetes users] can’t” Tim Hockin (@thockin) Kubernetes co-founder / pioneer & Co-chair of Kubernetes Network SIG “CRDs aren’t just for add-ons anymore - painting a picture for the future”, Lightning Talk, Kubecon North America 2018
  11. Standalone CRDs • Custom object with their own API endpoint

    • Store / retrieve structured data CRDs + Custom controllers • Declarative API What are CRDs?
  12. What are CRDs? apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: helmreleases.samplecontroller.k8s.io

    spec: group: samplecontroller.k8s.io version: v1alpha1 names: kind: HelmRelease plural: helmreleases scope: Namespaced Custom object with their own API endpoint
  13. $ kubectl create -f helm_release_crd.yaml customresourcedefinition.apiextensions.k8s.io/helmreleases.sample controller.k8s.io created $ kubectl

    get crd NAME CREATED AT helmreleases.samplecontroller.k8s.io 2019-03-23T05:21:43Z Custom object with their own API endpoint What are CRDs?
  14. apiVersion: samplecontroller.k8s.io/v1alpha1 kind: HelmRelease metadata: name: unicorn-release spec: releaseVersion: pink

    releaseName: unicorn Store / retrieve structured data What are CRDs?
  15. $ kubectl create -f unicorn-release-pink.yaml helmrelease.samplecontroller.k8s.io/unicorn-release created $ kubectl get

    helmreleases NAME AGE unicorn-release 36s Store / retrieve structured data What are CRDs?
  16. $ kubectl describe helmrelease unicorn-release Name: unicorn-release Namespace: default API

    Version: samplecontroller.k8s.io/v1alpha1 Kind: HelmRelease Metadata: ... Spec: Release Name: unicorn Release Version: pink Events: <none> Store / retrieve structured data What are CRDs?
  17. • Listen to any resource type • Ensure desired state

    = existing state • If desired state ≠ existing state, will take action to make existing state = desired state • This is implemented using your own logic! Clone kubernetes/sample-controller from GitHub for an example of a sample controller Custom controllers
  18. Helm Release Controller • Listen to CRDs of type HelmRelease

    • Ensures all desired Helm releases are installed / upgraded • Will install / upgrade the Helm release if not already installed / not upgraded to desired version Custom controllers
  19. (No) Helm Release Controller • Current situation: run a script

    to handle installation / upgrade / rollback of Helm resources • Need a way (i.e. Jenkins job) to run that script to install / upgrade / rollback Helm charts • Imperative rather than declarative Custom controllers
  20. Helm Release Controller • Cluster logic remains within the cluster

    • Declarative API: let the cluster manage itself • No need for additional script / Jenkins job Custom controllers
  21. Helm Release Controller • Automated rollback according to a logic

    of our own • Allow for custom business logic • No need to install / maintain the Helm Client on different servers Custom controllers
  22. U CRD’

  23. (Unicorn) HelmRelease New Helm Release? Y: Install Helm Release N:

    Upgrade Helm Release U CRD’
  24. U CRD’ :

  25. Clone of the existing Sample Controller from Kubernetes No update

    done to listeners, informers, event handlers, etc. Focus on SyncHandlers() which is responsible for ensuring that desired state = existing state kubernetes/sample-controller: https://github.com/kubernetes/sample-controller Helm Release Controller: the implementation
  26. Receive a CRD of type Helm Release Helm Release Controller:

    the implementation
  27. Get information on CRD of type HelmRelease Helm Release Controller:

    the implementation
  28. Get information about the current CRD (= desired state) Helm

    Release Controller: the implementation
  29. Install Helm release if it doesn’t already exist (= match

    desired state) Helm Release Controller: the implementation
  30. Check existing deployment (= existing state) Helm Release Controller: the

    implementation
  31. Upgrade existing release if necessary (= match desired state) Helm

    Release Controller: the implementation
  32. Update Helm Release status Helm Release Controller: the implementation

  33. Update Helm Release status Helm Release Controller: the implementation

  34. If an error happens, re-enqueue the event and retry later

    Helm Release Controller: the implementation
  35. Finally, return successful sync event Helm Release Controller: the implementation

  36. • Choice of programming language • Can enforce validation (example

    to follow) • Can support /status and /scale subresources (and maybe /exec and /log in the future) CRDs + Custom controllers: Other benefits
  37. apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: ... spec: ... validation: openAPIV3Schema:

    properties: spec: properties: releaseVersion: type: string pattern: "(pink|blue)" E
  38. HelmRelease Y: New Helm Release? Y: Install Helm Release N:

    Upgrade Helm Release Is this unicorn release pink or blue? N: reject create / update event E
  39. E :

  40. Thank you! Join my team! Apply to “[Senior] Devops Engineer

    - DevOps in Cloud Orchestration” on Workday (Link: https://lnkd.in/gkMHwMC), or directly introduce yourself to Dave Doran (he’s here tonight!). You will be able to work with trendy technologies AND join the karaoke club. How cool is that? Follow me on Twitter at @plallin to hear me brag about how much I walk (a lot!), how much I lift (really not a lot), and laugh about my struggles with Arch Linux.
  41. Pictures! All pictures downloaded from Pixabay https://pixabay.com/ Ship of containers:

    https://pixabay.com/photos/hamburg-port-of-hamburg-3021820/ Ending ship of containers: https://pixabay.com/photos/container-container-ship-port-1611490/ Unicorns: https://pixabay.com/vectors/unicorn-unicorn-crown-flower-crown-3392560/ Picture of a Helm on a blue background: https://pixabay.com/photos/steering-wheel-blue-twist-sea-981439/ Agenda ship sailing in the sun: https://pixabay.com/photos/sea-sailing-vessel-boat-ship-701079/