Introduction to Custom Resources in Kubernetes

This talk is an introduction to custom resources in Kubernetes. Custom resources are a way to extend the Kubernetes API to create objects of a kind. Custom resources are implemented with Custom Resource Definitions (CRDs) and, when coupled with a custom controller, they offer a way to implement your own declarative API.

An sample custom resource of type `HelmRelease` is introduced along with an example of a custom controller used to handle Helm releases.

Transcript

1. Introduction to Custom Resources Pauline Lallinec Workday Dublin Docker Meetup, March 2018
2. Software Engineer II - DevOps N - @plallin Workday Public Cloud Engineering We’re hiring! Best logo ever (or so I tell my director) ->
3. • Core concepts: Kubernetes, Helm, CRDs • Overview of the unicorn app • What are CRDs? • What are custom controllers? • Demo of CRDs + Custom controller using unicorns • Implementation of the Helm Release controller • More demo using unicorns No Agenda No Attenda
4. Before we start: Kubernetes “Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.” Long story short: it deploys and manages your (Docker) containers for you.
5. Before we start: Helm • Package manager for Kubernetes • Helm chart: a set of Kubernetes resources • Helm release: a version of a Helm Chart • 2 parts: The Helm client (Helm), and the Helm server (Tiller)
6. Custom Resources A way to create custom objects that live within your cluster, and are handled by a custom controller running a logic of your own. (Ideally) CRDs responds to CRUD events (Create, Read, Update, Delete) and allow you to implement your own declarative API.
7. Custom Resources Example CRDs in Workday S3Bucket a CRD to create, delete, update and retrieve data from S3 Buckets HelmRelease a CRD to install, upgrade, delete, and store information on Helm releases
8. I A very simple app! The app One single HTML page showing a unicorn, serviced by Python’s SimpleHTTPServer Kubernetes resources One deployment, with only 1 container containing the Unicorn app 3 Helm charts - Pink unicorn - Blue unicorn - Green unicorn
9. I : D
10. CRDs: why the interest? “In the future there should be nothing we [Kubernetes Developers] can do that you [Kubernetes users] can’t” Tim Hockin (@thockin) Kubernetes co-founder / pioneer & Co-chair of Kubernetes Network SIG “CRDs aren’t just for add-ons anymore - painting a picture for the future”, Lightning Talk, Kubecon North America 2018
11. Standalone CRDs • Custom object with their own API endpoint • Store / retrieve structured data CRDs + Custom controllers • Declarative API What are CRDs?
12. What are CRDs? apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: helmreleases.samplecontroller.k8s.io spec: group: samplecontroller.k8s.io version: v1alpha1 names: kind: HelmRelease plural: helmreleases scope: Namespaced Custom object with their own API endpoint
13. $ kubectl create -f helm_release_crd.yaml customresourcedefinition.apiextensions.k8s.io/helmreleases.sample controller.k8s.io created $ kubectl get crd NAME CREATED AT helmreleases.samplecontroller.k8s.io 2019-03-23T05:21:43Z Custom object with their own API endpoint What are CRDs?
14. apiVersion: samplecontroller.k8s.io/v1alpha1 kind: HelmRelease metadata: name: unicorn-release spec: releaseVersion: pink releaseName: unicorn Store / retrieve structured data What are CRDs?
15. $ kubectl create -f unicorn-release-pink.yaml helmrelease.samplecontroller.k8s.io/unicorn-release created $ kubectl get helmreleases NAME AGE unicorn-release 36s Store / retrieve structured data What are CRDs?
16. $ kubectl describe helmrelease unicorn-release Name: unicorn-release Namespace: default API Version: samplecontroller.k8s.io/v1alpha1 Kind: HelmRelease Metadata: ... Spec: Release Name: unicorn Release Version: pink Events: <none> Store / retrieve structured data What are CRDs?
17. • Listen to any resource type • Ensure desired state = existing state • If desired state ≠ existing state, will take action to make existing state = desired state • This is implemented using your own logic! Clone kubernetes/sample-controller from GitHub for an example of a sample controller Custom controllers
18. Helm Release Controller • Listen to CRDs of type HelmRelease • Ensures all desired Helm releases are installed / upgraded • Will install / upgrade the Helm release if not already installed / not upgraded to desired version Custom controllers
19. (No) Helm Release Controller • Current situation: run a script to handle installation / upgrade / rollback of Helm resources • Need a way (i.e. Jenkins job) to run that script to install / upgrade / rollback Helm charts • Imperative rather than declarative Custom controllers
20. Helm Release Controller • Cluster logic remains within the cluster • Declarative API: let the cluster manage itself • No need for additional script / Jenkins job Custom controllers
21. Helm Release Controller • Automated rollback according to a logic of our own • Allow for custom business logic • No need to install / maintain the Helm Client on different servers Custom controllers
22. U CRD’
23. (Unicorn) HelmRelease New Helm Release? Y: Install Helm Release N: Upgrade Helm Release U CRD’
24. U CRD’ :
25. Clone of the existing Sample Controller from Kubernetes No update done to listeners, informers, event handlers, etc. Focus on SyncHandlers() which is responsible for ensuring that desired state = existing state kubernetes/sample-controller: https://github.com/kubernetes/sample-controller Helm Release Controller: the implementation
26. Receive a CRD of type Helm Release Helm Release Controller: the implementation
27. Get information on CRD of type HelmRelease Helm Release Controller: the implementation
28. Get information about the current CRD (= desired state) Helm Release Controller: the implementation
29. Install Helm release if it doesn’t already exist (= match desired state) Helm Release Controller: the implementation
30. Check existing deployment (= existing state) Helm Release Controller: the implementation
31. Upgrade existing release if necessary (= match desired state) Helm Release Controller: the implementation
32. Update Helm Release status Helm Release Controller: the implementation
33. Update Helm Release status Helm Release Controller: the implementation
34. If an error happens, re-enqueue the event and retry later Helm Release Controller: the implementation
35. Finally, return successful sync event Helm Release Controller: the implementation
36. • Choice of programming language • Can enforce validation (example to follow) • Can support /status and /scale subresources (and maybe /exec and /log in the future) CRDs + Custom controllers: Other benefits
37. apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: ... spec: ... validation: openAPIV3Schema: properties: spec: properties: releaseVersion: type: string pattern: "(pink|blue)" E
38. HelmRelease Y: New Helm Release? Y: Install Helm Release N: Upgrade Helm Release Is this unicorn release pink or blue? N: reject create / update event E
39. E :
40. Thank you! Join my team! Apply to “[Senior] Devops Engineer - DevOps in Cloud Orchestration” on Workday (Link: https://lnkd.in/gkMHwMC), or directly introduce yourself to Dave Doran (he’s here tonight!). You will be able to work with trendy technologies AND join the karaoke club. How cool is that? Follow me on Twitter at @plallin to hear me brag about how much I walk (a lot!), how much I lift (really not a lot), and laugh about my struggles with Arch Linux.
41. Pictures! All pictures downloaded from Pixabay https://pixabay.com/ Ship of containers: https://pixabay.com/photos/hamburg-port-of-hamburg-3021820/ Ending ship of containers: https://pixabay.com/photos/container-container-ship-port-1611490/ Unicorns: https://pixabay.com/vectors/unicorn-unicorn-crown-flower-crown-3392560/ Picture of a Helm on a blue background: https://pixabay.com/photos/steering-wheel-blue-twist-sea-981439/ Agenda ship sailing in the sun: https://pixabay.com/photos/sea-sailing-vessel-boat-ship-701079/