Using Helm to achieve frictionless deployments

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Using Helm to achieve frictionless deployments Pauline Lallinec, Workday Public Cloud Continuous Lifecycle London, July 2020

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

99.96% uptime 44 million users 195 billion transactions* What is Workday? * FY20

Slide 5

Slide 5 text

Software Engineer - DevOps Non-stop karaoke machine @plallin Workday + Public Cloud = Scylla Amazon AWS (US, EU, Canada) + Workday DC 4 teams, 2 continents

Slide 6

Slide 6 text

● Overview of Kubernetes resources ● Helm: package manager for Kubernetes ● Helm chart structures ● The need for Helm charts release automation ● Overview of custom resources & custom controllers ● Example: Custom Helm release controllers ● Helm operators Agenda

Slide 7

Slide 7 text

About Kubernetes “Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.” Long story short: it deploys and manages your (Docker) containers for you.

Slide 8

Slide 8 text

About Kubernetes Kubernetes = YAML ﬁles apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 3 template: spec: containers: - name: busybox image: busybox:latest ...

Slide 9

Slide 9 text

About Kubernetes Packaging an application for Kubernetes: ● Deployment ● Service ● Conﬁgmaps ● ...

Slide 10

Slide 10 text

About Kubernetes Solution: use a package manager ● Package application ● Delivery consistency ● Templating

Slide 11

Slide 11 text

About Helm ● Package manager for Kubernetes ● Helm chart: a set of Kubernetes resources ● Helm release: a version of a Helm Chart ● 2 versions of Helm: Helm 2 and Helm 3 ○ Favor using Helm 3

Slide 12

Slide 12 text

Introducing unicorns A very simple app! The app One single HTML page showing a unicorn, serviced by Python’s SimpleHTTPServer Kubernetes resources One deployment, with only 1 container containing the Unicorn app A service, to allow me access the website from my laptop 3 versions: pink, blue, green

Slide 13

Slide 13 text

The need for Helm release automation

Slide 14

Slide 14 text

The need for Helm release automation Problem Need to ship more Kubernetes resources Solution Helm for Kubernetes packaging + versioning Next Deploy Helm releases reliably

Slide 15

Slide 15 text

The need for Helm release automation Priority: reliability Solution Script handling upgrades & automatically rollbacks failed releases

Slide 16

Slide 16 text

The need for Helm release automation Priority: reliability Solution Script handling upgrades & automatically rollbacks failed releases Problems ● Lack of automation ● Does not scale ● Additional server maintenance (Jenkins)

Slide 17

Slide 17 text

The need for Helm release automation Requirements ● Automation ● Reliability ● Observability

Slide 18

Slide 18 text

The need for Helm release automation Requirements ● Automation ● Reliability ● Observability Solution An in-cluster service to manage all incoming Helm releases Choices: ● Own custom controller ● Flux Helm Operator

Slide 19

Slide 19 text

Building your own custom controller / operator What is it? ● Your own Kubernetes controller ● Running your own logic

Slide 20

Slide 20 text

Building your own custom controller / operator Pros Cons Your own code, with ability to add custom features and logic Your own code, with responsibility to maintain and extend it Can manage non cloud-native services Initially, need to commit time to develop it Automated rollback Control over delivery

Slide 21

Slide 21 text

Flux Helm Operator What is it? ● Operator oﬀered by WeaveWorks ● Weave Flux: CI/CD for Helm charts ● Flux Helm Operator: Helm release manager

Slide 22

Slide 22 text

Installing Flux Helm Operator Pros Cons Someone else’s code, beneﬁtting from community inputs Someone else’s code Open source & community-driven In most companies, need to go through security review / approval process Production-ready Updates subjected to external PR review & approval Regularly updated by Fluxcd + community No control over delivery

Slide 23

Slide 23 text

Custom resources & Custom controllers

Slide 24

Slide 24 text

Custom resources A way to create custom objects that live within your cluster, and are handled by a custom controller running a logic of your own. (Ideally) CRDs responds to CRUD events (Create, Read, Update, Delete) and allow you to implement your own declarative API.

Slide 25

Slide 25 text

Standalone CRDs ● Custom object with their own API endpoint ● Store / retrieve structured data CRDs + Custom controllers ● Declarative API Custom resource deﬁnitions

Slide 26

Slide 26 text

Helm Releases ● Object type: HelmRelease ● Object definition: ○ Release name The name of the application ○ Release version The version of application Example: custom resources

Slide 27

Slide 27 text

Custom resource deﬁnitions apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: helmreleases.samplecontroller.k8s.io spec: group: samplecontroller.k8s.io version: v1alpha1 names: kind: HelmRelease plural: helmreleases scope: Namespaced Custom object with their own API endpoint

Slide 28

Slide 28 text

$ kubectl create -f helm_release_crd.yaml customresourcedefinition.apiextensions.k8s.io/helmreleases.sample controller.k8s.io created $ kubectl get crd NAME CREATED AT helmreleases.samplecontroller.k8s.io 2019-03-23T05:21:43Z Custom object with their own API endpoint Custom resource deﬁnitions

Slide 29

Slide 29 text

apiVersion: samplecontroller.k8s.io/v1alpha1 kind: HelmRelease metadata: name: unicorn-release spec: releaseVersion: pink releaseName: unicorn Store / retrieve structured data Custom resource deﬁnitions

Slide 30

Slide 30 text

$ kubectl create -f unicorn-release-pink.yaml helmrelease.samplecontroller.k8s.io/unicorn-release created $ kubectl get helmreleases NAME AGE unicorn-release 36s Store / retrieve structured data Custom resource deﬁnitions

Slide 31

Slide 31 text

$ kubectl describe helmrelease unicorn-release Name: unicorn-release Namespace: default API Version: samplecontroller.k8s.io/v1alpha1 Kind: HelmRelease Metadata: ... Spec: Release Name: unicorn Release Version: pink Events: Store / retrieve structured data Custom resource deﬁnitions

Slide 32

Slide 32 text

Custom controllers

Slide 33

Slide 33 text

● Watches the current state of the cluster ● Ensure desired state of cluster = current state of cluster ● If desired state ≠ current state, will take action to make them match Controller pattern

Slide 34

Slide 34 text

● Listen to any resource type ● Ensure existing state of resource type = desired state of resource type ● If desired state ≠ existing state, will take action to make existing state = desired state ● This is implemented using your own logic! Clone kubernetes/sample-controller from GitHub for an example of a sample controller Custom controllers

Slide 35

Slide 35 text

Helm Release Controller ● Listen to CRDs of type HelmRelease ● Ensures all desired Helm releases are installed / upgraded ● Will install / upgrade the Helm release if not already installed / not upgraded to desired version Example: custom controllers

Slide 36

Slide 36 text

● Cluster logic remains within the cluster ● Declarative API: let the cluster manage itself ● No need for additional script / Jenkins job Helm Release Controller

Slide 37

Slide 37 text

● Automated rollback according to a logic of our own ● Allow for custom business logic ● (Helm 2 only) No need to install / maintain the Helm Client on diﬀerent servers Helm Release Controller

Slide 38

Slide 38 text

(Unicorn) HelmRelease New Helm Release? Y: Install Helm Release N: Upgrade Helm Release Unicorns CRD’d

Slide 39

Slide 39 text

Unicorns CRD’d : the demo

Slide 40

Slide 40 text

Clone of the existing Sample Controller from Kubernetes No update done to listeners, informers, event handlers, etc. Focus on SyncHandlers() which is responsible for ensuring that desired state = existing state kubernetes/sample-controller: https://github.com/kubernetes/sample-controller Helm Release Controller: the implementation

Slide 41

Slide 41 text

Receive a CRD of type Helm Release Helm Release Controller: the implementation

Slide 42

Slide 42 text

Get information on CRD of type HelmRelease Helm Release Controller: the implementation

Slide 43

Slide 43 text

Get information about the current CRD (= desired state) Helm Release Controller: the implementation

Slide 44

Slide 44 text

Install Helm release if it doesn’t already exist (= match desired state) Helm Release Controller: the implementation

Slide 45

Slide 45 text

Check existing deployment (= existing state) Helm Release Controller: the implementation

Slide 46

Slide 46 text

Upgrade existing release if necessary (= match desired state) Helm Release Controller: the implementation

Slide 47

Slide 47 text

Update Helm Release status Helm Release Controller: the implementation

Slide 48

Slide 48 text

Update Helm Release status Helm Release Controller: the implementation

Slide 49

Slide 49 text

If an error happens, re-enqueue the event and retry later Helm Release Controller: the implementation

Slide 50

Slide 50 text

Finally, return successful sync event Helm Release Controller: the implementation

Slide 51

Slide 51 text

● Choice of programming language ● Can enforce validation ● Can support /status and /scale subresources (and maybe /exec and /log in the future*) * https://github.com/kubernetes/kubernetes/issues/72637 CRDs + Custom controllers: Other beneﬁts

Slide 52

Slide 52 text

Flux Helm Operator

Slide 53

Slide 53 text

● Extension to Weave Flux ● Essentially a custom controller built by Flux ● Open-source ● Production ready ● Handles rollback of failed Helm releases ● Uses GitOps ﬂow ● Compatible with Helm 2 and Helm 3 Flux Helm Operator

Slide 54

Slide 54 text

Flux Helm Operator - Reacts to commits in a repo - Create a custom resource of type “HelmRelease” - The operator watches that resource and installs / upgrades the release in question

Slide 55

Slide 55 text

Flux Helm Operator The HelmRelease custom resource: ● Oﬀers many customization option ● Allows you to set up access to your chart repository ● Provides visibility on its status in `kubectl describe` https://github.com/fluxcd/helm-operator/blob/master/chart/helm-operator/README.md https://github.com/fluxcd/helm-operator/blob/master/deploy/crds.yaml

Slide 56

Slide 56 text

● Use cases for Helm ● Why we need Helm release automation ● Comparison of custom controllers vs Helm operators ● Overview of custom resources & custom controllers ● Example Helm release custom controllers ● Example Helm operator Key takeaways

Slide 57

Slide 57 text

This presentation features not only my work, but my entire team’s work, and therefore I would like to recognize their contribution :-) Thank you Scylla + Fabrication Team Slide not included in the presentation

Slide 58

Slide 58 text

Farouq Cathal Adrian Sathish David John Rob Lucas Joe Aideen Declan

Slide 59

Slide 59 text

Thank you! Learn more more about engineering at Workday! medium.com/workday-engineering Learn more about opportunities at Workday! workday.com/careers Learn more about me! @plallin plallin.dev