Implementing Microservices as Kubernetes Operators

Slide 1

Slide 1 text

#devconf_cz @jewzaam / @thedoh Implementing Microservices as Kubernetes Operators 1 Naveen Malik, Sr. Principal SRE Lisa Seelye, Sr. SRE

Slide 2

Slide 2 text

#devconf_cz @jewzaam / @thedoh You will leave with... 2 ● High level understanding Kubernetes ● Understand why Operators are so great ● Why we believe Operators are Microservices ● How to create, build, deploy and test a new Operator

Slide 3

Slide 3 text

#devconf_cz @jewzaam / @thedoh Naveen Malik (@jewzaam) Senior Principal Site Reliability Engineer 1. Software Engineer / Architect 2. Father 3. Runner 4. Maker About Your Speakers 3 Lisa Seelye (@thedoh) Senior Site Reliability Engineer 1. Sysadmin & Sw Eng background 2. Canadian-in-training 3. Alternate arch fan 4. Cat enthusiast

Slide 4

Slide 4 text

#devconf_cz @jewzaam / @thedoh 4

Slide 5

Slide 5 text

#devconf_cz @jewzaam / @thedoh What is Kubernetes? 5 ● Open-source platform ● Manages containerized workloads ● Facilitates declarative configuration & automation ● Provides standard Resources to express application deployments ● Supports custom Resources and logic around those Resources Ref https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

Slide 6

Slide 6 text

#devconf_cz @jewzaam / @thedoh Why are we here? 6 ● OpenShift SRE team operates OpenShift Dedicated ● Operators for all the things! OpenShift? ● Flavour of Kubernetes ● OpenShift is to Kubernetes as RHEL is to Linux OpenShift Dedicated? ● High availability, multi-AZ OpenShift clusters on AWS ● Deploy clusters into our AWS accounts or bring your own ● Fully managed and supported by Red Hat, 24x7, with 99.5% uptime SLA

Slide 7

Slide 7 text

#devconf_cz @jewzaam / @thedoh "world map" is licensed under CC0 1.0 7 5 2 1 2 2 SRE is hiring! Stop by at Red Hat booth to find out more! (January 15, 2020)

Slide 8

Slide 8 text

#devconf_cz @jewzaam / @thedoh How Does Kubernetes Apply Changes? 8

Slide 9

Slide 9 text

#devconf_cz @jewzaam / @thedoh ● Watches for change to resources ● Makes it so How do Resources and Controllers interact? 9 ● Use YAML to show desired state ● Controllers make it happen ● Examples (partial): ○ Deployment ○ Pod ○ CustomResourceDefinition Stock Kubernetes Resources Stock Kubernetes Controllers

Slide 10

Slide 10 text

#devconf_cz @jewzaam / @thedoh What are Operators? ● Custom behavior for existing resources ● Custom behavior for custom resources ● Move current state closer to desired state 10 ● They don’t ship with Kubernetes ● Models a specific application domain ● Anything is possible! Custom Resource Definitions Custom Kubernetes Controllers

Slide 11

Slide 11 text

#devconf_cz @jewzaam / @thedoh 11 Deployment Controller Waiting for changes

Slide 12

Slide 12 text

#devconf_cz @jewzaam / @thedoh 12 Deployment Controller Tries to create Pods New Deployment Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ...

Slide 13

Slide 13 text

#devconf_cz @jewzaam / @thedoh 13 Deployment Controller Error detected! New Deployment kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ^ # error! ---------/ ...

Slide 14

Slide 14 text

#devconf_cz @jewzaam / @thedoh 14 Deployment Controller Success! New Deployment Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7.9 ...

Slide 15

Slide 15 text

#devconf_cz @jewzaam / @thedoh 15 Deployment Controller Waiting for changes

Slide 16

Slide 16 text

#devconf_cz @jewzaam / @thedoh 16 What Does it Look Like for Operators?

Slide 17

Slide 17 text

#devconf_cz @jewzaam / @thedoh 17 Custom Controller Waiting for changes

Slide 18

Slide 18 text

#devconf_cz @jewzaam / @thedoh Why Implement Your Microservice as an Operator? 18

Slide 19

Slide 19 text

#devconf_cz @jewzaam / @thedoh What are Microservices? 19 ● Small with bounded contexts ● Communicate over a network ● Use technology-agnostic protocols ● Organized around business capabilities ● Independently deployable In addition, usually... ● Security ● Availability ● Scalability Processes with... Read More: https://en.wikipedia.org/wiki/Microservices

Slide 20

Slide 20 text

#devconf_cz @jewzaam / @thedoh What are Microservices? ● Have a narrow scope ● Communicate over platform’s network ● provide an API via Kubernetes resources ● Group controllers based on business need ● Are a deployable unit Kubernetes provides… ● Authentication & Authorization ● Highly available platform & deployments ● Scale deployment to meet needs 20 ● Small with bounded contexts ● Communicate over a network ● Use technology-agnostic protocols ● Organized around business capabilities ● Independently deployable In addition, usually... ● Security ● Availability ● Scalability Processes with... Kubernetes Operators... Read More: https://en.wikipedia.org/wiki/Microservices

Slide 21

Slide 21 text

#devconf_cz @jewzaam / @thedoh Operators are Microservices 21

Slide 22

Slide 22 text

#devconf_cz @jewzaam / @thedoh Using Microservices for Config Management 22

Slide 23

Slide 23 text

#devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 23 Configuration Inventory Apply Changes Cluster 1

Slide 24

Slide 24 text

#devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 24 Configuration Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3

Slide 25

Slide 25 text

#devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 25 Configuration Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...

Slide 26

Slide 26 text

#devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 26 Configuration Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...

Slide 27

Slide 27 text

#devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 27 Configuration Inventory Apply Changes Cluster 1 Desired State

Slide 28

Slide 28 text

#devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 28 Configuration Inventory Apply Changes Cluster 1 Controller Desired State Realized State

Slide 29

Slide 29 text

#devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 29 Configuration Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...

Slide 30

Slide 30 text

#devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 30 Configuration Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...

Slide 31

Slide 31 text

#devconf_cz @jewzaam / @thedoh Systems Administration as Code 31

Slide 32

Slide 32 text

#devconf_cz @jewzaam / @thedoh Operators, by Example 32

Slide 33

Slide 33 text

#devconf_cz @jewzaam / @thedoh Operator Example: Install & Configure 33 cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator

Slide 34

Slide 34 text

#devconf_cz @jewzaam / @thedoh Installing & Configuring Software, Without Operators 34 ● Install Prometheus, Alertmanager, and Grafana by hand ○ Did you make them highly available? ● Install kube-state-metrics on your own ● Install node_exporter on your own ● Write all the “glue” to configure Prometheus, Alertmanager and Grafana. Don’t forget, Deployments don’t watch ConfigMaps automatically

Slide 35

Slide 35 text

#devconf_cz @jewzaam / @thedoh ● Installs & Configures ○ Prometheus-operator (Prometheus and Alertmanager) ○ Grafana ○ Kube-state-metrics ○ Node_exporter ● Prometheus-operator custom resources ○ alerting rules ○ configuration Cluster-monitoring-operator, Operator Value 35

Slide 36

Slide 36 text

#devconf_cz @jewzaam / @thedoh apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring Cluster-monitoring-operator, Custom Resource 36

Slide 37

Slide 37 text

#devconf_cz @jewzaam / @thedoh Cluster-monitoring-operator, Custom Resource 37 apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring spec: groups: - name: sre-node-unschedulable rules: - alert: KubeNodeUnschedulableSRE expr: kube_node_spec_unschedulable > 0 for: 1h labels: severity: warning namespace: openshift-monitoring ...

Slide 38

Slide 38 text

#devconf_cz @jewzaam / @thedoh Operator Example: Provision Credentials 38 cloud-credential-operator https://github.com/openshift/cloud-credential-operator

Slide 39

Slide 39 text

#devconf_cz @jewzaam / @thedoh AWS Account Credentials, Without Operators 39 ● Send a request to IT to get credentials ● They validate your privilege and access, and then send to you, encrypted ● You keep them secure somehow ● Put them into Secret object ● Figure out how to scale this across x clusters, y users, z cloud providers

Slide 40

Slide 40 text

#devconf_cz @jewzaam / @thedoh ● User creates CredentialsRequest object ○ Operator watches for it ● Operator talks to Amazon to create credential pair ● Operator stores results in a Secret ● User accesses the Secret How cloud-credential-operator mints credentials 40

Slide 41

Slide 41 text

#devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf Cloud-credential-operator CredentialsRequest 41

Slide 42

Slide 42 text

Slide 43

Slide 43 text

#devconf_cz @jewzaam / @thedoh Cloud-credential-operator CredentialsRequest 43 apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - ec2:DescribeInstances resource: "*"

Slide 44

Slide 44 text

#devconf_cz @jewzaam / @thedoh Cloud-credential-operator Resulting Secret 44 apiVersion: v1 kind: Secret metadata: name: secret-to-store-credentials namespace: devconf type: Opaque data: aws_secret_access_key: bm90IGEgcmVhbCBzZWNyZXQK aws_access_key_id: REVGSU5JVEVMWSBub3QgYSByZWFsIHNlY3JldAo=

Slide 45

Slide 45 text

#devconf_cz @jewzaam / @thedoh Operator Example: Manage Certificates 45 service-ca-operator https://github.com/openshift/service-ca-operator

Slide 46

Slide 46 text

#devconf_cz @jewzaam / @thedoh Service-ca-operator, Without Operators 46 ● Connect to the appropriate cluster and create a CSR ● kubectl certificate approve a CertificateSigningRequest ● Keep secrets safe ● Figure out how to scale over n clusters

Slide 47

Slide 47 text

#devconf_cz @jewzaam / @thedoh ● Native objects are annotated ● Operator’s controllers watch for them ● Creates certificates and stores in a Secret How service-ca-operator signs certificates 47

Slide 48

Slide 48 text

#devconf_cz @jewzaam / @thedoh apiVersion: v1 kind: Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name : "devconf-service-certs" Service-ca-operator Details 48

Slide 49

Slide 49 text

#devconf_cz @jewzaam / @thedoh Service-ca-operator Details 49 apiVersion: v1 kind: Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ...

Slide 50

Slide 50 text

#devconf_cz @jewzaam / @thedoh Service-ca-operator Details 50 apiVersion: v1 kind: Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ... --- apiVersion: v1 kind: ConfigMap metadata: annotations: service.beta.openshift.io/inject-cabundle : "true" name: "devconf-service-cabundle " data: {} # .service-ca.crt added by operator

Slide 51

Slide 51 text

#devconf_cz @jewzaam / @thedoh Operators Increase Velocity, Scalability, Availability, ... 51

Slide 52

Slide 52 text

#devconf_cz @jewzaam / @thedoh Operator Demo: Create Pod 52 pod-operator (contrived example)

Slide 53

Slide 53 text

#devconf_cz @jewzaam / @thedoh Why create this operator? 53 Show how to… 1. Create a golang operator 2. Create a Custom Resource Definition (CRD) 3. Create a Custom Controller

Slide 54

Slide 54 text

#devconf_cz @jewzaam / @thedoh What does the Operator do? 54 Create Pod from a Custom Resource Ensure Pod exists for each Custom Resource Cleanup Pod when Custom Resource is deleted

Slide 55

Slide 55 text

#devconf_cz @jewzaam / @thedoh Tools 55 ● operator-sdk ○ software development toolkit (SDK) for building Kubernetes applications ● podman ○ daemonless container engine for developing, managing, and running containers ● kubectl ○ command line tool for controlling Kubernetes clusters

Slide 56

Slide 56 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 56

Slide 57

Slide 57 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 57 1. operator-sdk new pod-operator

Slide 58

Slide 58 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 58 1. operator-sdk new 2. operator-sdk add api pod-operator PodRequest (CRD) --- spec:

Slide 59

Slide 59 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 59 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller pod-operator PodRequest (CRD) --- spec: Controller --- - watch PodRequest & Pod - assign Owner - create Pod

Slide 60

Slide 60 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 60 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod

Slide 61

Slide 61 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 61 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io

Slide 62

Slide 62 text

#devconf_cz @jewzaam / @thedoh How is the Operator created? 62 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push 7. kubectl create pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io

Slide 63

Slide 63 text

#devconf_cz @jewzaam / @thedoh What are we testing? 63 Create Pod from a Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io

Slide 64

Slide 64 text

#devconf_cz @jewzaam / @thedoh What are we testing? 64 Ensure Pod exists for each Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io

Slide 65

Slide 65 text

#devconf_cz @jewzaam / @thedoh What are we testing? 65 Cleanup Pod when Custom Resource is deleted pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io

Slide 66

Slide 66 text

#devconf_cz @jewzaam / @thedoh Demo Time! 66

Slide 67

Slide 67 text

#devconf_cz @jewzaam / @thedoh Bootstrapping an operator from scratch! 67 export GO111MODULE=on export QUAY_USERNAME=nmalik export GITHUB_USERNAME=jewzaam export OP_VERSION=0.0.1 export OP_NAME=pod-operator export OP_API_GROUP=pod.jewzaam.org export OP_API_VERSION=v1alpha1 export OP_KIND=PodRequest # bootstrap the operator mkdir -p $GOPATH/src/github.com/ $GITHUB_USERNAME cd $GOPATH/src/github.com/ $GITHUB_USERNAME operator-sdk new $OP_NAME cd $OP_NAME # generate CRD and Controller operator-sdk add api --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND operator-sdk add controller --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND

Slide 68

Slide 68 text

#devconf_cz @jewzaam / @thedoh Customize the API 68 ... type PodRequestSpec struct { Name string `json:"name"` Image string `json:"image"` Command []string `json:"command"` } ...

Slide 69

Slide 69 text

#devconf_cz @jewzaam / @thedoh Customize the Controller 69 func newPodForCR(cr *podv1alpha1.PodRequest) *corev1.Pod { labels := map[string]string{ "app": cr.Name, } return &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: cr.Name, Namespace: cr.Namespace, Labels: labels, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: cr.Spec.Name, Image: cr.Spec.Image, Command: cr.Spec.Command, }, }, }, } }

Slide 70

Slide 70 text

#devconf_cz @jewzaam / @thedoh Build and Publish: Operator 70 # build operator operator-sdk build quay.io/ $QUAY_USERNAME /$OP_NAME:latest --image-builder podman # push operator images podman push quay.io/ $QUAY_USERNAME /$OP_NAME:latest

Slide 71

Slide 71 text

#devconf_cz @jewzaam / @thedoh Deploy! 71 sed -i "s|REPLACE_IMAGE|quay.io/ $QUAY_USERNAME /$OP_NAME:demo|g" deploy/operator.yaml kubectl create namespace pod-operator kubectl create -f deploy/ -f deploy/crds/

Slide 72

Slide 72 text

#devconf_cz @jewzaam / @thedoh Make a Request 72 # create the PodRequest cat << EOF > podrequest.yaml apiVersion: pod.jewzaam.org/v1alpha1 kind: PodRequest metadata: name: busybox namespace: pod-operator spec: name: busybox image: busybox command: [ "/bin/sh", "-ec", "while :; do echo 'Hello DevConf.CZ 2020!'; sleep 5; done" ] EOF kubectl create -f podrequest.yaml # verify it’s doing what we expected kubectl logs busybox -f

Slide 73

Slide 73 text

#devconf_cz @jewzaam / @thedoh Delete the Request 73 # delete and verify it's gone kubectl delete podrequest busybox # verify it's gone (or terminating) kubectl get pods -l app=busybox

Slide 74

Slide 74 text

#devconf_cz @jewzaam / @thedoh Distributing Operator Based Services 74

Slide 75

Slide 75 text

#devconf_cz @jewzaam / @thedoh Distribution Venues 75 ● Raw manifest files, eg in Github ● Operator Lifecycle Manager (OLM) ● OperatorHub.io ● ...

Slide 76

Slide 76 text

#devconf_cz @jewzaam / @thedoh Bringing it all Together 76

Slide 77

Slide 77 text

#devconf_cz @jewzaam / @thedoh Operators do Work In the Cluster 77

Slide 78

Slide 78 text

#devconf_cz @jewzaam / @thedoh The End 78 Slide Deck: Source Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator

Slide 79

Slide 79 text

#devconf_cz @jewzaam / @thedoh What questions can we answer? 79 Slide Deck: Source Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator