Implementing Microservices as Kubernetes Operators

Kubernetes provides a rich framework for developers that enable the creation and management of microservices with relatively little code. By implementing operators as a service [1][2], the developer will benefit from the existing underlying Kubernetes conventions in addition to the highly available elements of the cluster. The resulting service will be accessible via a highly available API, backed by a replicated data store, and have built in user management and authorization. This REST API is capable of implementing all CRUD operations. Audience members will gain a new perspective of operators and their uses.

[1] https://developers.redhat.com/blog/2018/12/18/kubernetes-operators-in-depth/
[2] https://github.com/operator-framework/operator-sdk

Key takeaways:
* How using operators provides the benefits of the platform: RBAC, HA, DR
* Operators are powerful utilities to help configure administer Openshift
* Using Openshift as an API provides a rich framework to the developer.

Transcript

1. #devconf_cz @jewzaam / @thedoh Implementing Microservices as Kubernetes Operators 1 Naveen Malik, Sr. Principal SRE Lisa Seelye, Sr. SRE
2. #devconf_cz @jewzaam / @thedoh You will leave with... 2 • High level understanding Kubernetes • Understand why Operators are so great • Why we believe Operators are Microservices • How to create, build, deploy and test a new Operator
3. #devconf_cz @jewzaam / @thedoh Naveen Malik (@jewzaam) Senior Principal Site Reliability Engineer 1. Software Engineer / Architect 2. Father 3. Runner 4. Maker About Your Speakers 3 Lisa Seelye (@thedoh) Senior Site Reliability Engineer 1. Sysadmin & Sw Eng background 2. Canadian-in-training 3. Alternate arch fan 4. Cat enthusiast
4. #devconf_cz @jewzaam / @thedoh 4
5. #devconf_cz @jewzaam / @thedoh What is Kubernetes? 5 • Open-source platform • Manages containerized workloads • Facilitates declarative configuration & automation • Provides standard Resources to express application deployments • Supports custom Resources and logic around those Resources Ref https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
6. #devconf_cz @jewzaam / @thedoh Why are we here? 6 • OpenShift SRE team operates OpenShift Dedicated • Operators for all the things! OpenShift? • Flavour of Kubernetes • OpenShift is to Kubernetes as RHEL is to Linux OpenShift Dedicated? • High availability, multi-AZ OpenShift clusters on AWS • Deploy clusters into our AWS accounts or bring your own • Fully managed and supported by Red Hat, 24x7, with 99.5% uptime SLA
7. #devconf_cz @jewzaam / @thedoh "world map" is licensed under CC0 1.0 7 5 2 1 2 2 SRE is hiring! Stop by at Red Hat booth to find out more! (January 15, 2020)
8. #devconf_cz @jewzaam / @thedoh How Does Kubernetes Apply Changes? 8
9. #devconf_cz @jewzaam / @thedoh • Watches for change to resources • Makes it so How do Resources and Controllers interact? 9 • Use YAML to show desired state • Controllers make it happen • Examples (partial): ◦ Deployment ◦ Pod ◦ CustomResourceDefinition Stock Kubernetes Resources Stock Kubernetes Controllers
10. #devconf_cz @jewzaam / @thedoh What are Operators? • Custom behavior for existing resources • Custom behavior for custom resources • Move current state closer to desired state 10 • They don’t ship with Kubernetes • Models a specific application domain • Anything is possible! Custom Resource Definitions Custom Kubernetes Controllers
11. #devconf_cz @jewzaam / @thedoh 11 Deployment Controller Waiting for changes
12. #devconf_cz @jewzaam / @thedoh 12 Deployment Controller Tries to create Pods New Deployment Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ...
13. #devconf_cz @jewzaam / @thedoh 13 Deployment Controller Error detected! New Deployment kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ^ # error! ---------/ ...
14. #devconf_cz @jewzaam / @thedoh 14 Deployment Controller Success! New Deployment Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7.9 ...
15. #devconf_cz @jewzaam / @thedoh 15 Deployment Controller Waiting for changes
16. #devconf_cz @jewzaam / @thedoh 16 What Does it Look Like for Operators?
17. #devconf_cz @jewzaam / @thedoh 17 Custom Controller Waiting for changes
18. #devconf_cz @jewzaam / @thedoh Why Implement Your Microservice as an Operator? 18
19. #devconf_cz @jewzaam / @thedoh What are Microservices? 19 • Small with bounded contexts • Communicate over a network • Use technology-agnostic protocols • Organized around business capabilities • Independently deployable In addition, usually... • Security • Availability • Scalability Processes with... Read More: https://en.wikipedia.org/wiki/Microservices
20. #devconf_cz @jewzaam / @thedoh What are Microservices? • Have a narrow scope • Communicate over platform’s network • provide an API via Kubernetes resources • Group controllers based on business need • Are a deployable unit Kubernetes provides… • Authentication & Authorization • Highly available platform & deployments • Scale deployment to meet needs 20 • Small with bounded contexts • Communicate over a network • Use technology-agnostic protocols • Organized around business capabilities • Independently deployable In addition, usually... • Security • Availability • Scalability Processes with... Kubernetes Operators... Read More: https://en.wikipedia.org/wiki/Microservices
21. #devconf_cz @jewzaam / @thedoh Operators are Microservices 21
22. #devconf_cz @jewzaam / @thedoh Using Microservices for Config Management 22
23. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 23 Configuration Inventory Apply Changes Cluster 1
24. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 24 Configuration Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3
25. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 25 Configuration Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...
26. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 26 Configuration Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...
27. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 27 Configuration Inventory Apply Changes Cluster 1 Desired State
28. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 28 Configuration Inventory Apply Changes Cluster 1 Controller Desired State Realized State
29. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 29 Configuration Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...
30. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 30 Configuration Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...
31. #devconf_cz @jewzaam / @thedoh Systems Administration as Code 31
32. #devconf_cz @jewzaam / @thedoh Operators, by Example 32
33. #devconf_cz @jewzaam / @thedoh Operator Example: Install & Configure 33 cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator
34. #devconf_cz @jewzaam / @thedoh Installing & Configuring Software, Without Operators 34 • Install Prometheus, Alertmanager, and Grafana by hand ◦ Did you make them highly available? • Install kube-state-metrics on your own • Install node_exporter on your own • Write all the “glue” to configure Prometheus, Alertmanager and Grafana. Don’t forget, Deployments don’t watch ConfigMaps automatically
35. #devconf_cz @jewzaam / @thedoh • Installs & Configures ◦ Prometheus-operator (Prometheus and Alertmanager) ◦ Grafana ◦ Kube-state-metrics ◦ Node_exporter • Prometheus-operator custom resources ◦ alerting rules ◦ configuration Cluster-monitoring-operator, Operator Value 35
36. #devconf_cz @jewzaam / @thedoh apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring Cluster-monitoring-operator, Custom Resource 36
37. #devconf_cz @jewzaam / @thedoh Cluster-monitoring-operator, Custom Resource 37 apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring spec: groups: - name: sre-node-unschedulable rules: - alert: KubeNodeUnschedulableSRE expr: kube_node_spec_unschedulable > 0 for: 1h labels: severity: warning namespace: openshift-monitoring ...
38. #devconf_cz @jewzaam / @thedoh Operator Example: Provision Credentials 38 cloud-credential-operator https://github.com/openshift/cloud-credential-operator
39. #devconf_cz @jewzaam / @thedoh AWS Account Credentials, Without Operators 39 • Send a request to IT to get credentials • They validate your privilege and access, and then send to you, encrypted • You keep them secure somehow • Put them into Secret object • Figure out how to scale this across x clusters, y users, z cloud providers
40. #devconf_cz @jewzaam / @thedoh • User creates CredentialsRequest object ◦ Operator watches for it • Operator talks to Amazon to create credential pair • Operator stores results in a Secret • User accesses the Secret How cloud-credential-operator mints credentials 40
41. #devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf Cloud-credential-operator CredentialsRequest 41
42. #devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf Cloud-credential-operator CredentialsRequest 42
43. #devconf_cz @jewzaam / @thedoh Cloud-credential-operator CredentialsRequest 43 apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - ec2:DescribeInstances resource: "*"
44. #devconf_cz @jewzaam / @thedoh Cloud-credential-operator Resulting Secret 44 apiVersion: v1 kind: Secret metadata: name: secret-to-store-credentials namespace: devconf type: Opaque data: aws_secret_access_key: bm90IGEgcmVhbCBzZWNyZXQK aws_access_key_id: REVGSU5JVEVMWSBub3QgYSByZWFsIHNlY3JldAo=
45. #devconf_cz @jewzaam / @thedoh Operator Example: Manage Certificates 45 service-ca-operator https://github.com/openshift/service-ca-operator
46. #devconf_cz @jewzaam / @thedoh Service-ca-operator, Without Operators 46 • Connect to the appropriate cluster and create a CSR • kubectl certificate approve a CertificateSigningRequest • Keep secrets safe • Figure out how to scale over n clusters
47. #devconf_cz @jewzaam / @thedoh • Native objects are annotated • Operator’s controllers watch for them • Creates certificates and stores in a Secret How service-ca-operator signs certificates 47
48. #devconf_cz @jewzaam / @thedoh apiVersion: v1 kind: Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name : "devconf-service-certs" Service-ca-operator Details 48
49. #devconf_cz @jewzaam / @thedoh Service-ca-operator Details 49 apiVersion: v1 kind: Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ...
50. #devconf_cz @jewzaam / @thedoh Service-ca-operator Details 50 apiVersion: v1 kind: Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ... --- apiVersion: v1 kind: ConfigMap metadata: annotations: service.beta.openshift.io/inject-cabundle : "true" name: "devconf-service-cabundle " data: {} # .service-ca.crt added by operator
51. #devconf_cz @jewzaam / @thedoh Operators Increase Velocity, Scalability, Availability, ... 51
52. #devconf_cz @jewzaam / @thedoh Operator Demo: Create Pod 52 pod-operator (contrived example)
53. #devconf_cz @jewzaam / @thedoh Why create this operator? 53 Show how to… 1. Create a golang operator 2. Create a Custom Resource Definition (CRD) 3. Create a Custom Controller
54. #devconf_cz @jewzaam / @thedoh What does the Operator do? 54 Create Pod from a Custom Resource Ensure Pod exists for each Custom Resource Cleanup Pod when Custom Resource is deleted
55. #devconf_cz @jewzaam / @thedoh Tools 55 • operator-sdk ◦ software development toolkit (SDK) for building Kubernetes applications • podman ◦ daemonless container engine for developing, managing, and running containers • kubectl ◦ command line tool for controlling Kubernetes clusters
56. #devconf_cz @jewzaam / @thedoh How is the Operator created? 56
57. #devconf_cz @jewzaam / @thedoh How is the Operator created? 57 1. operator-sdk new pod-operator
58. #devconf_cz @jewzaam / @thedoh How is the Operator created? 58 1. operator-sdk new 2. operator-sdk add api pod-operator PodRequest (CRD) --- spec:
59. #devconf_cz @jewzaam / @thedoh How is the Operator created? 59 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller pod-operator PodRequest (CRD) --- spec: Controller --- - watch PodRequest & Pod - assign Owner - create Pod
60. #devconf_cz @jewzaam / @thedoh How is the Operator created? 60 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod
61. #devconf_cz @jewzaam / @thedoh How is the Operator created? 61 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io
62. #devconf_cz @jewzaam / @thedoh How is the Operator created? 62 1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push 7. kubectl create pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io
63. #devconf_cz @jewzaam / @thedoh What are we testing? 63 Create Pod from a Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
64. #devconf_cz @jewzaam / @thedoh What are we testing? 64 Ensure Pod exists for each Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
65. #devconf_cz @jewzaam / @thedoh What are we testing? 65 Cleanup Pod when Custom Resource is deleted pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
66. #devconf_cz @jewzaam / @thedoh Demo Time! 66
67. #devconf_cz @jewzaam / @thedoh Bootstrapping an operator from scratch! 67 export GO111MODULE=on export QUAY_USERNAME=nmalik export GITHUB_USERNAME=jewzaam export OP_VERSION=0.0.1 export OP_NAME=pod-operator export OP_API_GROUP=pod.jewzaam.org export OP_API_VERSION=v1alpha1 export OP_KIND=PodRequest # bootstrap the operator mkdir -p $GOPATH/src/github.com/ $GITHUB_USERNAME cd $GOPATH/src/github.com/ $GITHUB_USERNAME operator-sdk new $OP_NAME cd $OP_NAME # generate CRD and Controller operator-sdk add api --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND operator-sdk add controller --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND
68. #devconf_cz @jewzaam / @thedoh Customize the API 68 ... type PodRequestSpec struct { Name string `json:"name"` Image string `json:"image"` Command []string `json:"command"` } ...
69. #devconf_cz @jewzaam / @thedoh Customize the Controller 69 func newPodForCR(cr *podv1alpha1.PodRequest) *corev1.Pod { labels := map[string]string{ "app": cr.Name, } return &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: cr.Name, Namespace: cr.Namespace, Labels: labels, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: cr.Spec.Name, Image: cr.Spec.Image, Command: cr.Spec.Command, }, }, }, } }
70. #devconf_cz @jewzaam / @thedoh Build and Publish: Operator 70 # build operator operator-sdk build quay.io/ $QUAY_USERNAME /$OP_NAME:latest --image-builder podman # push operator images podman push quay.io/ $QUAY_USERNAME /$OP_NAME:latest
71. #devconf_cz @jewzaam / @thedoh Deploy! 71 sed -i "s|REPLACE_IMAGE|quay.io/ $QUAY_USERNAME /$OP_NAME:demo|g" deploy/operator.yaml kubectl create namespace pod-operator kubectl create -f deploy/ -f deploy/crds/
72. #devconf_cz @jewzaam / @thedoh Make a Request 72 # create the PodRequest cat << EOF > podrequest.yaml apiVersion: pod.jewzaam.org/v1alpha1 kind: PodRequest metadata: name: busybox namespace: pod-operator spec: name: busybox image: busybox command: [ "/bin/sh", "-ec", "while :; do echo 'Hello DevConf.CZ 2020!'; sleep 5; done" ] EOF kubectl create -f podrequest.yaml # verify it’s doing what we expected kubectl logs busybox -f
73. #devconf_cz @jewzaam / @thedoh Delete the Request 73 # delete and verify it's gone kubectl delete podrequest busybox # verify it's gone (or terminating) kubectl get pods -l app=busybox
74. #devconf_cz @jewzaam / @thedoh Distributing Operator Based Services 74
75. #devconf_cz @jewzaam / @thedoh Distribution Venues 75 • Raw manifest files, eg in Github • Operator Lifecycle Manager (OLM) • OperatorHub.io • ...
76. #devconf_cz @jewzaam / @thedoh Bringing it all Together 76
77. #devconf_cz @jewzaam / @thedoh Operators do Work In the Cluster 77
78. #devconf_cz @jewzaam / @thedoh The End 78 Slide Deck: Source Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator
79. #devconf_cz @jewzaam / @thedoh What questions can we answer? 79 Slide Deck: Source Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator