Implementing Microservices as Kubernetes Operators

Implementing Microservices as Kubernetes Operators

Kubernetes provides a rich framework for developers that enable the creation and management of microservices with relatively little code. By implementing operators as a service [1][2], the developer will benefit from the existing underlying Kubernetes conventions in addition to the highly available elements of the cluster. The resulting service will be accessible via a highly available API, backed by a replicated data store, and have built in user management and authorization. This REST API is capable of implementing all CRUD operations. Audience members will gain a new perspective of operators and their uses.

[1] https://developers.redhat.com/blog/2018/12/18/kubernetes-operators-in-depth/
[2] https://github.com/operator-framework/operator-sdk

Key takeaways:
* How using operators provides the benefits of the platform: RBAC, HA, DR
* Operators are powerful utilities to help configure administer Openshift
* Using Openshift as an API provides a rich framework to the developer.

9067acc5914589a6cb467b298344e106?s=128

Naveen Malik

January 26, 2020
Tweet

Transcript

  1. #devconf_cz @jewzaam / @thedoh Implementing Microservices as Kubernetes Operators 1

    Naveen Malik, Sr. Principal SRE Lisa Seelye, Sr. SRE
  2. #devconf_cz @jewzaam / @thedoh You will leave with... 2 •

    High level understanding Kubernetes • Understand why Operators are so great • Why we believe Operators are Microservices • How to create, build, deploy and test a new Operator
  3. #devconf_cz @jewzaam / @thedoh Naveen Malik (@jewzaam) Senior Principal Site

    Reliability Engineer 1. Software Engineer / Architect 2. Father 3. Runner 4. Maker About Your Speakers 3 Lisa Seelye (@thedoh) Senior Site Reliability Engineer 1. Sysadmin & Sw Eng background 2. Canadian-in-training 3. Alternate arch fan 4. Cat enthusiast
  4. #devconf_cz @jewzaam / @thedoh 4

  5. #devconf_cz @jewzaam / @thedoh What is Kubernetes? 5 • Open-source

    platform • Manages containerized workloads • Facilitates declarative configuration & automation • Provides standard Resources to express application deployments • Supports custom Resources and logic around those Resources Ref https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
  6. #devconf_cz @jewzaam / @thedoh Why are we here? 6 •

    OpenShift SRE team operates OpenShift Dedicated • Operators for all the things! OpenShift? • Flavour of Kubernetes • OpenShift is to Kubernetes as RHEL is to Linux OpenShift Dedicated? • High availability, multi-AZ OpenShift clusters on AWS • Deploy clusters into our AWS accounts or bring your own • Fully managed and supported by Red Hat, 24x7, with 99.5% uptime SLA
  7. #devconf_cz @jewzaam / @thedoh "world map" is licensed under CC0

    1.0 7 5 2 1 2 2 SRE is hiring! Stop by at Red Hat booth to find out more! (January 15, 2020)
  8. #devconf_cz @jewzaam / @thedoh How Does Kubernetes Apply Changes? 8

  9. #devconf_cz @jewzaam / @thedoh • Watches for change to resources

    • Makes it so How do Resources and Controllers interact? 9 • Use YAML to show desired state • Controllers make it happen • Examples (partial): ◦ Deployment ◦ Pod ◦ CustomResourceDefinition Stock Kubernetes Resources Stock Kubernetes Controllers
  10. #devconf_cz @jewzaam / @thedoh What are Operators? • Custom behavior

    for existing resources • Custom behavior for custom resources • Move current state closer to desired state 10 • They don’t ship with Kubernetes • Models a specific application domain • Anything is possible! Custom Resource Definitions Custom Kubernetes Controllers
  11. #devconf_cz @jewzaam / @thedoh 11 Deployment Controller Waiting for changes

  12. #devconf_cz @jewzaam / @thedoh 12 Deployment Controller Tries to create

    Pods New Deployment Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ...
  13. #devconf_cz @jewzaam / @thedoh 13 Deployment Controller Error detected! New

    Deployment kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ^ # error! ---------/ ...
  14. #devconf_cz @jewzaam / @thedoh 14 Deployment Controller Success! New Deployment

    Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7.9 ...
  15. #devconf_cz @jewzaam / @thedoh 15 Deployment Controller Waiting for changes

  16. #devconf_cz @jewzaam / @thedoh 16 What Does it Look Like

    for Operators?
  17. #devconf_cz @jewzaam / @thedoh 17 Custom Controller Waiting for changes

  18. #devconf_cz @jewzaam / @thedoh Why Implement Your Microservice as an

    Operator? 18
  19. #devconf_cz @jewzaam / @thedoh What are Microservices? 19 • Small

    with bounded contexts • Communicate over a network • Use technology-agnostic protocols • Organized around business capabilities • Independently deployable In addition, usually... • Security • Availability • Scalability Processes with... Read More: https://en.wikipedia.org/wiki/Microservices
  20. #devconf_cz @jewzaam / @thedoh What are Microservices? • Have a

    narrow scope • Communicate over platform’s network • provide an API via Kubernetes resources • Group controllers based on business need • Are a deployable unit Kubernetes provides… • Authentication & Authorization • Highly available platform & deployments • Scale deployment to meet needs 20 • Small with bounded contexts • Communicate over a network • Use technology-agnostic protocols • Organized around business capabilities • Independently deployable In addition, usually... • Security • Availability • Scalability Processes with... Kubernetes Operators... Read More: https://en.wikipedia.org/wiki/Microservices
  21. #devconf_cz @jewzaam / @thedoh Operators are Microservices 21

  22. #devconf_cz @jewzaam / @thedoh Using Microservices for Config Management 22

  23. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 23 Configuration

    Inventory Apply Changes Cluster 1
  24. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 24 Configuration

    Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3
  25. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 25 Configuration

    Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...
  26. #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 26 Configuration

    Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...
  27. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 27 Configuration

    Inventory Apply Changes Cluster 1 Desired State
  28. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 28 Configuration

    Inventory Apply Changes Cluster 1 Controller Desired State Realized State
  29. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 29 Configuration

    Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...
  30. #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 30 Configuration

    Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...
  31. #devconf_cz @jewzaam / @thedoh Systems Administration as Code 31

  32. #devconf_cz @jewzaam / @thedoh Operators, by Example 32

  33. #devconf_cz @jewzaam / @thedoh Operator Example: Install & Configure 33

    cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator
  34. #devconf_cz @jewzaam / @thedoh Installing & Configuring Software, Without Operators

    34 • Install Prometheus, Alertmanager, and Grafana by hand ◦ Did you make them highly available? • Install kube-state-metrics on your own • Install node_exporter on your own • Write all the “glue” to configure Prometheus, Alertmanager and Grafana. Don’t forget, Deployments don’t watch ConfigMaps automatically
  35. #devconf_cz @jewzaam / @thedoh • Installs & Configures ◦ Prometheus-operator

    (Prometheus and Alertmanager) ◦ Grafana ◦ Kube-state-metrics ◦ Node_exporter • Prometheus-operator custom resources ◦ alerting rules ◦ configuration Cluster-monitoring-operator, Operator Value 35
  36. #devconf_cz @jewzaam / @thedoh apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels:

    prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring Cluster-monitoring-operator, Custom Resource 36
  37. #devconf_cz @jewzaam / @thedoh Cluster-monitoring-operator, Custom Resource 37 apiVersion: monitoring.coreos.com/v1

    kind: PrometheusRule metadata: labels: prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring spec: groups: - name: sre-node-unschedulable rules: - alert: KubeNodeUnschedulableSRE expr: kube_node_spec_unschedulable > 0 for: 1h labels: severity: warning namespace: openshift-monitoring ...
  38. #devconf_cz @jewzaam / @thedoh Operator Example: Provision Credentials 38 cloud-credential-operator

    https://github.com/openshift/cloud-credential-operator
  39. #devconf_cz @jewzaam / @thedoh AWS Account Credentials, Without Operators 39

    • Send a request to IT to get credentials • They validate your privilege and access, and then send to you, encrypted • You keep them secure somehow • Put them into Secret object • Figure out how to scale this across x clusters, y users, z cloud providers
  40. #devconf_cz @jewzaam / @thedoh • User creates CredentialsRequest object ◦

    Operator watches for it • Operator talks to Amazon to create credential pair • Operator stores results in a Secret • User accesses the Secret How cloud-credential-operator mints credentials 40
  41. #devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name:

    aws-creds-to-describe-all-instances namespace: devconf Cloud-credential-operator CredentialsRequest 41
  42. #devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name:

    aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf Cloud-credential-operator CredentialsRequest 42
  43. #devconf_cz @jewzaam / @thedoh Cloud-credential-operator CredentialsRequest 43 apiVersion: cloudcredential.openshift.io/v1 kind:

    CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - ec2:DescribeInstances resource: "*"
  44. #devconf_cz @jewzaam / @thedoh Cloud-credential-operator Resulting Secret 44 apiVersion: v1

    kind: Secret metadata: name: secret-to-store-credentials namespace: devconf type: Opaque data: aws_secret_access_key: bm90IGEgcmVhbCBzZWNyZXQK aws_access_key_id: REVGSU5JVEVMWSBub3QgYSByZWFsIHNlY3JldAo=
  45. #devconf_cz @jewzaam / @thedoh Operator Example: Manage Certificates 45 service-ca-operator

    https://github.com/openshift/service-ca-operator
  46. #devconf_cz @jewzaam / @thedoh Service-ca-operator, Without Operators 46 • Connect

    to the appropriate cluster and create a CSR • kubectl certificate approve a CertificateSigningRequest • Keep secrets safe • Figure out how to scale over n clusters
  47. #devconf_cz @jewzaam / @thedoh • Native objects are annotated •

    Operator’s controllers watch for them • Creates certificates and stores in a Secret How service-ca-operator signs certificates 47
  48. #devconf_cz @jewzaam / @thedoh apiVersion: v1 kind: Service metadata: name:

    "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name : "devconf-service-certs" Service-ca-operator Details 48
  49. #devconf_cz @jewzaam / @thedoh Service-ca-operator Details 49 apiVersion: v1 kind:

    Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ...
  50. #devconf_cz @jewzaam / @thedoh Service-ca-operator Details 50 apiVersion: v1 kind:

    Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ... --- apiVersion: v1 kind: ConfigMap metadata: annotations: service.beta.openshift.io/inject-cabundle : "true" name: "devconf-service-cabundle " data: {} # .service-ca.crt added by operator
  51. #devconf_cz @jewzaam / @thedoh Operators Increase Velocity, Scalability, Availability, ...

    51
  52. #devconf_cz @jewzaam / @thedoh Operator Demo: Create Pod 52 pod-operator

    (contrived example)
  53. #devconf_cz @jewzaam / @thedoh Why create this operator? 53 Show

    how to… 1. Create a golang operator 2. Create a Custom Resource Definition (CRD) 3. Create a Custom Controller
  54. #devconf_cz @jewzaam / @thedoh What does the Operator do? 54

    Create Pod from a Custom Resource Ensure Pod exists for each Custom Resource Cleanup Pod when Custom Resource is deleted
  55. #devconf_cz @jewzaam / @thedoh Tools 55 • operator-sdk ◦ software

    development toolkit (SDK) for building Kubernetes applications • podman ◦ daemonless container engine for developing, managing, and running containers • kubectl ◦ command line tool for controlling Kubernetes clusters
  56. #devconf_cz @jewzaam / @thedoh How is the Operator created? 56

  57. #devconf_cz @jewzaam / @thedoh How is the Operator created? 57

    1. operator-sdk new pod-operator
  58. #devconf_cz @jewzaam / @thedoh How is the Operator created? 58

    1. operator-sdk new 2. operator-sdk add api pod-operator PodRequest (CRD) --- spec:
  59. #devconf_cz @jewzaam / @thedoh How is the Operator created? 59

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller pod-operator PodRequest (CRD) --- spec: Controller --- - watch PodRequest & Pod - assign Owner - create Pod
  60. #devconf_cz @jewzaam / @thedoh How is the Operator created? 60

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod
  61. #devconf_cz @jewzaam / @thedoh How is the Operator created? 61

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io
  62. #devconf_cz @jewzaam / @thedoh How is the Operator created? 62

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push 7. kubectl create pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io
  63. #devconf_cz @jewzaam / @thedoh What are we testing? 63 Create

    Pod from a Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
  64. #devconf_cz @jewzaam / @thedoh What are we testing? 64 Ensure

    Pod exists for each Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
  65. #devconf_cz @jewzaam / @thedoh What are we testing? 65 Cleanup

    Pod when Custom Resource is deleted pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
  66. #devconf_cz @jewzaam / @thedoh Demo Time! 66

  67. #devconf_cz @jewzaam / @thedoh Bootstrapping an operator from scratch! 67

    export GO111MODULE=on export QUAY_USERNAME=nmalik export GITHUB_USERNAME=jewzaam export OP_VERSION=0.0.1 export OP_NAME=pod-operator export OP_API_GROUP=pod.jewzaam.org export OP_API_VERSION=v1alpha1 export OP_KIND=PodRequest # bootstrap the operator mkdir -p $GOPATH/src/github.com/ $GITHUB_USERNAME cd $GOPATH/src/github.com/ $GITHUB_USERNAME operator-sdk new $OP_NAME cd $OP_NAME # generate CRD and Controller operator-sdk add api --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND operator-sdk add controller --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND
  68. #devconf_cz @jewzaam / @thedoh Customize the API 68 ... type

    PodRequestSpec struct { Name string `json:"name"` Image string `json:"image"` Command []string `json:"command"` } ...
  69. #devconf_cz @jewzaam / @thedoh Customize the Controller 69 func newPodForCR(cr

    *podv1alpha1.PodRequest) *corev1.Pod { labels := map[string]string{ "app": cr.Name, } return &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: cr.Name, Namespace: cr.Namespace, Labels: labels, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: cr.Spec.Name, Image: cr.Spec.Image, Command: cr.Spec.Command, }, }, }, } }
  70. #devconf_cz @jewzaam / @thedoh Build and Publish: Operator 70 #

    build operator operator-sdk build quay.io/ $QUAY_USERNAME /$OP_NAME:latest --image-builder podman # push operator images podman push quay.io/ $QUAY_USERNAME /$OP_NAME:latest
  71. #devconf_cz @jewzaam / @thedoh Deploy! 71 sed -i "s|REPLACE_IMAGE|quay.io/ $QUAY_USERNAME

    /$OP_NAME:demo|g" deploy/operator.yaml kubectl create namespace pod-operator kubectl create -f deploy/ -f deploy/crds/
  72. #devconf_cz @jewzaam / @thedoh Make a Request 72 # create

    the PodRequest cat << EOF > podrequest.yaml apiVersion: pod.jewzaam.org/v1alpha1 kind: PodRequest metadata: name: busybox namespace: pod-operator spec: name: busybox image: busybox command: [ "/bin/sh", "-ec", "while :; do echo 'Hello DevConf.CZ 2020!'; sleep 5; done" ] EOF kubectl create -f podrequest.yaml # verify it’s doing what we expected kubectl logs busybox -f
  73. #devconf_cz @jewzaam / @thedoh Delete the Request 73 # delete

    and verify it's gone kubectl delete podrequest busybox # verify it's gone (or terminating) kubectl get pods -l app=busybox
  74. #devconf_cz @jewzaam / @thedoh Distributing Operator Based Services 74

  75. #devconf_cz @jewzaam / @thedoh Distribution Venues 75 • Raw manifest

    files, eg in Github • Operator Lifecycle Manager (OLM) • OperatorHub.io • ...
  76. #devconf_cz @jewzaam / @thedoh Bringing it all Together 76

  77. #devconf_cz @jewzaam / @thedoh Operators do Work In the Cluster

    77
  78. #devconf_cz @jewzaam / @thedoh The End 78 Slide Deck: Source

    Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator
  79. #devconf_cz @jewzaam / @thedoh What questions can we answer? 79

    Slide Deck: Source Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator