Implementing Microservices as Kubernetes Operators

Implementing Microservices as Kubernetes Operators

Kubernetes provides a rich framework for developers that enable the creation and management of microservices with relatively little code. By implementing operators as a service [1][2], the developer will benefit from the existing underlying Kubernetes conventions in addition to the highly available elements of the cluster. The resulting service will be accessible via a highly available API, backed by a replicated data store, and have built in user management and authorization. This REST API is capable of implementing all CRUD operations. Audience members will gain a new perspective of operators and their uses.

[1] https://developers.redhat.com/blog/2018/12/18/kubernetes-operators-in-depth/
[2] https://github.com/operator-framework/operator-sdk

Key takeaways:
* How using operators provides the benefits of the platform: RBAC, HA, DR
* Operators are powerful utilities to help configure administer Openshift
* Using Openshift as an API provides a rich framework to the developer.

9067acc5914589a6cb467b298344e106?s=128

Naveen Malik

January 26, 2020
Tweet

Transcript

  1. 1.
  2. 2.

    #devconf_cz @jewzaam / @thedoh You will leave with... 2 •

    High level understanding Kubernetes • Understand why Operators are so great • Why we believe Operators are Microservices • How to create, build, deploy and test a new Operator
  3. 3.

    #devconf_cz @jewzaam / @thedoh Naveen Malik (@jewzaam) Senior Principal Site

    Reliability Engineer 1. Software Engineer / Architect 2. Father 3. Runner 4. Maker About Your Speakers 3 Lisa Seelye (@thedoh) Senior Site Reliability Engineer 1. Sysadmin & Sw Eng background 2. Canadian-in-training 3. Alternate arch fan 4. Cat enthusiast
  4. 5.

    #devconf_cz @jewzaam / @thedoh What is Kubernetes? 5 • Open-source

    platform • Manages containerized workloads • Facilitates declarative configuration & automation • Provides standard Resources to express application deployments • Supports custom Resources and logic around those Resources Ref https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
  5. 6.

    #devconf_cz @jewzaam / @thedoh Why are we here? 6 •

    OpenShift SRE team operates OpenShift Dedicated • Operators for all the things! OpenShift? • Flavour of Kubernetes • OpenShift is to Kubernetes as RHEL is to Linux OpenShift Dedicated? • High availability, multi-AZ OpenShift clusters on AWS • Deploy clusters into our AWS accounts or bring your own • Fully managed and supported by Red Hat, 24x7, with 99.5% uptime SLA
  6. 7.

    #devconf_cz @jewzaam / @thedoh "world map" is licensed under CC0

    1.0 7 5 2 1 2 2 SRE is hiring! Stop by at Red Hat booth to find out more! (January 15, 2020)
  7. 9.

    #devconf_cz @jewzaam / @thedoh • Watches for change to resources

    • Makes it so How do Resources and Controllers interact? 9 • Use YAML to show desired state • Controllers make it happen • Examples (partial): ◦ Deployment ◦ Pod ◦ CustomResourceDefinition Stock Kubernetes Resources Stock Kubernetes Controllers
  8. 10.

    #devconf_cz @jewzaam / @thedoh What are Operators? • Custom behavior

    for existing resources • Custom behavior for custom resources • Move current state closer to desired state 10 • They don’t ship with Kubernetes • Models a specific application domain • Anything is possible! Custom Resource Definitions Custom Kubernetes Controllers
  9. 12.

    #devconf_cz @jewzaam / @thedoh 12 Deployment Controller Tries to create

    Pods New Deployment Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ...
  10. 13.

    #devconf_cz @jewzaam / @thedoh 13 Deployment Controller Error detected! New

    Deployment kind: Deployment spec: replicas: 3 ... image: nginx:1.7,9 ^ # error! ---------/ ...
  11. 14.

    #devconf_cz @jewzaam / @thedoh 14 Deployment Controller Success! New Deployment

    Pod Pod Pod kind: Deployment spec: replicas: 3 ... image: nginx:1.7.9 ...
  12. 19.

    #devconf_cz @jewzaam / @thedoh What are Microservices? 19 • Small

    with bounded contexts • Communicate over a network • Use technology-agnostic protocols • Organized around business capabilities • Independently deployable In addition, usually... • Security • Availability • Scalability Processes with... Read More: https://en.wikipedia.org/wiki/Microservices
  13. 20.

    #devconf_cz @jewzaam / @thedoh What are Microservices? • Have a

    narrow scope • Communicate over platform’s network • provide an API via Kubernetes resources • Group controllers based on business need • Are a deployable unit Kubernetes provides… • Authentication & Authorization • Highly available platform & deployments • Scale deployment to meet needs 20 • Small with bounded contexts • Communicate over a network • Use technology-agnostic protocols • Organized around business capabilities • Independently deployable In addition, usually... • Security • Availability • Scalability Processes with... Kubernetes Operators... Read More: https://en.wikipedia.org/wiki/Microservices
  14. 24.
  15. 25.

    #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 25 Configuration

    Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...
  16. 26.

    #devconf_cz @jewzaam / @thedoh Monolith: Centralized Config Management 26 Configuration

    Inventory Apply Changes Cluster 1 Cluster 2 Cluster 3 Cluster N ...
  17. 28.

    #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 28 Configuration

    Inventory Apply Changes Cluster 1 Controller Desired State Realized State
  18. 29.

    #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 29 Configuration

    Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...
  19. 30.

    #devconf_cz @jewzaam / @thedoh Microservices: Distributed Config Management 30 Configuration

    Inventory Apply Changes Cluster 1 Controller Desired State Realized State Cluster 2 Controller Desired State Realized State Cluster N Controller Desired State Realized State ...
  20. 33.

    #devconf_cz @jewzaam / @thedoh Operator Example: Install & Configure 33

    cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator
  21. 34.

    #devconf_cz @jewzaam / @thedoh Installing & Configuring Software, Without Operators

    34 • Install Prometheus, Alertmanager, and Grafana by hand ◦ Did you make them highly available? • Install kube-state-metrics on your own • Install node_exporter on your own • Write all the “glue” to configure Prometheus, Alertmanager and Grafana. Don’t forget, Deployments don’t watch ConfigMaps automatically
  22. 35.

    #devconf_cz @jewzaam / @thedoh • Installs & Configures ◦ Prometheus-operator

    (Prometheus and Alertmanager) ◦ Grafana ◦ Kube-state-metrics ◦ Node_exporter • Prometheus-operator custom resources ◦ alerting rules ◦ configuration Cluster-monitoring-operator, Operator Value 35
  23. 36.

    #devconf_cz @jewzaam / @thedoh apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels:

    prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring Cluster-monitoring-operator, Custom Resource 36
  24. 37.

    #devconf_cz @jewzaam / @thedoh Cluster-monitoring-operator, Custom Resource 37 apiVersion: monitoring.coreos.com/v1

    kind: PrometheusRule metadata: labels: prometheus: sre-node-unschedulable role: alert-rules name: sre-node-unschedulable namespace: openshift-monitoring spec: groups: - name: sre-node-unschedulable rules: - alert: KubeNodeUnschedulableSRE expr: kube_node_spec_unschedulable > 0 for: 1h labels: severity: warning namespace: openshift-monitoring ...
  25. 39.

    #devconf_cz @jewzaam / @thedoh AWS Account Credentials, Without Operators 39

    • Send a request to IT to get credentials • They validate your privilege and access, and then send to you, encrypted • You keep them secure somehow • Put them into Secret object • Figure out how to scale this across x clusters, y users, z cloud providers
  26. 40.

    #devconf_cz @jewzaam / @thedoh • User creates CredentialsRequest object ◦

    Operator watches for it • Operator talks to Amazon to create credential pair • Operator stores results in a Secret • User accesses the Secret How cloud-credential-operator mints credentials 40
  27. 41.

    #devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name:

    aws-creds-to-describe-all-instances namespace: devconf Cloud-credential-operator CredentialsRequest 41
  28. 42.

    #devconf_cz @jewzaam / @thedoh apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name:

    aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf Cloud-credential-operator CredentialsRequest 42
  29. 43.

    #devconf_cz @jewzaam / @thedoh Cloud-credential-operator CredentialsRequest 43 apiVersion: cloudcredential.openshift.io/v1 kind:

    CredentialsRequest metadata: name: aws-creds-to-describe-all-instances namespace: devconf spec: secretRef: name: secret-to-store-credentials namespace: devconf providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - ec2:DescribeInstances resource: "*"
  30. 44.

    #devconf_cz @jewzaam / @thedoh Cloud-credential-operator Resulting Secret 44 apiVersion: v1

    kind: Secret metadata: name: secret-to-store-credentials namespace: devconf type: Opaque data: aws_secret_access_key: bm90IGEgcmVhbCBzZWNyZXQK aws_access_key_id: REVGSU5JVEVMWSBub3QgYSByZWFsIHNlY3JldAo=
  31. 46.

    #devconf_cz @jewzaam / @thedoh Service-ca-operator, Without Operators 46 • Connect

    to the appropriate cluster and create a CSR • kubectl certificate approve a CertificateSigningRequest • Keep secrets safe • Figure out how to scale over n clusters
  32. 47.

    #devconf_cz @jewzaam / @thedoh • Native objects are annotated •

    Operator’s controllers watch for them • Creates certificates and stores in a Secret How service-ca-operator signs certificates 47
  33. 48.

    #devconf_cz @jewzaam / @thedoh apiVersion: v1 kind: Service metadata: name:

    "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name : "devconf-service-certs" Service-ca-operator Details 48
  34. 49.

    #devconf_cz @jewzaam / @thedoh Service-ca-operator Details 49 apiVersion: v1 kind:

    Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ...
  35. 50.

    #devconf_cz @jewzaam / @thedoh Service-ca-operator Details 50 apiVersion: v1 kind:

    Service metadata: name: "devconf" annotations: service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs" --- apiVersion: v1 kind: Secret metadata: name: "devconf-service-certs" data: ... --- apiVersion: v1 kind: ConfigMap metadata: annotations: service.beta.openshift.io/inject-cabundle : "true" name: "devconf-service-cabundle " data: {} # .service-ca.crt added by operator
  36. 53.

    #devconf_cz @jewzaam / @thedoh Why create this operator? 53 Show

    how to… 1. Create a golang operator 2. Create a Custom Resource Definition (CRD) 3. Create a Custom Controller
  37. 54.

    #devconf_cz @jewzaam / @thedoh What does the Operator do? 54

    Create Pod from a Custom Resource Ensure Pod exists for each Custom Resource Cleanup Pod when Custom Resource is deleted
  38. 55.

    #devconf_cz @jewzaam / @thedoh Tools 55 • operator-sdk ◦ software

    development toolkit (SDK) for building Kubernetes applications • podman ◦ daemonless container engine for developing, managing, and running containers • kubectl ◦ command line tool for controlling Kubernetes clusters
  39. 58.

    #devconf_cz @jewzaam / @thedoh How is the Operator created? 58

    1. operator-sdk new 2. operator-sdk add api pod-operator PodRequest (CRD) --- spec:
  40. 59.

    #devconf_cz @jewzaam / @thedoh How is the Operator created? 59

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller pod-operator PodRequest (CRD) --- spec: Controller --- - watch PodRequest & Pod - assign Owner - create Pod
  41. 60.

    #devconf_cz @jewzaam / @thedoh How is the Operator created? 60

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod
  42. 61.

    #devconf_cz @jewzaam / @thedoh How is the Operator created? 61

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io
  43. 62.

    #devconf_cz @jewzaam / @thedoh How is the Operator created? 62

    1. operator-sdk new 2. operator-sdk add api 3. operator-sdk add controller 4. edit code 5. operator-sdk build 6. podman push 7. kubectl create pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod quay.io
  44. 63.

    #devconf_cz @jewzaam / @thedoh What are we testing? 63 Create

    Pod from a Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
  45. 64.

    #devconf_cz @jewzaam / @thedoh What are we testing? 64 Ensure

    Pod exists for each Custom Resource pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
  46. 65.

    #devconf_cz @jewzaam / @thedoh What are we testing? 65 Cleanup

    Pod when Custom Resource is deleted pod-operator PodRequest (CRD) --- spec: name: “busybox” image: “busybox” command: [“sleep 3600”] Controller --- - watch PodRequest & Pod - assign Owner - create Pod PodRequest (CR) Pod quay.io
  47. 67.

    #devconf_cz @jewzaam / @thedoh Bootstrapping an operator from scratch! 67

    export GO111MODULE=on export QUAY_USERNAME=nmalik export GITHUB_USERNAME=jewzaam export OP_VERSION=0.0.1 export OP_NAME=pod-operator export OP_API_GROUP=pod.jewzaam.org export OP_API_VERSION=v1alpha1 export OP_KIND=PodRequest # bootstrap the operator mkdir -p $GOPATH/src/github.com/ $GITHUB_USERNAME cd $GOPATH/src/github.com/ $GITHUB_USERNAME operator-sdk new $OP_NAME cd $OP_NAME # generate CRD and Controller operator-sdk add api --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND operator-sdk add controller --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND
  48. 68.

    #devconf_cz @jewzaam / @thedoh Customize the API 68 ... type

    PodRequestSpec struct { Name string `json:"name"` Image string `json:"image"` Command []string `json:"command"` } ...
  49. 69.

    #devconf_cz @jewzaam / @thedoh Customize the Controller 69 func newPodForCR(cr

    *podv1alpha1.PodRequest) *corev1.Pod { labels := map[string]string{ "app": cr.Name, } return &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: cr.Name, Namespace: cr.Namespace, Labels: labels, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: cr.Spec.Name, Image: cr.Spec.Image, Command: cr.Spec.Command, }, }, }, } }
  50. 70.

    #devconf_cz @jewzaam / @thedoh Build and Publish: Operator 70 #

    build operator operator-sdk build quay.io/ $QUAY_USERNAME /$OP_NAME:latest --image-builder podman # push operator images podman push quay.io/ $QUAY_USERNAME /$OP_NAME:latest
  51. 71.

    #devconf_cz @jewzaam / @thedoh Deploy! 71 sed -i "s|REPLACE_IMAGE|quay.io/ $QUAY_USERNAME

    /$OP_NAME:demo|g" deploy/operator.yaml kubectl create namespace pod-operator kubectl create -f deploy/ -f deploy/crds/
  52. 72.

    #devconf_cz @jewzaam / @thedoh Make a Request 72 # create

    the PodRequest cat << EOF > podrequest.yaml apiVersion: pod.jewzaam.org/v1alpha1 kind: PodRequest metadata: name: busybox namespace: pod-operator spec: name: busybox image: busybox command: [ "/bin/sh", "-ec", "while :; do echo 'Hello DevConf.CZ 2020!'; sleep 5; done" ] EOF kubectl create -f podrequest.yaml # verify it’s doing what we expected kubectl logs busybox -f
  53. 73.

    #devconf_cz @jewzaam / @thedoh Delete the Request 73 # delete

    and verify it's gone kubectl delete podrequest busybox # verify it's gone (or terminating) kubectl get pods -l app=busybox
  54. 75.

    #devconf_cz @jewzaam / @thedoh Distribution Venues 75 • Raw manifest

    files, eg in Github • Operator Lifecycle Manager (OLM) • OperatorHub.io • ...
  55. 78.

    #devconf_cz @jewzaam / @thedoh The End 78 Slide Deck: Source

    Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator
  56. 79.

    #devconf_cz @jewzaam / @thedoh What questions can we answer? 79

    Slide Deck: Source Code: https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator