Implementing Microservices as Kubernetes Operators

#devconf_cz @jewzaam / @thedoh
Implementing Microservices
as Kubernetes Operators
1
Naveen Malik, Sr. Principal SRE
Lisa Seelye, Sr. SRE

View Slide

You will leave with...
2
● High level understanding Kubernetes
● Understand why Operators are so great
● Why we believe Operators are Microservices
● How to create, build, deploy and test a new Operator

View Slide

Naveen Malik (@jewzaam)
Senior Principal Site Reliability Engineer
1. Software Engineer / Architect
2. Father
3. Runner
4. Maker
About Your Speakers
3
Lisa Seelye (@thedoh)
Senior Site Reliability Engineer
1. Sysadmin & Sw Eng background
2. Canadian-in-training
3. Alternate arch fan
4. Cat enthusiast

View Slide

4

View Slide

What is Kubernetes?
5
● Open-source platform
● Manages containerized workloads
● Facilitates declarative configuration & automation
● Provides standard Resources to express application deployments
● Supports custom Resources and logic around those Resources
Ref https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

View Slide

Why are we here?
6
● OpenShift SRE team operates OpenShift Dedicated
● Operators for all the things!
OpenShift?
● Flavour of Kubernetes
● OpenShift is to Kubernetes as RHEL is to Linux
OpenShift Dedicated?
● High availability, multi-AZ OpenShift clusters on AWS
● Deploy clusters into our AWS accounts or bring your own
● Fully managed and supported by Red Hat, 24x7, with 99.5% uptime SLA

View Slide

"world map" is licensed under CC0 1.0 7
5
2
1
2
2
SRE is hiring!
Stop by at
Red Hat
booth to find
out more!
(January 15, 2020)

View Slide

How Does Kubernetes Apply Changes?
8

View Slide

● Watches for change to resources
● Makes it so
How do Resources and Controllers interact?
9
● Use YAML to show desired state
● Controllers make it happen
● Examples (partial):
○ Deployment
○ Pod
○ CustomResourceDefinition
Stock Kubernetes Resources Stock Kubernetes Controllers

View Slide

What are Operators?
● Custom behavior for existing resources
● Custom behavior for custom resources
● Move current state closer to desired state
10
● They don’t ship with Kubernetes
● Models a specific application domain
● Anything is possible!
Custom Resource Definitions Custom Kubernetes Controllers

View Slide

11
Deployment
Controller
Waiting for
changes

View Slide

12
Deployment
Controller
Tries to create
Pods
New
Deployment Pod
Pod
Pod
kind: Deployment
spec:
replicas: 3
...
image: nginx:1.7,9
...

View Slide

13
Deployment
Controller
Error detected!
New
Deployment
kind: Deployment
spec:
replicas: 3
...
image: nginx:1.7,9
^
# error! ---------/
...

View Slide

14
Deployment
Controller
Success!
New
Deployment Pod
Pod
Pod
kind: Deployment
spec:
replicas: 3
...
image: nginx:1.7.9
...

View Slide

15
Deployment
Controller
Waiting for
changes

View Slide

16
What Does it Look Like for
Operators?

View Slide

17
Custom Controller
Waiting for
changes

View Slide

Why Implement Your
Microservice as an Operator?
18

View Slide

What are Microservices?
19
● Small with bounded contexts
● Communicate over a network
● Use technology-agnostic protocols
● Organized around business capabilities
● Independently deployable
In addition, usually...
● Security
● Availability
● Scalability
Processes with...
Read More: https://en.wikipedia.org/wiki/Microservices

View Slide

What are Microservices?
● Have a narrow scope
● Communicate over platform’s network
● provide an API via Kubernetes resources
● Group controllers based on business need
● Are a deployable unit
Kubernetes provides…
● Authentication & Authorization
● Highly available platform & deployments
● Scale deployment to meet needs
20
● Small with bounded contexts
● Communicate over a network
● Use technology-agnostic protocols
● Organized around business capabilities
● Independently deployable
In addition, usually...
● Security
● Availability
● Scalability
Processes with... Kubernetes Operators...
Read More: https://en.wikipedia.org/wiki/Microservices

View Slide

Operators are Microservices
21

View Slide

Using Microservices for
Config Management
22

View Slide

Monolith: Centralized Config Management
23
Configuration
Inventory
Apply Changes
Cluster 1

View Slide

24
Configuration
Inventory
Apply Changes
Cluster 1
Cluster 2
Cluster 3

View Slide

25
Configuration
Inventory
Apply Changes
Cluster 1
Cluster 2
Cluster 3
Cluster N
...

View Slide

26
Configuration
Inventory
Apply Changes
Cluster 1
Cluster 2
Cluster 3
Cluster N
...

View Slide

Microservices: Distributed Config Management
27
Configuration
Inventory
Apply Changes
Cluster 1
Desired State

View Slide

28
Configuration
Inventory
Apply Changes
Cluster 1
Controller
Desired State Realized State

View Slide

29
Configuration
Inventory
Apply Changes
Cluster 1
Controller
Cluster 2
Controller
Cluster N
Controller
...

View Slide

30
Configuration
Inventory
Apply Changes
Cluster 1
Controller
Cluster 2
Controller
Cluster N
Controller
...

View Slide

Systems
Administration
as Code
31

View Slide

Operators, by Example
32

View Slide

Operator Example: Install & Configure
33
cluster-monitoring-operator
https://github.com/openshift/cluster-monitoring-operator

View Slide

Installing & Configuring Software, Without Operators
34
● Install Prometheus, Alertmanager, and Grafana by hand
○ Did you make them highly available?
● Install kube-state-metrics on your own
● Install node_exporter on your own
● Write all the “glue” to configure Prometheus, Alertmanager and Grafana. Don’t
forget, Deployments don’t watch ConfigMaps automatically

View Slide

● Installs & Configures
○ Prometheus-operator (Prometheus and Alertmanager)
○ Grafana
○ Kube-state-metrics
○ Node_exporter
● Prometheus-operator custom resources
○ alerting rules
○ configuration
Cluster-monitoring-operator, Operator Value
35

View Slide

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: sre-node-unschedulable
role: alert-rules
name: sre-node-unschedulable
namespace: openshift-monitoring
Cluster-monitoring-operator, Custom Resource
36

View Slide

Cluster-monitoring-operator, Custom Resource
37
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: sre-node-unschedulable
role: alert-rules
name: sre-node-unschedulable
spec:
groups:
- name: sre-node-unschedulable
rules:
- alert: KubeNodeUnschedulableSRE
expr: kube_node_spec_unschedulable > 0
for: 1h
labels:
severity: warning
...

View Slide

Operator Example: Provision Credentials
38
cloud-credential-operator
https://github.com/openshift/cloud-credential-operator

View Slide

AWS Account Credentials, Without Operators
39
● Send a request to IT to get credentials
● They validate your privilege and access, and then send to you, encrypted
● You keep them secure somehow
● Put them into Secret object
● Figure out how to scale this across x clusters, y users, z cloud providers

View Slide

● User creates CredentialsRequest object
○ Operator watches for it
● Operator talks to Amazon to create credential pair
● Operator stores results in a Secret
● User accesses the Secret
How cloud-credential-operator mints credentials
40

View Slide

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
name: aws-creds-to-describe-all-instances
namespace: devconf
Cloud-credential-operator CredentialsRequest
41

View Slide

metadata:
namespace: devconf
spec:
secretRef:
name: secret-to-store-credentials
namespace: devconf
42

View Slide

43
metadata:
namespace: devconf
spec:
secretRef:
namespace: devconf
providerSpec:
kind: AWSProviderSpec
statementEntries:
- effect: Allow
action:
- ec2:DescribeInstances
resource: "*"

View Slide

Cloud-credential-operator Resulting Secret
44
apiVersion: v1
kind: Secret
metadata:
namespace: devconf
type: Opaque
data:
aws_secret_access_key: bm90IGEgcmVhbCBzZWNyZXQK
aws_access_key_id: REVGSU5JVEVMWSBub3QgYSByZWFsIHNlY3JldAo=

View Slide

Operator Example: Manage Certificates
45
service-ca-operator
https://github.com/openshift/service-ca-operator

View Slide

Service-ca-operator, Without Operators
46
● Connect to the appropriate cluster and create a CSR
● kubectl certificate approve a CertificateSigningRequest
● Keep secrets safe
● Figure out how to scale over n clusters

View Slide

● Native objects are annotated
● Operator’s controllers watch for them
● Creates certificates and stores in a Secret
How service-ca-operator signs certificates
47

View Slide

apiVersion: v1
kind: Service
metadata:
name: "devconf"
annotations:
service.beta.openshift.io/serving-cert-secret-name
: "devconf-service-certs"
Service-ca-operator Details
48

View Slide

49
apiVersion: v1
kind: Service
metadata:
name: "devconf"
annotations:
service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs"
---
apiVersion: v1
kind: Secret
metadata:
name: "devconf-service-certs"
data: ...

View Slide

50
apiVersion: v1
kind: Service
metadata:
name: "devconf"
annotations:
service.beta.openshift.io/serving-cert-secret-name: "devconf-service-certs"
---
apiVersion: v1
kind: Secret
metadata:
name: "devconf-service-certs"
data: ...
---
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
service.beta.openshift.io/inject-cabundle
: "true"
name: "devconf-service-cabundle
"
data: {} # .service-ca.crt added by operator

View Slide

Operators Increase
Velocity, Scalability,
Availability, ...
51

View Slide

Operator Demo: Create Pod
52
pod-operator (contrived example)

View Slide

Why create this operator?
53
Show how to…
1. Create a golang operator
2. Create a Custom Resource Definition (CRD)
3. Create a Custom Controller

View Slide

What does the Operator do?
54
Create Pod from a Custom Resource
Ensure Pod exists for each Custom Resource
Cleanup Pod when Custom Resource is deleted

View Slide

Tools
55
● operator-sdk
○ software development toolkit (SDK) for building Kubernetes applications
● podman
○ daemonless container engine for developing, managing, and running containers
● kubectl
○ command line tool for controlling Kubernetes clusters

View Slide

How is the Operator created?
56

View Slide

57
1. operator-sdk new
pod-operator

View Slide

58
1. operator-sdk new
2. operator-sdk add api
pod-operator
PodRequest (CRD)
---
spec:

View Slide

59
1. operator-sdk new
3. operator-sdk add controller
pod-operator
PodRequest (CRD)
---
spec:
Controller
---
- watch PodRequest & Pod
- assign Owner
- create Pod

View Slide

60
1. operator-sdk new
4. edit code
pod-operator
PodRequest (CRD)
---
spec:
name: “busybox”
image: “busybox”
command: [“sleep 3600”]
Controller
---
- assign Owner
- create Pod

View Slide

61
1. operator-sdk new
4. edit code
5. operator-sdk build
6. podman push
pod-operator
PodRequest (CRD)
---
spec:
name: “busybox”
Controller
---
- assign Owner
- create Pod
quay.io

View Slide

62
1. operator-sdk new
4. edit code
5. operator-sdk build
6. podman push
7. kubectl create
pod-operator
PodRequest (CRD)
---
spec:
name: “busybox”
Controller
---
- assign Owner
- create Pod
quay.io

View Slide

What are we testing?
63
Create Pod from a Custom Resource
pod-operator
PodRequest (CRD)
---
spec:
name: “busybox”
Controller
---
- assign Owner
- create Pod
PodRequest (CR) Pod
quay.io

View Slide

64
Ensure Pod exists for each Custom Resource
pod-operator
PodRequest (CRD)
---
spec:
name: “busybox”
Controller
---
- assign Owner
- create Pod
PodRequest (CR) Pod
quay.io

View Slide

65
Cleanup Pod when Custom Resource is deleted
pod-operator
PodRequest (CRD)
---
spec:
name: “busybox”
Controller
---
- assign Owner
- create Pod
PodRequest (CR) Pod
quay.io

View Slide

Demo Time!
66

View Slide

Bootstrapping an operator from scratch!
67
export GO111MODULE=on
export QUAY_USERNAME=nmalik
export GITHUB_USERNAME=jewzaam
export OP_VERSION=0.0.1
export OP_NAME=pod-operator
export OP_API_GROUP=pod.jewzaam.org
export OP_API_VERSION=v1alpha1
export OP_KIND=PodRequest
# bootstrap the operator
mkdir -p $GOPATH/src/github.com/ $GITHUB_USERNAME
cd $GOPATH/src/github.com/ $GITHUB_USERNAME
operator-sdk new $OP_NAME
cd $OP_NAME
# generate CRD and Controller
operator-sdk add api --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND
operator-sdk add controller --api-version= $OP_API_GROUP/$OP_API_VERSION --kind=$OP_KIND

View Slide

Customize the API
68
...
type PodRequestSpec struct {
Name string `json:"name"`
Image string `json:"image"`
Command []string `json:"command"`
}
...

View Slide

Customize the Controller
69
func newPodForCR(cr *podv1alpha1.PodRequest) *corev1.Pod {
labels := map[string]string{
"app": cr.Name,
}
return &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: cr.Name,
Namespace: cr.Namespace,
Labels: labels,
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: cr.Spec.Name,
Image: cr.Spec.Image,
Command: cr.Spec.Command,
},
},
},
}
}

View Slide

Build and Publish: Operator
70
# build operator
operator-sdk build quay.io/ $QUAY_USERNAME /$OP_NAME:latest --image-builder podman
# push operator images
podman push quay.io/ $QUAY_USERNAME /$OP_NAME:latest

View Slide

Deploy!
71
sed -i "s|REPLACE_IMAGE|quay.io/ $QUAY_USERNAME /$OP_NAME:demo|g" deploy/operator.yaml
kubectl create namespace pod-operator
kubectl create -f deploy/ -f deploy/crds/

View Slide

Make a Request
72
# create the PodRequest
cat << EOF > podrequest.yaml
apiVersion: pod.jewzaam.org/v1alpha1
kind: PodRequest
metadata:
name: busybox
namespace: pod-operator
spec:
name: busybox
image: busybox
command: [ "/bin/sh", "-ec", "while :; do echo 'Hello DevConf.CZ 2020!'; sleep 5; done" ]
EOF
kubectl create -f podrequest.yaml
# verify it’s doing what we expected
kubectl logs busybox -f

View Slide

Delete the Request
73
# delete and verify it's gone
kubectl delete podrequest busybox
# verify it's gone (or terminating)
kubectl get pods -l app=busybox

View Slide

Distributing Operator
Based Services
74

View Slide

Distribution Venues
75
● Raw manifest files, eg in Github
● Operator Lifecycle Manager (OLM)
● OperatorHub.io
● ...

View Slide

Bringing it all Together
76

View Slide

Operators do Work
In the Cluster
77

View Slide

The End
78
Slide Deck: Source Code:
https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator

View Slide

What questions can we answer?
79
Slide Deck: Source Code:
https://speakerdeck.com/jewzaam/implementing-microservices-as-kubernetes-operators https://github.com/jewzaam/pod-operator

View Slide

Implementing Microservices as Kubernetes Operators

Implementing Microservices as Kubernetes Operators

Naveen Malik

More Decks by Naveen Malik

Other Decks in Technology

Featured

Transcript