Kubernetes Operator with Go Campinas, SP, MAR, 2020 Matheus Moraes Sensedia Software Engineer Claudio Oliveira Sensedia Lead Solutions Architect 

Agenda Kubernetes Kubernetes Controller Operator pattern Custom Resources Definition (CRD’s) The Operator Framework Best Practices 

Kubernetes 

Everything in kubernetes is event, it means the things happens asynchronously 

No content

and controllers react to these events and do something important 

k8s Controllers 

kubernetes controllers is one way to extend k8s features 

Control loop Kubernetes-native application 

Introducing APIrator Mock for developers Easy deployments on k8s Useful for tests Creates and prepares instances for you, quickly and effective 

github.com/apirator/apirator 

Introducing CRD... 

What is a CRD? CRD is an acronym for Custom Resource Definition 

but wait…..What is Custom Resources??? 

What is a CR? A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind POD is built-in Kind Deployment is built-in Kind Service is built-in Kind APIMock is not a built-in kind is a Custom Resource 

CRD is the way to tell instructions to kubernetes some Custom Resources properties like “template” 

CRD example https://github.com/apirator/apirator/blob/develop/deploy/crds/apirator.io_apimocks_crd.yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: apimocks.apirator.io spec: version: v1alpha1 group: apirator.io names: kind: APIMock listKind: APIMockList plural: apimocks singular: apimock scope: Namespaced validation: openAPIV3Schema: type: object properties: spec: type: object properties: definition: type: string description: 'OpenAPI Specification' apiVersion: apirator.io/v1alpha1 kind: APIMock metadata: name: example-apimock spec: definition: | openapi: "3.0.0" info: title: Simple API overview version: 2.0.0 1 1 2 2 3 3 4 4 CR example 

{ Deploy CRD Deploy CRD to the cluster kubectl apply -f apimock.yaml kubectl command line 

almost there, controllers and CRDs… and about operator pattern 

Operator Pattern “An Operator is a Controller that uses a CRD to encapsulate operational knowledge for a specific application in an algorithmic and automated form. The Operator pattern allows us to extend the Controller pattern from the preceding chapter for more flexibility and greater expressiveness.” from Kubernetes Patterns Book https://learning.oreilly.com/library/view/kubernetes-patterns/9781492050278/ch23.html#Operator 

in other words, the operator pattern encapsulates our sysadmin in a software… because sysadmin maybe get some tired and our code in golang is tireless 

Options to build Operators!!! Metacontroller from GCP kubebuilder from SIG API Machinery Operator Framework from CoreOS/RedHat 

Why Operator Framework??? golang based scaffolding projects in golang operator-cli that helps to create image, deploy and test automatically creation of boilerplate code to establish connect on kubernetes apply best practices to run custom controllers reasonable documentation 

CLI operator-sdk Scaffold a new project operator-sdk new apirator-operator --repo=github.com/apirator/apirator Create a new Custom Resource operator-sdk add api --api-version=apirator.io/v1alpha1 --kind=APIMock Create a new Custom Controller operator-sdk add controller --api-version=apirator.io/v1alpha1 --kind=APIMock Generates our Custom Resource Definition operator-sdk generate crds 

Let’s GO!!! 

Our CRD is a golang struct type APIMock struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec APIMockSpec `json:"spec,omitempty"` Status APIMockStatus `json:"status,omitempty"` } type APIMockSpec struct { // OpenAPI Specification Definition string `json:"definition,omitempty"` } apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: apimocks.apirator.io spec: additionalPrinterColumns: # omitted version: v1alpha1 group: apirator.io names: kind: APIMock listKind: APIMockList plural: apimocks singular: apimock scope: Namespaced validation: openAPIV3Schema: type: object properties: spec: type: object properties: definition: type: string description: 'OpenAPI Specification' 1 2 4 2 3 5 5 4 3 pkg/apis/apirator/v1alpha1/apimock_types.go 1 

Remember controller is an infinite loop...The reconcile function is called every time when things get changed 

Reconcile function func (r *ReconcileAPIMock) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &apiratorv1alpha1.APIMock{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) if err != nil { if errors.IsNotFound(err) { return reconcile.Result{}, nil } return reconcile.Result{}, err } if errOas := oas.Validate(instance.Spec.Definition); errOas != nil { if err := r.markAsInvalidOAS(instance); err != nil { return reconcile.Result{}, err } return reconcile.Result{}, errOas } } 

Requeue only if necessary 

The request may be requeued and the reconcile loop triggered again: // Reconcile successful - don't requeue return reconcile.Result{}, nil // Reconcile failed due to error - requeue return reconcile.Result{}, err // Requeue for any reason other than error return reconcile.Result{Requeue: true}, nil // Reconcile for any reason than error after 5 seconds return reconcile.Result{RequeueAfter: time.Second*5}, nil Example: container image from Pod 

every time check through the kubernetes object to ensure consistency 

Resource Status depErr := r.EnsureDeployment(instance) if depErr != nil { instance.Status.Phase = apirator.ERROR err = r.client.Status().Update(context.TODO(), instance) if err != nil { /* omitted */ } } /status 

always define Owner References for Kubernetes Garbage Collector 

Owner References d := &v1.Deployment{ TypeMeta: metav1.TypeMeta{ Kind: "Deployment", APIVersion: "apps/v1", }, ObjectMeta: metav1.ObjectMeta{ Name: mock.GetName(), Namespace: mock.GetNamespace(), Labels: labels.LabelForAPIMock(mock), OwnerReferences: []metav1.OwnerReference{ *metav1.NewControllerRef(mock, mock.GroupVersionKind()), }, }, Spec: v1.DeploymentSpec{} apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: api: example-apimock managed-by: apirator name: example-apimock namespace: oas ownerReferences: - apiVersion: apirator.io/v1alpha1 controller: true kind: APIMock name: example-apimock uid: 5ad30a02-6481-11ea-9916-42010a8000c7 spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: api: example-apimock managed-by: apirator 

handle idempotency carefully, avoid duplicated objects 

pkg/controller/apimock/apimock_controller.go var _ reconcile.Reconciler = &ReconcileAPIMock{} type ReconcileAPIMock struct { client client.Client scheme *runtime.Scheme } pkg/controller/apimock/deployment.go func (r *ReconcileAPIMock) EnsureDeployment(mock *v1alpha1.APIMock) error { svcK8s := &v1.Deployment{} err := r.client.Get(context.TODO(), types.NamespacedName{ Name: mock.GetName(), Namespace: mock.Namespace, }, svcK8s) if err != nil && errors.IsNotFound(err) { // omitted code return nil } else if err != nil { log.Error(err, "Failed to get Deployment") return err } return nil } 

and about update objects? 

Diff between desired and existing objects https://github.com/google/go-cmp import "github.com/google/go-cmp/cmp" if r.isOwnedBy(existing, owner) { desired.SetResourceVersion(existing.GetResourceVersion()) diff := cmp.Diff(existing, desired, options) if diff != "" { err = r.client.Update(ctx, desired) if err != nil { return errors.Wrap(err, "failed to update "+kind+" "+namespace+"/"+name) } logger.Debug(kind + " " + namespace + "/" + name + " updated") } else { logger.Debug(existing.GetSelfLink() + " unchanged") } return nil } ignoredFields = [...]string{ "ObjectMeta.SelfLink", "ObjectMeta.UID", "ObjectMeta.ResourceVersion", "ObjectMeta.Generation", "ObjectMeta.CreationTimestamp", "ObjectMeta.Finalizers", "ObjectMeta.ManagedFields", "TypeMeta.APIVersion", } 

the Reconcile struct has a pre-built k8s config client with desired security configuration (RBAC), take this advantage. USE it!!! 

No content

keep the legibility and simplicity every time, it should be designed for sysadmin guys, not for the business team 

github.com/apirator/apirator 

No content

Demo time 

Thanks! Any questions ? mfariam matheusfm matheusfm claudioed claudioed claudioed You can find us at: