Kubernetes Operator with Go

Transcript

1. Kubernetes Operator with Go Campinas, SP, MAR, 2020 Matheus Moraes Sensedia Software Engineer Claudio Oliveira Sensedia Lead Solutions Architect
2. Agenda Kubernetes Kubernetes Controller Operator pattern Custom Resources Definition (CRD’s) The Operator Framework Best Practices
3. Kubernetes
4. Everything in kubernetes is event, it means the things happens asynchronously
6. and controllers react to these events and do something important
7. k8s Controllers
8. kubernetes controllers is one way to extend k8s features
9. Control loop Kubernetes-native application
10. Introducing APIrator Mock for developers Easy deployments on k8s Useful for tests Creates and prepares instances for you, quickly and effective
11. github.com/apirator/apirator
12. Introducing CRD...
13. What is a CRD? CRD is an acronym for Custom Resource Definition
14. but wait…..What is Custom Resources???
15. What is a CR? A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind POD is built-in Kind Deployment is built-in Kind Service is built-in Kind APIMock is not a built-in kind is a Custom Resource
16. CRD is the way to tell instructions to kubernetes some Custom Resources properties like “template”
17. CRD example https://github.com/apirator/apirator/blob/develop/deploy/crds/apirator.io_apimocks_crd.yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: apimocks.apirator.io spec: version: v1alpha1 group: apirator.io names: kind: APIMock listKind: APIMockList plural: apimocks singular: apimock scope: Namespaced validation: openAPIV3Schema: type: object properties: spec: type: object properties: definition: type: string description: 'OpenAPI Specification' apiVersion: apirator.io/v1alpha1 kind: APIMock metadata: name: example-apimock spec: definition: | openapi: "3.0.0" info: title: Simple API overview version: 2.0.0 1 1 2 2 3 3 4 4 CR example
18. { Deploy CRD Deploy CRD to the cluster kubectl apply -f apimock.yaml kubectl command line
19. almost there, controllers and CRDs… and about operator pattern
20. Operator Pattern “An Operator is a Controller that uses a CRD to encapsulate operational knowledge for a specific application in an algorithmic and automated form. The Operator pattern allows us to extend the Controller pattern from the preceding chapter for more flexibility and greater expressiveness.” from Kubernetes Patterns Book https://learning.oreilly.com/library/view/kubernetes-patterns/9781492050278/ch23.html#Operator
21. in other words, the operator pattern encapsulates our sysadmin in a software… because sysadmin maybe get some tired and our code in golang is tireless
22. Options to build Operators!!! Metacontroller from GCP kubebuilder from SIG API Machinery Operator Framework from CoreOS/RedHat
23. Why Operator Framework??? golang based scaffolding projects in golang operator-cli that helps to create image, deploy and test automatically creation of boilerplate code to establish connect on kubernetes apply best practices to run custom controllers reasonable documentation
24. CLI operator-sdk Scaffold a new project operator-sdk new apirator-operator --repo=github.com/apirator/apirator Create a new Custom Resource operator-sdk add api --api-version=apirator.io/v1alpha1 --kind=APIMock Create a new Custom Controller operator-sdk add controller --api-version=apirator.io/v1alpha1 --kind=APIMock Generates our Custom Resource Definition operator-sdk generate crds
25. Let’s GO!!!
26. Our CRD is a golang struct type APIMock struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec APIMockSpec `json:"spec,omitempty"` Status APIMockStatus `json:"status,omitempty"` } type APIMockSpec struct { // OpenAPI Specification Definition string `json:"definition,omitempty"` } apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: apimocks.apirator.io spec: additionalPrinterColumns: # omitted version: v1alpha1 group: apirator.io names: kind: APIMock listKind: APIMockList plural: apimocks singular: apimock scope: Namespaced validation: openAPIV3Schema: type: object properties: spec: type: object properties: definition: type: string description: 'OpenAPI Specification' 1 2 4 2 3 5 5 4 3 pkg/apis/apirator/v1alpha1/apimock_types.go 1
27. Remember controller is an infinite loop...The reconcile function is called every time when things get changed
28. Reconcile function func (r *ReconcileAPIMock) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &apiratorv1alpha1.APIMock{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) if err != nil { if errors.IsNotFound(err) { return reconcile.Result{}, nil } return reconcile.Result{}, err } if errOas := oas.Validate(instance.Spec.Definition); errOas != nil { if err := r.markAsInvalidOAS(instance); err != nil { return reconcile.Result{}, err } return reconcile.Result{}, errOas } }
29. Requeue only if necessary
30. The request may be requeued and the reconcile loop triggered again: // Reconcile successful - don't requeue return reconcile.Result{}, nil // Reconcile failed due to error - requeue return reconcile.Result{}, err // Requeue for any reason other than error return reconcile.Result{Requeue: true}, nil // Reconcile for any reason than error after 5 seconds return reconcile.Result{RequeueAfter: time.Second*5}, nil Example: container image from Pod
31. every time check through the kubernetes object to ensure consistency
32. Resource Status depErr := r.EnsureDeployment(instance) if depErr != nil { instance.Status.Phase = apirator.ERROR err = r.client.Status().Update(context.TODO(), instance) if err != nil { /* omitted */ } } /status
33. always define Owner References for Kubernetes Garbage Collector
34. Owner References d := &v1.Deployment{ TypeMeta: metav1.TypeMeta{ Kind: "Deployment", APIVersion: "apps/v1", }, ObjectMeta: metav1.ObjectMeta{ Name: mock.GetName(), Namespace: mock.GetNamespace(), Labels: labels.LabelForAPIMock(mock), OwnerReferences: []metav1.OwnerReference{ *metav1.NewControllerRef(mock, mock.GroupVersionKind()), }, }, Spec: v1.DeploymentSpec{} apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: api: example-apimock managed-by: apirator name: example-apimock namespace: oas ownerReferences: - apiVersion: apirator.io/v1alpha1 controller: true kind: APIMock name: example-apimock uid: 5ad30a02-6481-11ea-9916-42010a8000c7 spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: api: example-apimock managed-by: apirator
35. handle idempotency carefully, avoid duplicated objects
36. pkg/controller/apimock/apimock_controller.go var _ reconcile.Reconciler = &ReconcileAPIMock{} type ReconcileAPIMock struct { client client.Client scheme *runtime.Scheme } pkg/controller/apimock/deployment.go func (r *ReconcileAPIMock) EnsureDeployment(mock *v1alpha1.APIMock) error { svcK8s := &v1.Deployment{} err := r.client.Get(context.TODO(), types.NamespacedName{ Name: mock.GetName(), Namespace: mock.Namespace, }, svcK8s) if err != nil && errors.IsNotFound(err) { // omitted code return nil } else if err != nil { log.Error(err, "Failed to get Deployment") return err } return nil }
37. and about update objects?
38. Diff between desired and existing objects https://github.com/google/go-cmp import "github.com/google/go-cmp/cmp" if r.isOwnedBy(existing, owner) { desired.SetResourceVersion(existing.GetResourceVersion()) diff := cmp.Diff(existing, desired, options) if diff != "" { err = r.client.Update(ctx, desired) if err != nil { return errors.Wrap(err, "failed to update "+kind+" "+namespace+"/"+name) } logger.Debug(kind + " " + namespace + "/" + name + " updated") } else { logger.Debug(existing.GetSelfLink() + " unchanged") } return nil } ignoredFields = [...]string{ "ObjectMeta.SelfLink", "ObjectMeta.UID", "ObjectMeta.ResourceVersion", "ObjectMeta.Generation", "ObjectMeta.CreationTimestamp", "ObjectMeta.Finalizers", "ObjectMeta.ManagedFields", "TypeMeta.APIVersion", }
39. the Reconcile struct has a pre-built k8s config client with desired security configuration (RBAC), take this advantage. USE it!!!
41. keep the legibility and simplicity every time, it should be designed for sysadmin guys, not for the business team
42. github.com/apirator/apirator
44. Demo time
45. Thanks! Any questions ? mfariam matheusfm matheusfm claudioed claudioed claudioed You can find us at: