Extending Kubernetes with Operators/Controllers

Within the Kubernetes community, there has always been a desire to continue to add new features and functionality into the apiserver; you see this every day from the version of each primitive you use. This desire grew so much that the community introduced custom aggregate apiservers these allow you to build custom apis and run them on the master with coordination from the exposed apiserver, but they require you to run your own storage layer or convince your cluster admin to let you use the primary etcd cluster. The community then created ThirdPartyResources; these gave you the ability to create custom resources that were registered with the apiserver and were able to be run what CoreOS coined as the Operator pattern. TPRs, while great, eventually were rebuilt and moved into apiextensions and called CRDs. In this talk, we will build an Operator using CRDs, the k8s code-generation libraries and informer libraries which are generated.

CRDs and Operators are excellent for anything from managing some custom resource like public keys in an SSH authorized_key file or to help you to create easily deployable custom-off-the-shelf software that any cluster administrator or developer could deploy into their environment without having to conform to a predefined spec.

Transcript

1. Extending Kubernetes with CRDs and Controllers/Operators Chris Hein | Developer Advocate | AWS | CNCF Ambassador heichris@amazon.com @christopherhein christopherhein
2. What is Kubernetes?
3. Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications
4. Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications
5. What does all of that mean {api}
6. Let’s see it $ kubectl get pods -v=7 GET https://apiserver/api/v1/namespaces/default/pods?limit=500 Request Headers: Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216 Response Status: 200 OK in 145 milliseconds Response Headers: Content-Type: application/json Content-Length: 1909 Date: Wed, 28 Nov 2018 00:23:05 GMT Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634 Response Body: ...
7. API Layer { "paths": [ "/api", "/api/v1", "/apis", "/apis/", "/apis/admissionregistration.k8s.io", "/apis/admissionregistration.k8s.io/v1beta1", ... ] }
8. Kubernetes objects • Record of intent • Persistent entities • Marshal the desired state of your cluster • Examples: Pods, Services, Ingress, NetworkPolicies, ConfigMaps, Secrets, etc.
9. How do they work?
10. Control Loop state feedback reconcile
11. What do we get out of this complexity Immutable Declarative Self-healing {kind}
12. How does it connect?
13. Controllers
14. attachdetach, bootstrapsigner, clusterrole- aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, garbagecollector, job, namespace, horizontalpodautoscaling, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv- protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after- finished bootstrapsigner, tokencleaner
15. Control Loop state feedback reconcile
16. How do you extend it?
17. Extending Kubernetes • Built to be highly configurable • Uses the control loop pattern • Extensions incl. Storage, Device, Network plugins, Aggregate apiservers, CRDs and more • Awesome Operators list - https://git.io/vh6Qm
18. CustomResourceDefinitions (CRDs)
19. CustomResourceDefinitions (CRDs) • Dynamically creates RESTful routes in the apiserver • Defined with OR without a Pod to use them • Uses YAML to tell Kubernetes to create them • Built-in Mechanisms for .Status and sub-resources • Ability to validate at the apiserver
20. CRD YAML Standard K8s primitives style apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com version: v1 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct
21. Defined Custom Resource Again… Standard K8s primitives style apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image
22. What do you do with CRDs?
23. Build Operators
24. “Operators are domain specific” - Kris Nova
25. Operators • Coined by CoreOS • Standalone Pod in a Kubernetes cluster • Watches kube-apiserver for updates • Can watch any Kubernetes resource • Allows you to take action on new or updates to existing resources changes
26. Simple use case • https://git.io/fjOZ3 • Uses client-go and exposes a CRD for managing authorized_key files on your nodes • Deploy the operator + CRDs then deploy an AuthorizedKey Resource for each employee !!! DEMO USE ONLY !!!
27. Demo
28. Code Generation FTW! !
29. # apis/node/types.go // +genclient // +genclient:noStatus // +genclient:nonNamespaced // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object type AuthorizedKey struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata"` Data AuthorizedKeyData `json:"data"` } type AuthorizedKeyData struct { Key string `json:"key"` }
30. # hack/update-codegen.sh scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" projectdir="$(pwd | sed "s#$GOPATH\/src\/##g")" cd ${scriptdir}/vendor/k8s.io/code-generator && ./generate-groups.sh \ all \ ${projectdir}/generated \ ${projectdir}/apis \ "node.chrishein.com:v1alpha1" \
31. # generated/... generated/ !"" clientset # $"" versioned # !"" clientset.go # !"" doc.go # !"" fake # # !"" clientset_generated.go # # !"" doc.go # # $"" register.go # !"" scheme # # !"" doc.go # # $"" register.go # $"" typed ...
32. # main.go cfg, _ := clientcmd.BuildConfigFromFlags(masterURL, kubeconfig) nodeClient, _ := clientset.NewForConfig(cfg) clientInformerFactory := informers.NewSharedInformerFactory(nodeClient, time.Minute*30) controller := ctrl.New(nodeClient, clientInformerFactory.Node().V1alpha1().AuthorizedKeys()) clientInformerFactory.Start(stopCh) if err = controller.Run(1, stopCh); err != nil { klog.Fatalf("error running controller error=%s", err.Error()) }
33. # controller/controller.go#New authKeyInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs { AddFunc: controller.enqueueAuthKey, UpdateFunc: func(old, new interface{}) { newKey := new.(*nodev1alpha1.AuthorizedKey) oldKey := old.(*nodev1alpha1.AuthorizedKey) if newKey.ResourceVersion == oldKey.ResourceVersion { return } controller.enqueueAuthKey(new) }, DeleteFunc: controller.deleteAuthKey, })
34. # controller/controller.go#syncHander _, name, _ := cache.SplitMetaNamespaceKey(key) authKey, _ := c.authKeyLister.Get(name) authorizedKeyFile = authorizedkey.File{ UID: authKey.Name, Key: authKey.Data.Key, } err = authorizedKeyFile.Sync(false) return nil
35. Other use cases
37. Demo AWS Service Operator https://git.io/fhWu3
38. Others… databases storage solution out of cluster tools ?
39. Thanks! Chris Hein | Developer Advocate | AWS | CNCF Ambassador heichris@amazon.com @christopherhein christopherhein