Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extending Kubernetes with Operators/Controllers

Extending Kubernetes with Operators/Controllers

Within the Kubernetes community, there has always been a desire to continue to add new features and functionality into the apiserver; you see this every day from the version of each primitive you use. This desire grew so much that the community introduced custom aggregate apiservers these allow you to build custom apis and run them on the master with coordination from the exposed apiserver, but they require you to run your own storage layer or convince your cluster admin to let you use the primary etcd cluster. The community then created ThirdPartyResources; these gave you the ability to create custom resources that were registered with the apiserver and were able to be run what CoreOS coined as the Operator pattern. TPRs, while great, eventually were rebuilt and moved into apiextensions and called CRDs. In this talk, we will build an Operator using CRDs, the k8s code-generation libraries and informer libraries which are generated.

CRDs and Operators are excellent for anything from managing some custom resource like public keys in an SSH authorized_key file or to help you to create easily deployable custom-off-the-shelf software that any cluster administrator or developer could deploy into their environment without having to conform to a predefined spec.

Christopher Hein

April 17, 2019

More Decks by Christopher Hein

Other Decks in Programming


  1. Extending Kubernetes with CRDs and Controllers/Operators Chris Hein | Developer

    Advocate | AWS | CNCF Ambassador [email protected] @christopherhein christopherhein
  2. What is Kubernetes?

  3. Open source container management platform Helps you run containers at

    scale Gives you primitives for building modern applications
  4. Open source container management platform Helps you run containers at

    scale Gives you primitives for building modern applications
  5. What does all of that mean {api}

  6. Let’s see it $ kubectl get pods -v=7 GET https://apiserver/api/v1/namespaces/default/pods?limit=500

    Request Headers: Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216 Response Status: 200 OK in 145 milliseconds Response Headers: Content-Type: application/json Content-Length: 1909 Date: Wed, 28 Nov 2018 00:23:05 GMT Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634 Response Body: ...
  7. API Layer { "paths": [ "/api", "/api/v1", "/apis", "/apis/", "/apis/admissionregistration.k8s.io",

    "/apis/admissionregistration.k8s.io/v1beta1", ... ] }
  8. Kubernetes objects • Record of intent • Persistent entities •

    Marshal the desired state of your cluster • Examples: Pods, Services, Ingress, NetworkPolicies, ConfigMaps, Secrets, etc.
  9. How do they work?

  10. Control Loop state feedback reconcile

  11. What do we get out of this complexity Immutable Declarative

    Self-healing {kind}
  12. How does it connect?

  13. Controllers

  14. attachdetach, bootstrapsigner, clusterrole- aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment,

    disruption, endpoint, garbagecollector, job, namespace, horizontalpodautoscaling, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv- protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after- finished bootstrapsigner, tokencleaner
  15. Control Loop state feedback reconcile

  16. How do you extend it?

  17. Extending Kubernetes • Built to be highly configurable • Uses

    the control loop pattern • Extensions incl. Storage, Device, Network plugins, Aggregate apiservers, CRDs and more • Awesome Operators list - https://git.io/vh6Qm
  18. CustomResourceDefinitions (CRDs)

  19. CustomResourceDefinitions (CRDs) • Dynamically creates RESTful routes in the apiserver

    • Defined with OR without a Pod to use them • Uses YAML to tell Kubernetes to create them • Built-in Mechanisms for .Status and sub-resources • Ability to validate at the apiserver
  20. CRD YAML Standard K8s primitives style apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition

    metadata: name: crontabs.stable.example.com spec: group: stable.example.com version: v1 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct
  21. Defined Custom Resource Again… Standard K8s primitives style apiVersion: "stable.example.com/v1"

    kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image
  22. What do you do with CRDs?

  23. Build Operators

  24. “Operators are domain specific” - Kris Nova

  25. Operators • Coined by CoreOS • Standalone Pod in a

    Kubernetes cluster • Watches kube-apiserver for updates • Can watch any Kubernetes resource • Allows you to take action on new or updates to existing resources changes
  26. Simple use case • https://git.io/fjOZ3 • Uses client-go and exposes

    a CRD for managing authorized_key files on your nodes • Deploy the operator + CRDs then deploy an AuthorizedKey Resource for each employee !!! DEMO USE ONLY !!!
  27. Demo

  28. Code Generation FTW! !

  29. # apis/node/types.go // +genclient // +genclient:noStatus // +genclient:nonNamespaced // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

    type AuthorizedKey struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata"` Data AuthorizedKeyData `json:"data"` } type AuthorizedKeyData struct { Key string `json:"key"` }
  30. # hack/update-codegen.sh scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd

    )" projectdir="$(pwd | sed "s#$GOPATH\/src\/##g")" cd ${scriptdir}/vendor/k8s.io/code-generator && ./generate-groups.sh \ all \ ${projectdir}/generated \ ${projectdir}/apis \ "node.chrishein.com:v1alpha1" \
  31. # generated/... generated/ !"" clientset # $"" versioned # !""

    clientset.go # !"" doc.go # !"" fake # # !"" clientset_generated.go # # !"" doc.go # # $"" register.go # !"" scheme # # !"" doc.go # # $"" register.go # $"" typed ...
  32. # main.go cfg, _ := clientcmd.BuildConfigFromFlags(masterURL, kubeconfig) nodeClient, _ :=

    clientset.NewForConfig(cfg) clientInformerFactory := informers.NewSharedInformerFactory(nodeClient, time.Minute*30) controller := ctrl.New(nodeClient, clientInformerFactory.Node().V1alpha1().AuthorizedKeys()) clientInformerFactory.Start(stopCh) if err = controller.Run(1, stopCh); err != nil { klog.Fatalf("error running controller error=%s", err.Error()) }
  33. # controller/controller.go#New authKeyInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs { AddFunc: controller.enqueueAuthKey, UpdateFunc: func(old, new interface{})

    { newKey := new.(*nodev1alpha1.AuthorizedKey) oldKey := old.(*nodev1alpha1.AuthorizedKey) if newKey.ResourceVersion == oldKey.ResourceVersion { return } controller.enqueueAuthKey(new) }, DeleteFunc: controller.deleteAuthKey, })
  34. # controller/controller.go#syncHander _, name, _ := cache.SplitMetaNamespaceKey(key) authKey, _ :=

    c.authKeyLister.Get(name) authorizedKeyFile = authorizedkey.File{ UID: authKey.Name, Key: authKey.Data.Key, } err = authorizedKeyFile.Sync(false) return nil
  35. Other use cases

  36. None
  37. Demo AWS Service Operator https://git.io/fhWu3

  38. Others… databases storage solution out of cluster tools ?

  39. Thanks! Chris Hein | Developer Advocate | AWS | CNCF

    Ambassador [email protected] @christopherhein christopherhein