Slide 1

Slide 1 text

Extending Kubernetes with CRDs and Controllers/Operators Chris Hein | Developer Advocate | AWS | CNCF Ambassador [email protected] @christopherhein christopherhein

Slide 2

Slide 2 text

What is Kubernetes?

Slide 3

Slide 3 text

Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications

Slide 4

Slide 4 text

Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications

Slide 5

Slide 5 text

What does all of that mean {api}

Slide 6

Slide 6 text

Let’s see it $ kubectl get pods -v=7 GET https://apiserver/api/v1/namespaces/default/pods?limit=500 Request Headers: Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216 Response Status: 200 OK in 145 milliseconds Response Headers: Content-Type: application/json Content-Length: 1909 Date: Wed, 28 Nov 2018 00:23:05 GMT Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634 Response Body: ...

Slide 7

Slide 7 text

API Layer { "paths": [ "/api", "/api/v1", "/apis", "/apis/", "/apis/admissionregistration.k8s.io", "/apis/admissionregistration.k8s.io/v1beta1", ... ] }

Slide 8

Slide 8 text

Kubernetes objects • Record of intent • Persistent entities • Marshal the desired state of your cluster • Examples: Pods, Services, Ingress, NetworkPolicies, ConfigMaps, Secrets, etc.

Slide 9

Slide 9 text

How do they work?

Slide 10

Slide 10 text

Control Loop state feedback reconcile

Slide 11

Slide 11 text

What do we get out of this complexity Immutable Declarative Self-healing {kind}

Slide 12

Slide 12 text

How does it connect?

Slide 13

Slide 13 text

Controllers

Slide 14

Slide 14 text

attachdetach, bootstrapsigner, clusterrole- aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, garbagecollector, job, namespace, horizontalpodautoscaling, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv- protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after- finished bootstrapsigner, tokencleaner

Slide 15

Slide 15 text

Control Loop state feedback reconcile

Slide 16

Slide 16 text

How do you extend it?

Slide 17

Slide 17 text

Extending Kubernetes • Built to be highly configurable • Uses the control loop pattern • Extensions incl. Storage, Device, Network plugins, Aggregate apiservers, CRDs and more • Awesome Operators list - https://git.io/vh6Qm

Slide 18

Slide 18 text

CustomResourceDefinitions (CRDs)

Slide 19

Slide 19 text

CustomResourceDefinitions (CRDs) • Dynamically creates RESTful routes in the apiserver • Defined with OR without a Pod to use them • Uses YAML to tell Kubernetes to create them • Built-in Mechanisms for .Status and sub-resources • Ability to validate at the apiserver

Slide 20

Slide 20 text

CRD YAML Standard K8s primitives style apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com version: v1 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct

Slide 21

Slide 21 text

Defined Custom Resource Again… Standard K8s primitives style apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image

Slide 22

Slide 22 text

What do you do with CRDs?

Slide 23

Slide 23 text

Build Operators

Slide 24

Slide 24 text

“Operators are domain specific” - Kris Nova

Slide 25

Slide 25 text

Operators • Coined by CoreOS • Standalone Pod in a Kubernetes cluster • Watches kube-apiserver for updates • Can watch any Kubernetes resource • Allows you to take action on new or updates to existing resources changes

Slide 26

Slide 26 text

Simple use case • https://git.io/fjOZ3 • Uses client-go and exposes a CRD for managing authorized_key files on your nodes • Deploy the operator + CRDs then deploy an AuthorizedKey Resource for each employee !!! DEMO USE ONLY !!!

Slide 27

Slide 27 text

Demo

Slide 28

Slide 28 text

Code Generation FTW! !

Slide 29

Slide 29 text

# apis/node/types.go // +genclient // +genclient:noStatus // +genclient:nonNamespaced // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object type AuthorizedKey struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata"` Data AuthorizedKeyData `json:"data"` } type AuthorizedKeyData struct { Key string `json:"key"` }

Slide 30

Slide 30 text

# hack/update-codegen.sh scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" projectdir="$(pwd | sed "s#$GOPATH\/src\/##g")" cd ${scriptdir}/vendor/k8s.io/code-generator && ./generate-groups.sh \ all \ ${projectdir}/generated \ ${projectdir}/apis \ "node.chrishein.com:v1alpha1" \

Slide 31

Slide 31 text

# generated/... generated/ !"" clientset # $"" versioned # !"" clientset.go # !"" doc.go # !"" fake # # !"" clientset_generated.go # # !"" doc.go # # $"" register.go # !"" scheme # # !"" doc.go # # $"" register.go # $"" typed ...

Slide 32

Slide 32 text

# main.go cfg, _ := clientcmd.BuildConfigFromFlags(masterURL, kubeconfig) nodeClient, _ := clientset.NewForConfig(cfg) clientInformerFactory := informers.NewSharedInformerFactory(nodeClient, time.Minute*30) controller := ctrl.New(nodeClient, clientInformerFactory.Node().V1alpha1().AuthorizedKeys()) clientInformerFactory.Start(stopCh) if err = controller.Run(1, stopCh); err != nil { klog.Fatalf("error running controller error=%s", err.Error()) }

Slide 33

Slide 33 text

# controller/controller.go#New authKeyInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs { AddFunc: controller.enqueueAuthKey, UpdateFunc: func(old, new interface{}) { newKey := new.(*nodev1alpha1.AuthorizedKey) oldKey := old.(*nodev1alpha1.AuthorizedKey) if newKey.ResourceVersion == oldKey.ResourceVersion { return } controller.enqueueAuthKey(new) }, DeleteFunc: controller.deleteAuthKey, })

Slide 34

Slide 34 text

# controller/controller.go#syncHander _, name, _ := cache.SplitMetaNamespaceKey(key) authKey, _ := c.authKeyLister.Get(name) authorizedKeyFile = authorizedkey.File{ UID: authKey.Name, Key: authKey.Data.Key, } err = authorizedKeyFile.Sync(false) return nil

Slide 35

Slide 35 text

Other use cases

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Demo AWS Service Operator https://git.io/fhWu3

Slide 38

Slide 38 text

Others… databases storage solution out of cluster tools ?

Slide 39

Slide 39 text

Thanks! Chris Hein | Developer Advocate | AWS | CNCF Ambassador [email protected] @christopherhein christopherhein