Tweet
Tweet

Slide 1

Slide 1 text

It’s all about reconciliation Anatomy of a kubernetes controller

Slide 2

Slide 2 text

About me ● Red Hatter ● Doing distributed systems for more than 10 years ● OpenShift CNF networking ● Passionate about open source @fedepaol fedepaol@gmail.com fpaoline@redhat.com

Slide 3

Slide 3 text

Kubernetes “Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation”. Wikipedia

Slide 4

Slide 4 text

Let’s jump on the Hype Train https://flic.kr/p/ao8qFJ

Slide 5

Slide 5 text

The Model

Slide 6

Slide 6 text

Declarative over imperative apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend spec: replicas: 3 selector: matchLabels: tier: frontend template: metadata: labels: tier: frontend spec: containers: - name: php-redis image: gcr.io/google_samples/gb-frontend:v3 Pod 1 Pod 3 Pod 2

Slide 7

Slide 7 text

Declarative over imperative apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend spec: replicas: 2 selector: matchLabels: tier: frontend template: metadata: labels: tier: frontend spec: containers: - name: php-redis image: gcr.io/google_samples/gb-frontend:v3 Pod 1 Pod 3 Pod 2

Slide 8

Slide 8 text

Stable Equilibrium ETCD API Server Controller 1 Controller 2

Slide 9

Slide 9 text

Complications https://flic.kr/p/HCWbEi

Slide 10

Slide 10 text

Knock! Knock! Race Condition! Who’s There? ● Optimistic Concurrency ● Conflicts are resolved with failures ● The client must implement the extra logic of retrying ETCD API Server Controller 1 Controller 2

Slide 11

Slide 11 text

Missed Events ● Network is not reliable ● Pods can die ● Nodes can die! https://flic.kr/p/ofjycj

Slide 12

Slide 12 text

Edge Driven vs Level Driven # of replicas

Slide 13

Slide 13 text

Edge Driven vs Level Driven 0 -> 1 1 -> 0 0 -> 2 With edges, the behaviour depends on the variation of the data observed at edge location.

Slide 14

Slide 14 text

Edge Driven vs Level Driven 0 1 0 2 With levels, we care about the level, in our case the state of the system at a given time.

Slide 15

Slide 15 text

Edge Triggered, Level Driven It’s fine to use the events (edges) as waking up points, but we must use the system state (the level) to implement our logic.

Slide 16

Slide 16 text

Kubernetes Types

Slide 17

Slide 17 text

Kubernetes Types ● Group /apis/batch/v1/namespaces/FOO/jobs

Slide 18

Slide 18 text

Kubernetes Types ● Group ● Version /apis/batch/v1/namespaces/FOO/jobs

Slide 19

Slide 19 text

Kubernetes Types ● Group ● Version ● Resource /apis/batch/v1/namespaces/FOO/jobs

Slide 20

Slide 20 text

Kubernetes Types ● Group ● Version ● Resource /apis/batch/v1/namespaces/FOO/jobs KIND

Slide 21

Slide 21 text

Common traits of a K8s kind https://flic.kr/p/c7yUpJ type Pod struct { metav1.TypeMeta metav1.ObjectMeta Spec PodSpec Status PodStatus }

Slide 22

Slide 22 text

Type Meta type TypeMeta struct { Kind string APIVersion string } type ObjectMeta struct { Name string GenerateName string Namespace string SelfLink string UID types.UID ResourceVersion string Generation int64 . . Object Meta

Slide 23

Slide 23 text

Spec & Status Spec Status Controller Outer World

Slide 24

Slide 24 text

Spec & Status Spec Status User Controller Outer World

Slide 25

Slide 25 text

Spec & Status Spec Status User Controller Outer World

Slide 26

Slide 26 text

Spec & Status Spec Status User Controller Outer World

Slide 27

Slide 27 text

Client-Go

Slide 28

Slide 28 text

Pod k8s/api/core/v1 core/v1 Types Clients

Slide 29

Slide 29 text

Pod k8s/api/core/v1 DaemonSet k8s/api/apps/v1 core/v1 apps/v1 Types Clients

Slide 30

Slide 30 text

Pod k8s/api/core/v1 DaemonSet k8s/api/apps/v1 NetworkPolicy k8s/api/networking/v1 core/v1 apps/v1 networking/v1 Types Clients

Slide 31

Slide 31 text

Pod k8s/api/core/v1 DaemonSet k8s/api/apps/v1 NetworkPolicy k8s/api/networking/v1 core/v1 apps/v1 networking/v1 Types Clients Clientset

Slide 32

Slide 32 text

So how do I get a pod?

Slide 33

Slide 33 text

Client-Go config, _ := clientcmd.BuildConfigFromFlags("", "/home/fede/kubeconfig") clientset, _ := kubernetes.NewForConfig(config) pods, _ := clientset.CoreV1().Pods("testnamespace").List(v1.ListOptions{}) ds, _ := clientset.AppsV1().DaemonSet("testnamespace").Get("myset", v1.GetOptions{})

Slide 34

Slide 34 text

Client-Go config, _ := clientcmd.BuildConfigFromFlags("", "/home/fede/kubeconfig") clientset, _ := kubernetes.NewForConfig(config) pods, _ := clientset.CoreV1().Pods("testnamespace").List(v1.ListOptions{}) ds, _ := clientset.AppsV1().DaemonSet("testnamespace").Get("myset", v1.GetOptions{}) Get Delete Create Update List Patch Watch

Slide 35

Slide 35 text

Watch returns a watch.Interface type Interface interface { Stop() ResultChan() <-chan Event } type Event struct { Type EventType Object runtime.Object }

Slide 36

Slide 36 text

Informers

Slide 37

Slide 37 text

Informers ● Local cache ● Notify events ● Reconnects Watch Event Handler Lister Informer

Slide 38

Slide 38 text

Shared Informers import kubeinformers "k8s.io/client-go/informers" kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) podsInformer := kubeInformerFactory.Core().V1().Pods() podsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { }, ... }) podsInformer.Informer().Lister().Pods("namespace").Get("podname")

Slide 39

Slide 39 text

Work Queues

Slide 40

Slide 40 text

Work Queues type Interface interface { Add(item interface{}) Len() int Get() (item interface{}, shutdown bool) Done(item interface{}) ShutDown() ShuttingDown() bool }

Slide 41

Slide 41 text

Work Queues Handler Informer k1 k2 k3 Lister

Slide 42

Slide 42 text

Rate Limiting Work Queue func NewRateLimitingQueue(rateLimiter RateLimiter) RateLimitingInterface type RateLimitingInterface interface { DelayingInterface AddRateLimited(item interface{}) Forget(item interface{}) NumRequeues(item interface{}) int }

Slide 43

Slide 43 text

Rate Limiting Work Queue Handler Informer k1 k2 k3 Lister k3

Slide 44

Slide 44 text

Putting All Together

Slide 45

Slide 45 text

Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)

Slide 46

Slide 46 text

Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)

Slide 47

Slide 47 text

Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)

Slide 48

Slide 48 text

Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)

Slide 49

Slide 49 text

Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)

Slide 50

Slide 50 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)

Slide 51

Slide 51 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)

Slide 52

Slide 52 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)

Slide 53

Slide 53 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)

Slide 54

Slide 54 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)

Slide 55

Slide 55 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj) ns, name, _ := cache.SplitMetaNamespaceKey(key) pod, err := informer.Lister().Pods(namespace).Get(name) ….

Slide 56

Slide 56 text

Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)

Slide 57

Slide 57 text

Custom Types https://flic.kr/p/oRYji2

Slide 58

Slide 58 text

Custom Resource Definition (CRD) ● A kubernetes type that describes other types ● Let us enhance the capabilities of a k8s cluster ● From great power comes great responsibility

Slide 59

Slide 59 text

Custom Resource Definition (CRD) apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct

Slide 60

Slide 60 text

Code Generation

Slide 61

Slide 61 text

Code Generation ● Deep Copy ● ClientSet ● Informers ● Listers k8s.io/code-generator/generate-groups.sh all \ github.com/myorg/myprj/pkg/client \ github.com/myorg/myprj/pkg/apis \ example.com:v1

Slide 62

Slide 62 text

Code Generation tree pkg pkg └── apis └── example ├── register.go └── v1 ├── doc.go └── types.go

Slide 63

Slide 63 text

Code Generation tree pkg pkg └── apis └── example ├── register.go └── v1 ├── doc.go └── types.go // +k8s:deepcopy-gen=package // +k8s:defaulter-gen=TypeMeta // +groupName=foo.com package v1

Slide 64

Slide 64 text

Code Generation tree pkg pkg └── apis └── example ├── register.go └── v1 ├── doc.go └── types.go // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // TestType is a top-level type. A client is created for it. type TestType struct { metav1.TypeMeta `json:",inline"` // +optional metav1.ObjectMeta `json:"metadata,omitempty"` // +optional Status TestTypeStatus `json:"status,omitempty"` Spec TestTypeSpec `json:"spec,omitempty"` }

Slide 65

Slide 65 text

Taking “inspiration” https://github.com/kubernetes/sample-controller https://flic.kr/p/416Jh

Slide 66

Slide 66 text

Dynamic Client

Slide 67

Slide 67 text

Dynamic Client ● No need for additional dependencies ● Part of client-go (k8s.io/client-go/dynamic) ● Objects as map[string]interface{} ● Arrays as []interface{}

Slide 68

Slide 68 text

Dynamic Client gvr := schema.GroupVersionResource{Group: "", Resource: "pods", Version: "v1"} pod, err := dynamic.Interface.Resource(gvr).Get(context.Background(), "mypod", metav1.GetOptions{}) meta := pod["meta"] metaMap, ok := meta.(map[string]interface{}) name, ok := metaMap["name"] name, found, err := unstructured.NestedString(pod.Object, "metadata", "name")

Slide 69

Slide 69 text

Operators

Slide 70

Slide 70 text

Operators operator-sdk new goway-operator --repo github.com/fedepaol/goway-operator operator-sdk add api --api-version=goway.com/v1 --kind=Sample operator-sdk add controller --api-version=goway.com/v1 --kind=Sample . ├── build │ ├── bin │ │ ├── entrypoint │ │ └── user_setup │ └── Dockerfile ├── cmd │ └── manager │ └── main.go ├── deploy │ ├── crds │ │ ├── goway.com_samples_crd.yaml │ │ └── goway.com_v1_sample_cr.yaml │ ├── operator.yaml │ ├── role_binding.yaml │ ├── role.yaml │ └── service_account.yaml ├── go.mod ├── go.sum ├── pkg │ ├── apis │ │ ├── addtoscheme_goway_v1.go │ │ ├── apis.go │ │ └── goway │ │ ├── group.go │ │ └── v1 │ │ ├── doc.go │ │ ├── register.go │ │ ├── sample_types.go │ │ └── zz_generated.deepcopy.go │ └── controller │ ├── add_sample.go │ ├── controller.go │ └── sample │ └── sample_controller.go ├── tools.go └── version └── version.go github.com/operator-framework/operator-sdk

Slide 71

Slide 71 text

Operators Primary CR Operator Event

Slide 72

Slide 72 text

Operators Primary CR Secondary Operator Event Creates

Slide 73

Slide 73 text

Operators Primary CR Secondary Operator Event Owns Creates

Slide 74

Slide 74 text

Operators Primary CR Secondary Operator Owns

Slide 75

Slide 75 text

Operators Primary CR Secondary Operator Owns

Slide 76

Slide 76 text

Operators Primary CR Secondary Operator Owns Event

Slide 77

Slide 77 text

Operators err = c.Watch(&source.Kind{Type: &gowayv1.Sample{}}, &handler.EnqueueRequestForObject{}) if err != nil { return err } err = c.Watch(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &gowayv1.Sample{}, })

Slide 78

Slide 78 text

Operators err = c.Watch(&source.Kind{Type: &gowayv1.Sample{}}, &handler.EnqueueRequestForObject{}) if err != nil { return err } err = c.Watch(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &gowayv1.Sample{}, })

Slide 79

Slide 79 text

Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }

Slide 80

Slide 80 text

Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }

Slide 81

Slide 81 text

Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }

Slide 82

Slide 82 text

Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }

Slide 83

Slide 83 text

Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }

Slide 84

Slide 84 text

Operator Lifecycle Manager ● Dependency Management ● Upgrades Management ● “Marketplace” (operatorhub.io) ● Multitenancy

Slide 85

Slide 85 text

Embrace Chaos! https://flic.kr/p/7nGC3

Slide 86

Slide 86 text

Thanks! Any questions? @fedepaol fedepaol@gmail.com Slides at: speakerdeck.com/fedepaol fpaoline@redhat.com https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md