It's all about reconciliation - Anatomy of a kubernetes controller

Developing a native kubernetes app goes around two simple concepts: defining new data structures (custom resources) and delivering a kubernetes component that handles those structures.

Despite that, doing the first steps can be intimidating:
• How do we define (and consume) new resources?
• How about interacting with existing resources defined by somebody else?
• How do I even find their Go definition?

In this talk, I'll introduce:
• How to interact with kubernetes custom resource using non type safe clients
• How to leverage the code generation to avoid casting errors and reduce the proliferation of boilerplate code
• How informers, workqueues and other powerful tools make it possible to implement a bullet proof event loop
• Finally, I'll describe the operator-sdk, a framework that makes the implementation of a controller even simpler and more automated

Transcript

1. It’s all about reconciliation Anatomy of a kubernetes controller
2. About me • Red Hatter • Doing distributed systems for more than 10 years • OpenShift CNF networking • Passionate about open source @fedepaol fedepaol@gmail.com fpaoline@redhat.com
3. Kubernetes “Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation”. Wikipedia
4. Let’s jump on the Hype Train https://flic.kr/p/ao8qFJ
5. The Model
6. Declarative over imperative apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend spec: replicas: 3 selector: matchLabels: tier: frontend template: metadata: labels: tier: frontend spec: containers: - name: php-redis image: gcr.io/google_samples/gb-frontend:v3 Pod 1 Pod 3 Pod 2
7. Declarative over imperative apiVersion: apps/v1 kind: ReplicaSet metadata: name: frontend spec: replicas: 2 selector: matchLabels: tier: frontend template: metadata: labels: tier: frontend spec: containers: - name: php-redis image: gcr.io/google_samples/gb-frontend:v3 Pod 1 Pod 3 Pod 2
8. Stable Equilibrium ETCD API Server Controller 1 Controller 2
9. Complications https://flic.kr/p/HCWbEi
10. Knock! Knock! Race Condition! Who’s There? • Optimistic Concurrency • Conflicts are resolved with failures • The client must implement the extra logic of retrying ETCD API Server Controller 1 Controller 2
11. Missed Events • Network is not reliable • Pods can die • Nodes can die! https://flic.kr/p/ofjycj
12. Edge Driven vs Level Driven # of replicas
13. Edge Driven vs Level Driven 0 -> 1 1 -> 0 0 -> 2 With edges, the behaviour depends on the variation of the data observed at edge location.
14. Edge Driven vs Level Driven 0 1 0 2 With levels, we care about the level, in our case the state of the system at a given time.
15. Edge Triggered, Level Driven It’s fine to use the events (edges) as waking up points, but we must use the system state (the level) to implement our logic.
16. Kubernetes Types
17. Kubernetes Types • Group /apis/batch/v1/namespaces/FOO/jobs
18. Kubernetes Types • Group • Version /apis/batch/v1/namespaces/FOO/jobs
19. Kubernetes Types • Group • Version • Resource /apis/batch/v1/namespaces/FOO/jobs
20. Kubernetes Types • Group • Version • Resource /apis/batch/v1/namespaces/FOO/jobs KIND
21. Common traits of a K8s kind https://flic.kr/p/c7yUpJ type Pod struct { metav1.TypeMeta metav1.ObjectMeta Spec PodSpec Status PodStatus }
22. Type Meta type TypeMeta struct { Kind string APIVersion string } type ObjectMeta struct { Name string GenerateName string Namespace string SelfLink string UID types.UID ResourceVersion string Generation int64 . . Object Meta
23. Spec & Status Spec Status Controller Outer World
24. Spec & Status Spec Status User Controller Outer World
25. Spec & Status Spec Status User Controller Outer World
26. Spec & Status Spec Status User Controller Outer World
27. Client-Go
28. Pod k8s/api/core/v1 core/v1 Types Clients
29. Pod k8s/api/core/v1 DaemonSet k8s/api/apps/v1 core/v1 apps/v1 Types Clients
30. Pod k8s/api/core/v1 DaemonSet k8s/api/apps/v1 NetworkPolicy k8s/api/networking/v1 core/v1 apps/v1 networking/v1 Types Clients
31. Pod k8s/api/core/v1 DaemonSet k8s/api/apps/v1 NetworkPolicy k8s/api/networking/v1 core/v1 apps/v1 networking/v1 Types Clients Clientset
32. So how do I get a pod?
33. Client-Go config, _ := clientcmd.BuildConfigFromFlags("", "/home/fede/kubeconfig") clientset, _ := kubernetes.NewForConfig(config) pods, _ := clientset.CoreV1().Pods("testnamespace").List(v1.ListOptions{}) ds, _ := clientset.AppsV1().DaemonSet("testnamespace").Get("myset", v1.GetOptions{})
34. Client-Go config, _ := clientcmd.BuildConfigFromFlags("", "/home/fede/kubeconfig") clientset, _ := kubernetes.NewForConfig(config) pods, _ := clientset.CoreV1().Pods("testnamespace").List(v1.ListOptions{}) ds, _ := clientset.AppsV1().DaemonSet("testnamespace").Get("myset", v1.GetOptions{}) Get Delete Create Update List Patch Watch
35. Watch returns a watch.Interface type Interface interface { Stop() ResultChan() <-chan Event } type Event struct { Type EventType Object runtime.Object }
36. Informers
37. Informers • Local cache • Notify events • Reconnects Watch Event Handler Lister Informer
38. Shared Informers import kubeinformers "k8s.io/client-go/informers" kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) podsInformer := kubeInformerFactory.Core().V1().Pods() podsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { }, ... }) podsInformer.Informer().Lister().Pods("namespace").Get("podname")
39. Work Queues
40. Work Queues type Interface interface { Add(item interface{}) Len() int Get() (item interface{}, shutdown bool) Done(item interface{}) ShutDown() ShuttingDown() bool }
41. Work Queues Handler Informer k1 k2 k3 Lister
42. Rate Limiting Work Queue func NewRateLimitingQueue(rateLimiter RateLimiter) RateLimitingInterface type RateLimitingInterface interface { DelayingInterface AddRateLimited(item interface{}) Forget(item interface{}) NumRequeues(item interface{}) int }
43. Rate Limiting Work Queue Handler Informer k1 k2 k3 Lister k3
44. Putting All Together
45. Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)
46. Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)
47. Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)
48. Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)
49. Setting the informer and the queue kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30) informer := kubeInformerFactory.Core().V1().Pods() informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { pod := obj.(*v1.Pod) key, _ = cache.MetaNamespaceKeyFunc(obj) workqueue.Add(key) }, ... cache.WaitForCacheSync(stopCh, informer.Informer().HasSynced)
50. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)
51. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)
52. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)
53. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)
54. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)
55. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj) ns, name, _ := cache.SplitMetaNamespaceKey(key) pod, err := informer.Lister().Pods(namespace).Get(name) ….
56. Reading from the queue obj, shutdown := c.workqueue.Get() if shutdown { return false } err := func(obj interface{}) error { defer workqueue.Done(obj) if key, ok := obj.(string); !ok { // should never happen } if err := handle(key); err != nil { workqueue.AddRateLimited(key) return fmt.Errorf(“Help”) } c.workqueue.Forget(obj) return nil }(obj)
57. Custom Types https://flic.kr/p/oRYji2
58. Custom Resource Definition (CRD) • A kubernetes type that describes other types • Let us enhance the capabilities of a k8s cluster • From great power comes great responsibility
59. Custom Resource Definition (CRD) apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string image: type: string replicas: type: integer scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct
60. Code Generation
61. Code Generation • Deep Copy • ClientSet • Informers • Listers k8s.io/code-generator/generate-groups.sh all \ github.com/myorg/myprj/pkg/client \ github.com/myorg/myprj/pkg/apis \ example.com:v1
62. Code Generation tree pkg pkg └── apis └── example ├── register.go └── v1 ├── doc.go └── types.go
63. Code Generation tree pkg pkg └── apis └── example ├── register.go └── v1 ├── doc.go └── types.go // +k8s:deepcopy-gen=package // +k8s:defaulter-gen=TypeMeta // +groupName=foo.com package v1
64. Code Generation tree pkg pkg └── apis └── example ├── register.go └── v1 ├── doc.go └── types.go // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // TestType is a top-level type. A client is created for it. type TestType struct { metav1.TypeMeta `json:",inline"` // +optional metav1.ObjectMeta `json:"metadata,omitempty"` // +optional Status TestTypeStatus `json:"status,omitempty"` Spec TestTypeSpec `json:"spec,omitempty"` }
65. Taking “inspiration” https://github.com/kubernetes/sample-controller https://flic.kr/p/416Jh
66. Dynamic Client
67. Dynamic Client • No need for additional dependencies • Part of client-go (k8s.io/client-go/dynamic) • Objects as map[string]interface{} • Arrays as []interface{}
68. Dynamic Client gvr := schema.GroupVersionResource{Group: "", Resource: "pods", Version: "v1"} pod, err := dynamic.Interface.Resource(gvr).Get(context.Background(), "mypod", metav1.GetOptions{}) meta := pod["meta"] metaMap, ok := meta.(map[string]interface{}) name, ok := metaMap["name"] name, found, err := unstructured.NestedString(pod.Object, "metadata", "name")
69. Operators
70. Operators operator-sdk new goway-operator --repo github.com/fedepaol/goway-operator operator-sdk add api --api-version=goway.com/v1 --kind=Sample operator-sdk add controller --api-version=goway.com/v1 --kind=Sample . ├── build │ ├── bin │ │ ├── entrypoint │ │ └── user_setup │ └── Dockerfile ├── cmd │ └── manager │ └── main.go ├── deploy │ ├── crds │ │ ├── goway.com_samples_crd.yaml │ │ └── goway.com_v1_sample_cr.yaml │ ├── operator.yaml │ ├── role_binding.yaml │ ├── role.yaml │ └── service_account.yaml ├── go.mod ├── go.sum ├── pkg │ ├── apis │ │ ├── addtoscheme_goway_v1.go │ │ ├── apis.go │ │ └── goway │ │ ├── group.go │ │ └── v1 │ │ ├── doc.go │ │ ├── register.go │ │ ├── sample_types.go │ │ └── zz_generated.deepcopy.go │ └── controller │ ├── add_sample.go │ ├── controller.go │ └── sample │ └── sample_controller.go ├── tools.go └── version └── version.go github.com/operator-framework/operator-sdk
71. Operators Primary CR Operator Event
72. Operators Primary CR Secondary Operator Event Creates
73. Operators Primary CR Secondary Operator Event Owns Creates
74. Operators Primary CR Secondary Operator Owns
75. Operators Primary CR Secondary Operator Owns
76. Operators Primary CR Secondary Operator Owns Event
77. Operators err = c.Watch(&source.Kind{Type: &gowayv1.Sample{}}, &handler.EnqueueRequestForObject{}) if err != nil { return err } err = c.Watch(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &gowayv1.Sample{}, })
78. Operators err = c.Watch(&source.Kind{Type: &gowayv1.Sample{}}, &handler.EnqueueRequestForObject{}) if err != nil { return err } err = c.Watch(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &gowayv1.Sample{}, })
79. Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }
80. Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }
81. Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }
82. Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }
83. Operators func (r *ReconcileSample) Reconcile(request reconcile.Request) (reconcile.Result, error) { instance := &gowayv1.Sample{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) pod := newPodForCR(instance) controllerutil.SetControllerReference(instance, pod, r.scheme); err != nil { found := &corev1.Pod{} err = r.client.Get(context.TODO(), types.NamespacedName{Name: pod.Name, Namespace: pod.Namespace}, found) if err != nil && errors.IsNotFound(err) { r.client.Create(context.TODO(), pod) return reconcile.Result{}, nil } else if err != nil { return reconcile.Result{}, err } return reconcile.Result{}, nil }
84. Operator Lifecycle Manager • Dependency Management • Upgrades Management • “Marketplace” (operatorhub.io) • Multitenancy
85. Embrace Chaos! https://flic.kr/p/7nGC3
86. Thanks! Any questions? @fedepaol fedepaol@gmail.com Slides at: speakerdeck.com/fedepaol fpaoline@redhat.com https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md