Operator SDK 帶你玩轉 Kubernetes

Operator SDK 帶你玩轉 Kubernetes

[CC-BY-SA 4.0]

從觀念角度介紹 Kubernetes 的 Operator Pattern 如何解決大家在管理叢集時繁瑣卻又重複的人工作業,並使用 Operator SDK 來示範如何自己撰寫 Golang Based Operator 來解決問題。

先從 Operator Pattern 介紹整個概念,並舉出幾個例子來讓聽眾理解如何將 Kubernetes Controller 參與其中,以扮演整個 Life-cycle 中負責進行原本重複的人工作業。
舉出幾個常見的 Operator Pattern 並以 Operator SDK 為例,介紹如何如何自己撰寫 Golang Based Operator 搭配 Kubernetes CRD 來擴充 K8s 的功能。

Bf4d26477f6c4304f73215b80843b0a9?s=128

David Kuo (Davy)

August 01, 2020
Tweet

Transcript

  1. 2020 Image credit: © COSCUP, CC-BY-SA 4.0

  2. 1

  3. 2 DevOps Kubernetes Cluster

  4. 3 kubectl apply -f DevOps Kubernetes Cluster

  5. 4 kubectl apply -f DevOps Kubernetes Cluster kubectl get ...

  6. 5 kubectl apply -f DevOps Kubernetes Cluster kubectl get ...

    kubectl apply/edit …
  7. 6 kubectl apply -f DevOps Kubernetes Cluster kubectl get ...

    kubectl apply/edit …
  8. 7 DevOps Kubernetes Cluster Bot (or scripts)

  9. 8 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply …
  10. 9 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get …
  11. 10 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get … kubtctl edit …
  12. 11 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get … kubtctl edit … bot update …
  13. 12 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get … kubtctl edit … bot update … alert
  14. 13 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get … kubtctl edit … bot update … alert https://www.botkube.io/
  15. 14 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get … kubtctl edit … bot update … alert K8s Cluster 1 K8s Cluster 2 Communicate Channel https://www.botkube.io/
  16. 15 DevOps Kubernetes Cluster Bot (or scripts) bot deploy …

    kubtctl apply … kubtctl get … kubtctl edit … bot update … alert https://github.com/learnk8s/xlskubectl
  17. 16 DevOps Kubernetes Cluster Bot (or scripts)

  18. 17 DevOps Kubernetes Cluster Bot (or scripts)

  19. 18 DevOps Kubernetes Cluster Bot (or scripts) https://kubernetes.io/docs/concepts/overview/components/

  20. 19 DevOps Kubernetes Cluster Bot (or scripts) https://kubernetes.io/docs/concepts/overview/components/

  21. 20 DevOps Kubernetes Cluster Bot (or scripts) ……

  22. 21 DevOps Kubernetes Cluster Bot (or scripts) …… Database Controller

    Changed event Commit changes
  23. 22 DevOps Kubernetes Cluster Bot (or scripts) …… Database Controller

    Changed event Commit changes
  24. 23 DevOps Kubernetes Cluster Bot (or scripts) …… Database Controller

    Changed event Commit changes Extendible?
  25. 24

  26. 25

  27. 26 https://coreos.com/blog/introducing-operators.html

  28. 27 https://coreos.com/blog/introducing-operators.html

  29. 28 https://coreos.com/blog/introducing-operators.html !

  30. 29

  31. 30

  32. 31

  33. 32

  34. 33 by

  35. 34 by

  36. 35 by

  37. 36 by Basic Install Automated application provisioning and configuration management

  38. 37 by Seamless Upgrades Patch and minor version upgrades supported

  39. 38 by Full Lifecycle App lifecycle, storage lifecycle (backup, failure

    recovery)
  40. 39 by Deep Insights Metrics, alerts, log processing and workload

    analysis
  41. 40 by Auto Pilot Horizontal/vertical scaling, auto config tuning, abnormal

    detection, scheduling tuning
  42. 41 by

  43. 42 by

  44. 43 by

  45. 44 by

  46. 45

  47. 46 kubernetes-sigs controller-runtime https://book.kubebuilder.io/ kubernetes-sigs kustomize

  48. 47

  49. 48

  50. 49

  51. 50

  52. 51 Queue Client Reconciler Cache Scheme Manager Controller

  53. 52 Queue Client Reconciler Cache Scheme Manager Controller

  54. 53 Queue Client Reconciler Cache Scheme Manager Controller

  55. 54 Queue Client Reconciler Cache Scheme Manager Controller

  56. 55 Queue Client Reconciler Cache Scheme Manager Controller

  57. 56

  58. 57 apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata name: crontabs.v1.example.com spec: names:

    kind: CronTab ...
  59. 58 YAML

  60. 59 YAML // Memcached is the Schema for the memcacheds

    API // +k8s:openapi-gen=true // +kubebuilder:subresource:status // +kubebuilder:resource:path=memcached,scope=Namespaced, shortName=mc // +kubebuilder:printcolumn:name="Size",type=integer, JSONPath=`.spec.size", description="Size of the memcached deployment" // +kubebuilder:printcolumn:name="Nodes",type=string, JSONPath=".status.nodes[*]"
  61. 60 YAML // Memcached is the Schema for the memcacheds

    API // +k8s:openapi-gen=true // +kubebuilder:subresource:status // +kubebuilder:resource:path=memcached,scope=Namespaced, shortName=mc // +kubebuilder:printcolumn:name="Size",type=integer, JSONPath=`.spec.size", description="Size of the memcached deployment" // +kubebuilder:printcolumn:name="Nodes",type=string, JSONPath=".status.nodes[*]" // +kubebuilder:rbac:groups=cache.example.com, resources=memcacheds, verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=cache.example.com, resources=memcacheds/status, verbs=get;update;patch // +kubebuilder:rbac:groups=apps, resources=deployments, verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=core, resources=pods, verbs=get;list;
  62. 61 YAML

  63. 62

  64. 63 Queue Reconciler

  65. Queue Reconciler 64 // add new Controller to mgr with

    r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). Complete(r) }
  66. Queue Reconciler 65 // add new Controller to mgr with

    r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). Complete(r) } Watch Memcached changes and enqueue Memcached
  67. Queue Reconciler 66 // add new Controller to mgr with

    r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). Complete(r) } Watch Deployment changes and enqueue Memcached owner Watch Memcached changes and enqueue Memcached
  68. 67 Queue Reconciler

  69. Queue Reconciler 68 // add new Controller to mgr with

    r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). WithEventFilter(&predicate.ResourceVersionChangedPredicate{}). Complete(r) }
  70. Queue Reconciler 69 // add new Controller to mgr with

    r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). WithEventFilter(&predicate.ResourceVersionChangedPredicate{}). Complete(r) } // Or ... predicate.NewPredicateFuncs( func(meta metav1.Object, object runtime.Object) bool { // filtering... return true })
  71. Queue Reconciler 70 // More powerful predicate.Funcs{ CreateFunc: func(e event.CreateEvent)

    bool { // ... }, UpdateFunc: func(e event.UpdateEvent) bool { // ... }, DeleteFunc: func(e event.DeleteEvent) bool { // ... }, GenericFunc: func(e event.GenericEvent) bool { // ... }, }
  72. Queue Reconciler 71

  73. Queue Controller 72

  74. Queue Controller 73 // The Controller will requeue the Request

    to be processed again // if the returned error is non-nil or Result.Requeue is true, // otherwise upon completion will remove the work from the queue. func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // ... }
  75. Queue Controller 74 // The Controller will requeue the Request

    to be processed again // if the returned error is non-nil or Result.Requeue is true, // otherwise upon completion will remove the work from the queue. func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // ... } Reconcile
  76. Queue Controller 75 Reconcile // Fetch the Memcached instance memcached

    := &cachev1alpha1.Memcached{} err := r.client.Get(context.TODO(), request.NamespacedName, memcached) if err != nil { if errors.IsNotFound(err) { reqLogger.Info("Ignoring since object must be deleted.") return reconcile.Result{}, nil } reqLogger.Error(err, "Failed to get Memcached.") return reconcile.Result{}, err }
  77. Queue Controller 76 Reconcile // Fetch the Memcached instance memcached

    := &cachev1alpha1.Memcached{} err := r.client.Get(context.TODO(), request.NamespacedName, memcached) if err != nil { if errors.IsNotFound(err) { reqLogger.Info("Ignoring since object must be deleted.") return reconcile.Result{}, nil } reqLogger.Error(err, "Failed to get Memcached.") return reconcile.Result{}, err } Fetch Resource
  78. Queue Controller 77 Reconcile Fetch Resource // Check if the

    Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err }
  79. Queue Controller 78 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err }
  80. Queue Controller 79 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err }
  81. Queue Controller 80 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err } // deploymentForMemcached returns a memcached Deployment object func (r *Reconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment { dep := &appsv1.Deployment{ /* ... */ } // Set Memcached instance as the owner of the Deployment. controllerutil.SetControllerReference(m, dep, r.scheme) return dep }
  82. Queue Controller 81 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err } // deploymentForMemcached returns a memcached Deployment object func (r *Reconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment { dep := &appsv1.Deployment{ /* ... */ } // Set Memcached instance as the owner of the Deployment. controllerutil.SetControllerReference(m, dep, r.scheme) return dep } Deployment ownerRef:
  83. Queue Controller 82 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err } // deploymentForMemcached returns a memcached Deployment object func (r *Reconciler) deploymentForMemcached(m *cachev1alpha1.Memcached) *appsv1.Deployment { dep := &appsv1.Deployment{ /* ... */ } // Set Memcached instance as the owner of the Deployment. controllerutil.SetControllerReference(m, dep, r.scheme) return dep } Deployment ownerRef: Memcached
  84. Queue Controller 83 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err }
  85. Queue Controller 84 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Check if the Deployment already exists, if not create a new one deployment := &appsv1.Deployment{} err = r.client.Get(context.TODO(), types.NamespacedName{ Name: memcached.Name, Namespace: memcached.Namespace, }, deployment) if err != nil && errors.IsNotFound(err) { dep := r.deploymentForMemcached(memcached) err = r.client.Create(context.TODO(), dep) if err != nil { return reconcile.Result{}, err } return reconcile.Result{Requeue: true}, nil } else if err != nil { return reconcile.Result{}, err }
  86. Queue Controller 85 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Ensure the deployment size is the same as the spec size := memcached.Spec.Size if *deployment.Spec.Replicas != size { deployment.Spec.Replicas = &size err = r.client.Update(context.TODO(), deployment) if err != nil { return reconcile.Result{}, err } }
  87. Queue Controller 86 Reconcile Fetch Resource Create Deployment Fetch Deployment

    // Ensure the deployment size is the same as the spec size := memcached.Spec.Size if *deployment.Spec.Replicas != size { deployment.Spec.Replicas = &size err = r.client.Update(context.TODO(), deployment) if err != nil { return reconcile.Result{}, err } } Reconcile Deployment
  88. Queue Controller 87 Reconcile Fetch Resource Create Deployment Fetch Deployment

    Reconcile Deployment // Update the Memcached status with the pod names // List the pod names for this memcached's deployment podNames := getPodNames(podList.Items) // Update status.Nodes if needed if !reflect.DeepEqual(podNames, memcached.Status.Nodes) { memcached.Status.Nodes = podNames err := r.client.Status().Update(context.TODO(), memcached) if err != nil { return reconcile.Result{}, err } }
  89. Queue Controller 88 Reconcile Fetch Resource Create Deployment Fetch Deployment

    Reconcile Deployment // Update the Memcached status with the pod names // List the pod names for this memcached's deployment podNames := getPodNames(podList.Items) // Update status.Nodes if needed if !reflect.DeepEqual(podNames, memcached.Status.Nodes) { memcached.Status.Nodes = podNames err := r.client.Status().Update(context.TODO(), memcached) if err != nil { return reconcile.Result{}, err } } Update Resource Status
  90. Queue Controller 89 Reconcile Fetch Resource Create Deployment Fetch Deployment

    Reconcile Deployment Update Resource Status
  91. 90

  92. 91 Deployment ownerRef: Memcached

  93. 92 Deployment ownerRef: Memcached

  94. 93 Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef:

  95. 94 Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef:

  96. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 95

  97. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 96

  98. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 97 // This

    is the equivalent of calling // Watches(&source.Kind{Type: <ForType-forInput>}, // &handler.EnqueueRequestForOwner{ // OwnerType: apiType, IsController: true, // }) func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder { input := OwnsInput{object: object} for _, opt := range opts { opt.ApplyToOwns(&input) } blder.ownsInput = append(blder.ownsInput, input) return blder } https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/builder/controller.go
  99. 98 // This is the equivalent of calling // Watches(&source.Kind{Type:

    <ForType-forInput>}, // &handler.EnqueueRequestForOwner{ // OwnerType: apiType, IsController: true, // }) func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder { input := OwnsInput{object: object} for _, opt := range opts { opt.ApplyToOwns(&input) } blder.ownsInput = append(blder.ownsInput, input) return blder } https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/builder/controller.go
  100. Deployment ownerRef: Memcached 99 // This is the equivalent of

    calling // Watches(&source.Kind{Type: <ForType-forInput>}, // &handler.EnqueueRequestForOwner{ // OwnerType: apiType, IsController: true, // }) func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder { input := OwnsInput{object: object} for _, opt := range opts { opt.ApplyToOwns(&input) } blder.ownsInput = append(blder.ownsInput, input) return blder } https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/builder/controller.go func (blder *Builder) doWatch() error { // ... // Watches the managed types for _, own := range blder.ownsInput { src := &source.Kind{Type: own.object} hdler := &handler.EnqueueRequestForOwner{ OwnerType: blder.forInput.object, IsController: true, } allPredicates := append([]predicate.Predicate(nil), blder.globalPredicates...) allPredicates = append(allPredicates, own.predicates...) if err := blder.ctrl.Watch(src, hdler, allPredicates...); err != nil { return err } } // ... }
  101. 100 // This is the equivalent of calling // Watches(&source.Kind{Type:

    <ForType-forInput>}, // &handler.EnqueueRequestForOwner{ // OwnerType: apiType, IsController: true, // }) func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder { input := OwnsInput{object: object} for _, opt := range opts { opt.ApplyToOwns(&input) } blder.ownsInput = append(blder.ownsInput, input) return blder } https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/builder/controller.go func (blder *Builder) doWatch() error { // ... // Watches the managed types for _, own := range blder.ownsInput { src := &source.Kind{Type: own.object} hdler := &handler.EnqueueRequestForOwner{ OwnerType: blder.forInput.object, IsController: true, } allPredicates := append([]predicate.Predicate(nil), blder.globalPredicates...) allPredicates = append(allPredicates, own.predicates...) if err := blder.ctrl.Watch(src, hdler, allPredicates...); err != nil { return err } } // ... }
  102. 101 // This is the equivalent of calling // Watches(&source.Kind{Type:

    <ForType-forInput>}, // &handler.EnqueueRequestForOwner{ // OwnerType: apiType, IsController: true, // }) func (blder *Builder) Owns(object runtime.Object, opts ...OwnsOption) *Builder { input := OwnsInput{object: object} for _, opt := range opts { opt.ApplyToOwns(&input) } blder.ownsInput = append(blder.ownsInput, input) return blder } https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/handler/enqueue_owner.go func (blder *Builder) doWatch() error { // ... // Watches the managed types for _, own := range blder.ownsInput { src := &source.Kind{Type: own.object} hdler := &handler.EnqueueRequestForOwner{ OwnerType: blder.forInput.object, IsController: true, } allPredicates := append([]predicate.Predicate(nil), blder.globalPredicates...) allPredicates = append(allPredicates, own.predicates...) if err := blder.ctrl.Watch(src, hdler, allPredicates...); err != nil { return err } } // ... } type EnqueueRequestForOwner struct { // ... } func (e *EnqueueRequestForOwner) Create( evt event.CreateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.Meta) { q.Add(req) } } func (e *EnqueueRequestForOwner) Update( evt event.UpdateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.MetaOld) { q.Add(req) } for _, req := range e.getOwnerReconcileRequest(evt.MetaNew) { q.Add(req) } } // ...Delete & Generic
  103. 102 https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/handler/enqueue_owner.go func (blder *Builder) doWatch() error { // ...

    // Watches the managed types for _, own := range blder.ownsInput { src := &source.Kind{Type: own.object} hdler := &handler.EnqueueRequestForOwner{ OwnerType: blder.forInput.object, IsController: true, } allPredicates := append([]predicate.Predicate(nil), blder.globalPredicates...) allPredicates = append(allPredicates, own.predicates...) if err := blder.ctrl.Watch(src, hdler, allPredicates...); err != nil { return err } } // ... } type EnqueueRequestForOwner struct { // ... } func (e *EnqueueRequestForOwner) Create( evt event.CreateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.Meta) { q.Add(req) } } func (e *EnqueueRequestForOwner) Update( evt event.UpdateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.MetaOld) { q.Add(req) } for _, req := range e.getOwnerReconcileRequest(evt.MetaNew) { q.Add(req) } } // ...Delete & Generic
  104. 103 https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/handler/enqueue_owner.go func (blder *Builder) doWatch() error { // ...

    // Watches the managed types for _, own := range blder.ownsInput { src := &source.Kind{Type: own.object} hdler := &handler.EnqueueRequestForOwner{ OwnerType: blder.forInput.object, IsController: true, } allPredicates := append([]predicate.Predicate(nil), blder.globalPredicates...) allPredicates = append(allPredicates, own.predicates...) if err := blder.ctrl.Watch(src, hdler, allPredicates...); err != nil { return err } } // ... } type EnqueueRequestForOwner struct { // ... } func (e *EnqueueRequestForOwner) Create( evt event.CreateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.Meta) { q.Add(req) } } func (e *EnqueueRequestForOwner) Update( evt event.UpdateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.MetaOld) { q.Add(req) } for _, req := range e.getOwnerReconcileRequest(evt.MetaNew) { q.Add(req) } } // ...Delete & Generic // getOwnerReconcileRequest looks at object and returns a slice of // reconcile.Request to reconcile owners of object that match e.OwnerType. func (e *EnqueueRequestForOwner) getOwnerReconcileRequest( object metav1.Object ) []reconcile.Request { var result []reconcile.Request for _, ref := range e.getOwnersReferences(object) { refGV, err := schema.ParseGroupVersion(ref.APIVersion) // ... if ref.Kind == e.groupKind.Kind && refGV.Group == e.groupKind.Group { // Match found - add a Request for the object referred to request := reconcile.Request{NamespacedName: types.NamespacedName{ Name: ref.Name, }} // ... result = append(result, request) } } return result }
  105. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 104 Watch Deployment

    changes and enqueue Memcached owner Watch Memcached changes and enqueue Memcached https://github.com/kubernetes-sigs/controller-runtime/blob/v0.6.1/pkg/handler/enqueue_owner.go type EnqueueRequestForOwner struct { // ... } func (e *EnqueueRequestForOwner) Create( evt event.CreateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.Meta) { q.Add(req) } } func (e *EnqueueRequestForOwner) Update( evt event.UpdateEvent, q workqueue.RateLimitingInterface) { for _, req := range e.getOwnerReconcileRequest(evt.MetaOld) { q.Add(req) } for _, req := range e.getOwnerReconcileRequest(evt.MetaNew) { q.Add(req) } } // ...Delete & Generic // getOwnerReconcileRequest looks at object and returns a slice of // reconcile.Request to reconcile owners of object that match e.OwnerType. func (e *EnqueueRequestForOwner) getOwnerReconcileRequest( object metav1.Object ) []reconcile.Request { var result []reconcile.Request for _, ref := range e.getOwnersReferences(object) { refGV, err := schema.ParseGroupVersion(ref.APIVersion) // ... if ref.Kind == e.groupKind.Kind && refGV.Group == e.groupKind.Group { // Match found - add a Request for the object referred to request := reconcile.Request{NamespacedName: types.NamespacedName{ Name: ref.Name, }} // ... result = append(result, request) } } return result }
  106. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 105 Watch Deployment

    changes and enqueue Memcached owner Watch Memcached changes and enqueue Memcached
  107. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 106 Watches(&source.Kind{Type: &appsv1.Deployment{}},

    &handler.EnqueueRequestForOwner{OwnerType: &cachev1alpha1.Memcached{}, IsController: true}) Watches(&source.Kind{Type: &cachev1alpha1.Memcached{}}, &handler.EnqueueRequestForObject{})
  108. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 107

  109. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 108 map :=

    handler.ToRequestsFunc( func(obj handler.MapObject) []reconcile.Request { var result []reconcile.Request for _, ref := range getOwnersReferences(obj) { for _, rref := range getOwnersReferences(ref) { for _, rrref := range getOwnersReferences(rref) { if isMemcachedObj(rrref) { result = append(result, reconcile.Request{ NamespacedName: types.NamespacedName{ Name: rrref.Name, }, }) } } } } return result })
  110. Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef: 109 // add

    new Controller to mgr with r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). Watches( &source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestsFromMapFunc{ ToRequests: map, }). Complete(r) }
  111. 110 Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef:

  112. 111 Deployment ownerRef: Memcached ReplicaSet ownerRef: Pod ownerRef:

  113. 112 Deployment ownerRef: template: Memcached ReplicaSet ownerRef: template: Pod ownerRef:

    label:
  114. 113 Deployment ownerRef: template: Memcached ReplicaSet ownerRef: template: Pod ownerRef:

    label:
  115. 114 Deployment ownerRef: template: Memcached ReplicaSet ownerRef: template: Pod ownerRef:

    label: // add new Controller to mgr with r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). Watches( &source.Kind{Type: &corev1.Pod{}}, &enqueueRequestsFromLabel{}, &builder.WithPredicates(&hasMemcachedLabelFilter{}), ). Complete(r) }
  116. 115 Deployment ownerRef: template: Memcached ReplicaSet ownerRef: template: Pod ownerRef:

    label: // add new Controller to mgr with r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { return ctrl.NewControllerManagedBy(mgr). Named("memcached-controller"). For(&cachev1alpha1.Memcached{}). Owns(&appsv1.Deployment{}). Owns(&corev1.Service{}). Watches( &source.Kind{Type: &corev1.Pod{}}, &enqueueRequestsFromLabel{}, &builder.WithPredicates(&hasMemcachedLabelFilter{}), ). Complete(r) }
  117. 116

  118. 117 Service FirewallRule ownerRef:

  119. 118 Service FirewallRule ownerRef:

  120. 119 Service FirewallRule ownerRef:

  121. 120 FirewallRule ownerRef:

  122. 121 FirewallRule ownerRef:

  123. 122

  124. 123

  125. 124

  126. 125

  127. 126

  128. 127

  129. 128 FirewallRule

  130. 129

  131. 130 FirewallRule

  132. 131 FirewallRule Finalizers:

  133. 132 FirewallRule Finalizers: DeletionTimestamp: FirewallRule Finalizers: DeletionTimestamp:

  134. 133 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status
  135. 134 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion
  136. 135 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion const FinalizerName = "finalizer.example.com/firewall" func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // Fetch FirewallRule... if rule.GetDeletionTimestamp() != nil { // If we have not done deletion yet if contain(rule.Finalizers, FinalizerName) { // Perform deletion... return reconcile.Result{}, r.removeFinalizer(rule) } } else { // If we haven't add finalizer yet if !contain(rule.Finalizers, FinalizerName) { r.addFinalizer(rule) } } // ... }
  137. 136 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion const FinalizerName = "finalizer.example.com/firewall" func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // Fetch FirewallRule... if rule.GetDeletionTimestamp() != nil { // If we have not done deletion yet if contain(rule.Finalizers, FinalizerName) { // Perform deletion... return reconcile.Result{}, r.removeFinalizer(rule) } } else { // If we haven't add finalizer yet if !contain(rule.Finalizers, FinalizerName) { r.addFinalizer(rule) } } // ... } Need to be deleted
  138. 137 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion const FinalizerName = "finalizer.example.com/firewall" func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // Fetch FirewallRule... if rule.GetDeletionTimestamp() != nil { // If we have not done deletion yet if contain(rule.Finalizers, FinalizerName) { // Perform deletion... return reconcile.Result{}, r.removeFinalizer(rule) } } // ... } Not finalized yet
  139. 138 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion const FinalizerName = "finalizer.example.com/firewall" func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // Fetch FirewallRule... if rule.GetDeletionTimestamp() != nil { // If we have not done deletion yet if contain(rule.Finalizers, FinalizerName) { // Perform deletion... return reconcile.Result{}, r.removeFinalizer(rule) } } // ... } Remember to remove finalizer
  140. 139 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion const FinalizerName = "finalizer.example.com/firewall" func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // Fetch FirewallRule... else { // If we haven't add finalizer yet if !contain(rule.Finalizers, FinalizerName) { r.addFinalizer(rule) } } // ... } Not add finalizer yet
  141. 140 FirewallRule Finalizers: DeletionTimestamp: Reconcile Fetch Resource Create Deployment Fetch

    Deployment Reconcile Deployment Update Resource Status Add Finalizer Perform Deletion const FinalizerName = "finalizer.example.com/firewall" func (r *Reconciler) Reconcile(request reconcile.Request) ( reconcile.Result, error) { // Fetch FirewallRule... else { // If we haven't add finalizer yet if !contain(rule.Finalizers, FinalizerName) { r.addFinalizer(rule) } } // ... } Remember to add finalizer
  142. 141 Reconcile Fetch Resource Create Deployment Fetch Deployment Reconcile Deployment

    Update Resource Status Add Finalizer Perform Deletion
  143. 142

  144. 143 StatefulSet replicas: 3 vcTemplate:

  145. 144 Pod 1 Pod 2 Pod 3 StatefulSet replicas: 3

    vcTemplate: PVC 1 PVC 2 PVC 3 PV 1 PV 2 PV 3
  146. 145 Pod 1 Pod 2 Pod 3 StatefulSet replicas: 3

    vcTemplate: PVC 1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F
  147. 146 Pod 1 Pod 2 Pod 3 StatefulSet replicas: 2

    vcTemplate: PVC 1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F
  148. 147 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F
  149. 148 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F
  150. 149 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F
  151. 150 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F Drainer Drainer
  152. 151 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F Drainer Drainer
  153. 152 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PVC 3 PV 1 PV 2 PV 3 A B C D E F
  154. 153 Pod 1 Pod 2 StatefulSet replicas: 2 vcTemplate: PVC

    1 PVC 2 PV 1 PV 2 A B C D E F
  155. 154

  156. 155

  157. 156

  158. 157

  159. š 158

  160. 2020 Image credit: © COSCUP, CC-BY-SA 4.0