Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Controllers and Service Catalog; The Power of CRDs

Controllers and Service Catalog; The Power of CRDs

Custom Resource Definitions (CRDs) make it easy for you to add new types of primitives that can be managed by the internal control loop of Kubernetes. In this presentation, Chris Hein will provide an in-depth look at how he has been using CRDs to model external services and some of the benefits and pitfalls of using this style. We'll begin by diving into what CRDs are and how they can be used. Why he choose to model cloud resources using CRDs as opposed to Service Catalog. Continuing into a review of what it's like to build using the control loop and we'll end by discussing golang & Kubernetes code generation and how you can speed up your development by using these tools.

Christopher Hein

November 17, 2019

More Decks by Christopher Hein

Other Decks in Programming


  1. Controllers & Service Catalog; The Power of CRDs Chris Hein

    CNCF Ambassador [email protected] @christopherhein
  2. @christopherhein Purpose of this talk

  3. @christopherhein I want you to build controllers…

  4. @christopherhein Why?

  5. @christopherhein because you can

  6. @christopherhein …and because they can help you.

  7. @christopherhein Some Basics…

  8. @christopherhein What is Kubernetes?

  9. @christopherhein {api}

  10. @christopherhein Show me $ kubectl get pods -v=7 GET https://apiserver/api/v1/namespaces/default/pods?limit=500

    Request Headers: 
 Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json 
 User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216 Response Status: 200 OK in 145 milliseconds
 Response Headers:
 Content-Type: application/json
 Content-Length: 1909
 Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634 
 Response Body: ...
  11. @christopherhein What does it expose?

  12. @christopherhein Kubernetes objects

  13. @christopherhein bindings, componentstatuses, configmaps, endpoints, events, limitranges, namespaces, nodes, persistentvolumeclaims,

    persistentvolumes, pods, podtemplates, replicationcontrollers, resourcequotas, secrets, serviceaccounts, services, mutatingwebhookconfigurations.admissionregistration.k8s.io, validatingwebhookconfigurations.admissionregistration.k8s.io, customresourcedefinitions.apiextensions.k8s.io, apiservices.apiregistration.k8s.io, controllerrevisions.apps, daemonsets.apps, deployments.apps, replicasets.apps, statefulsets.apps, tokenreviews.authentication.k8s.io, localsubjectaccessreviews.authorization.k8s.io, selfsubjectaccessreviews.authorization.k8s.io, selfsubjectrulesreviews.authorization.k8s.io, subjectaccessreviews.authorization.k8s.io, horizontalpodautoscalers.autoscaling, cronjobs.batch, jobs.batch, certificatesigningrequests.certificates.k8s.io, leases.coordination.k8s.io, events.events.k8s.io, daemonsets.extensions, deployments.extensions, ingresses.extensions, networkpolicies.extensions, podsecuritypolicies.extensions, replicasets.extensions, ingresses.networking.k8s.io, networkpolicies.networking.k8s.io, runtimeclasses.node.k8s.io, poddisruptionbudgets.policy, podsecuritypolicies.policy, clusterrolebindings.rbac.authorization.k8s.io, clusterroles.rbac.authorization.k8s.io, rolebindings.rbac.authorization.k8s.io, roles.rbac.authorization.k8s.io, priorityclasses.scheduling.k8s.io, csidrivers.storage.k8s.io, csinodes.storage.k8s.io, storageclasses.storage.k8s.io, volumeattachments.storage.k8s.io
  14. @christopherhein How do they work?

  15. @christopherhein State Feedback Reconcile

  16. @christopherhein …but how?

  17. @christopherhein Controllers

  18. @christopherhein attachdetach, bootstrapsigner, clusterrole- aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset,

    deployment, disruption, endpoint, garbagecollector, job, namespace, horizontalpodautoscaling, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv- protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after- finished bootstrapsigner, tokencleaner
  19. @christopherhein State Feedback Reconcile

  20. @christopherhein But this talk is about extending…

  21. @christopherhein We’re in luck…

  22. @christopherhein Kubernetes was built to be extensible

  23. @christopherhein API Aggregation

  24. @christopherhein Service Catalog

  25. @christopherhein Service Catalog Features • Standardized specification - https://openservicebrokerapi.org •

    Allows many “Service Brokers” to be configured • Manage external services and set up Application binding from k8s • Two Kinds to use - ServiceInstance & ServiceBinding*
  26. @christopherhein Service Catalog Example apiVersion: servicecatalog.k8s.io/v1beta1
 kind: ServiceInstance

 namespace: default
 clusterServiceClassExternalName: mysql-5-7-database
 clusterServicePlanExternalName: database
  27. @christopherhein Kubernetes Interfaces • CSI - Container Storage Interface -

    EFS, EBS, ScaleIO… • CRI - Container Runtime Interface - cri-o, Docker, rkt… • CNI - Container Network Interface - Calico, Cilium, Weave… • DPI - Device Plugin Interface - AMD GPU, KubeVirt … • SMI* - Service Mesh Interface - Linkerd, Maesh, Istio…
  28. @christopherhein and my personal favorite…

  29. @christopherhein Custom Resource Definitions (CRDs)

  30. @christopherhein Why you ask?

  31. @christopherhein Creates RESTFUL routes in the API Server $ kubectl

    get authorizedkeys christopherhein-key -o yaml apiVersion: node.hein.dev/v1alpha1
 kind: AuthorizedKey
 name: christopherhein-key
 selfLink: /apis/node.hein.dev/v1alpha1/authorizedkeys/christopherhein-key
 key: ssh-rsa ...
  32. @christopherhein Uses YAML and kubectl to create apiVersion: apiextensions.k8s.io/v1

 name: authorizedkeys.node.hein.dev
 group: node.hein.dev
 kind: AuthorizedKey
 plural: authorizedkeys
 - name: v1alpha1
 served: true
 storage: true
  33. @christopherhein Validations built-in # ... CRD .spec

 type: string
 pattern: "#ssh-rsa AAAA[0-9A-Za-z+/]+[=]{0,3} ([^@][email protected][^@]+)#"
  34. @christopherhein .Status & .Scale Sub-resources # ... CRD .spec

 labelSelectorPath: .status.labelSelector
 specReplicasPath: .spec.deployments
 statusReplicasPath: .status.deployments
 status: {}
  35. @christopherhein Even more… • Pruning unknown fields • Defaulting •

    Multiple versions • Client side validation • Additional column printers for kubectl • CRDs graduated to GA in 1.16
  36. @christopherhein They’re the bees knees

  37. @christopherhein How do they work?

  38. @christopherhein Controllers

  39. @christopherhein Controllers (didn’t we learn about this?)

  40. @christopherhein Building a custom controller • They can run as

    a Pod • Leader election for High availability • Communicate with the API Server • Manage external resource OR internal resources • In-charge of reconciling Kubernetes objects including CRD types
  41. @christopherhein Best Practices

  42. @christopherhein .Metadata.OwnerReferences # ... CRD .metadata
 - apiVersion: ecr.awsctrl.io/v1alpha1

    controller: true
 blockOwnerDeletion: true
 kind: Repository
 name: awsctrl-repository
 uid: 01411f5c-665c-4849-aebc-c838289d603
  43. @christopherhein Incremental reconciliation # ... ctrl.Reconcile
 if ok := utils.ContainsFinalizer(obj.ObjectMeta,

    utils.Name); !ok {
 log.Info("Adding Stack Finalizer")
 if err := r.addCFNFinalizer(ctx, &instance); err != nil {
 return ctrl.Result{}, err
 return ctrl.Result{}, nil
  44. @christopherhein Dynamic configuration # ... ctrl.Reconcile
 var config selfv1alpha1.Config

    err := r.Get(ctx, req.NamespacedName, &config); err != nil {
 return ctrl.Result{}, err
 log.Info("Configuring the Client")
 if err := r.Client.Configure(&config.Spec.API); err != nil {
 return ctrl.Result{}, err
  45. @christopherhein Avoid split brain where possible o, err := cfn.CreateStack(c,

 if err != nil {
 return err
 } $ aws cloudformation update-stack \
 --stack-name "<name>" \
 # ... other params
  46. @christopherhein Use Namespaces Resources # ... .spec
 scope: Namespaced •

    Reduces security risks with lower RBAC permissions • Helps NaaS to work • Most resources don't need cluster scope*
  47. @christopherhein Use Kubernetes v1.Events # ... ctrl.Reconcile
 recorder.Event(obj, "Normal", "CreateStack",

    "User initiated")
 annos := map[string]string{"retried": "true"}
 recoder.AnnotatedEventf(obj, annos, "Normal", "DeleteStack", "")
  48. @christopherhein Report .Status # ... r.UpdateStatus
 stackstatus := cloudformationv1alpha1.StackStatus{

 StackID: stackID,
 Status: status,
 Message: &message,
 ObservedGeneration: instanceCopy.Generation,
 objCopy.Status = stackstatus
 r.Status().Update(ctx, &objCopy)
  49. @christopherhein Use multiple CRDs* apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition

 group: ecr.awsctrl.io
 kind: Repository
 listKind: RepositoryList
 plural: repositories
 singular: repository apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 name: stacks.cloudformation.awsctrl.io
 group: cloudformation.awsctrl.io
 kind: Stack
 listKind: StackList
 plural: stacks
 singular: stack
  50. @christopherhein Use an SDK/Framework $ kubebuilder init --domain awsctrl.io

    kubebuilder create api \
 --group ecr \
 --version v1alpha1 \
 --kind Repository $ operator-sdk new app-operator --repo awsctrl.io
 $ operator-sdk add api --api-version ecr.awsctrl.io/v1alpha1 --kind Repository
 $ operator-sdk add controller \
 --api-version ecr.awsctrl.io/v1alpha1 \
 --kind Repository
  51. @christopherhein Examples

  52. @christopherhein source: operatorhub.io

  53. Thanks! Chris Hein CNCF Ambassador [email protected] @christopherhein