Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Controllers and Service Catalog; The Power of CRDs

Controllers and Service Catalog; The Power of CRDs

Custom Resource Definitions (CRDs) make it easy for you to add new types of primitives that can be managed by the internal control loop of Kubernetes. In this presentation, Chris Hein will provide an in-depth look at how he has been using CRDs to model external services and some of the benefits and pitfalls of using this style. We'll begin by diving into what CRDs are and how they can be used. Why he choose to model cloud resources using CRDs as opposed to Service Catalog. Continuing into a review of what it's like to build using the control loop and we'll end by discussing golang & Kubernetes code generation and how you can speed up your development by using these tools.

Christopher Hein

November 17, 2019
Tweet

More Decks by Christopher Hein

Other Decks in Programming

Transcript

  1. Controllers & Service Catalog; The Power of CRDs Chris Hein

    CNCF Ambassador [email protected] @christopherhein
  2. @christopherhein Purpose of this talk

  3. @christopherhein I want you to build controllers…

  4. @christopherhein Why?

  5. @christopherhein because you can

  6. @christopherhein …and because they can help you.

  7. @christopherhein Some Basics…

  8. @christopherhein What is Kubernetes?

  9. @christopherhein {api}

  10. @christopherhein Show me $ kubectl get pods -v=7 GET https://apiserver/api/v1/namespaces/default/pods?limit=500

    Request Headers: 
 Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json 
 User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216 Response Status: 200 OK in 145 milliseconds
 Response Headers:
 Content-Type: application/json
 Content-Length: 1909
 Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634 
 Response Body: ...
  11. @christopherhein What does it expose?

  12. @christopherhein Kubernetes objects

  13. @christopherhein bindings, componentstatuses, configmaps, endpoints, events, limitranges, namespaces, nodes, persistentvolumeclaims,

    persistentvolumes, pods, podtemplates, replicationcontrollers, resourcequotas, secrets, serviceaccounts, services, mutatingwebhookconfigurations.admissionregistration.k8s.io, validatingwebhookconfigurations.admissionregistration.k8s.io, customresourcedefinitions.apiextensions.k8s.io, apiservices.apiregistration.k8s.io, controllerrevisions.apps, daemonsets.apps, deployments.apps, replicasets.apps, statefulsets.apps, tokenreviews.authentication.k8s.io, localsubjectaccessreviews.authorization.k8s.io, selfsubjectaccessreviews.authorization.k8s.io, selfsubjectrulesreviews.authorization.k8s.io, subjectaccessreviews.authorization.k8s.io, horizontalpodautoscalers.autoscaling, cronjobs.batch, jobs.batch, certificatesigningrequests.certificates.k8s.io, leases.coordination.k8s.io, events.events.k8s.io, daemonsets.extensions, deployments.extensions, ingresses.extensions, networkpolicies.extensions, podsecuritypolicies.extensions, replicasets.extensions, ingresses.networking.k8s.io, networkpolicies.networking.k8s.io, runtimeclasses.node.k8s.io, poddisruptionbudgets.policy, podsecuritypolicies.policy, clusterrolebindings.rbac.authorization.k8s.io, clusterroles.rbac.authorization.k8s.io, rolebindings.rbac.authorization.k8s.io, roles.rbac.authorization.k8s.io, priorityclasses.scheduling.k8s.io, csidrivers.storage.k8s.io, csinodes.storage.k8s.io, storageclasses.storage.k8s.io, volumeattachments.storage.k8s.io
  14. @christopherhein How do they work?

  15. @christopherhein State Feedback Reconcile

  16. @christopherhein …but how?

  17. @christopherhein Controllers

  18. @christopherhein attachdetach, bootstrapsigner, clusterrole- aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset,

    deployment, disruption, endpoint, garbagecollector, job, namespace, horizontalpodautoscaling, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv- protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after- finished bootstrapsigner, tokencleaner
  19. @christopherhein State Feedback Reconcile

  20. @christopherhein But this talk is about extending…

  21. @christopherhein We’re in luck…

  22. @christopherhein Kubernetes was built to be extensible

  23. @christopherhein API Aggregation

  24. @christopherhein Service Catalog

  25. @christopherhein Service Catalog Features • Standardized specification - https://openservicebrokerapi.org •

    Allows many “Service Brokers” to be configured • Manage external services and set up Application binding from k8s • Two Kinds to use - ServiceInstance & ServiceBinding*
  26. @christopherhein Service Catalog Example apiVersion: servicecatalog.k8s.io/v1beta1
 kind: ServiceInstance
 metadata:
 name:

    example-mysql-database-instance
 namespace: default
 spec:
 clusterServiceClassExternalName: mysql-5-7-database
 clusterServicePlanExternalName: database
  27. @christopherhein Kubernetes Interfaces • CSI - Container Storage Interface -

    EFS, EBS, ScaleIO… • CRI - Container Runtime Interface - cri-o, Docker, rkt… • CNI - Container Network Interface - Calico, Cilium, Weave… • DPI - Device Plugin Interface - AMD GPU, KubeVirt … • SMI* - Service Mesh Interface - Linkerd, Maesh, Istio…
  28. @christopherhein and my personal favorite…

  29. @christopherhein Custom Resource Definitions (CRDs)

  30. @christopherhein Why you ask?

  31. @christopherhein Creates RESTFUL routes in the API Server $ kubectl

    get authorizedkeys christopherhein-key -o yaml apiVersion: node.hein.dev/v1alpha1
 kind: AuthorizedKey
 metadata:
 name: christopherhein-key
 selfLink: /apis/node.hein.dev/v1alpha1/authorizedkeys/christopherhein-key
 data:
 key: ssh-rsa ...
  32. @christopherhein Uses YAML and kubectl to create apiVersion: apiextensions.k8s.io/v1
 kind:

    CustomResourceDefinition
 metadata:
 name: authorizedkeys.node.hein.dev
 spec:
 group: node.hein.dev
 names:
 kind: AuthorizedKey
 plural: authorizedkeys
 versions:
 - name: v1alpha1
 served: true
 storage: true
  33. @christopherhein Validations built-in # ... CRD .spec
 validation:
 openAPIV3Schema:
 properties:


    spec:
 properties:
 key:
 type: string
 pattern: "#ssh-rsa AAAA[0-9A-Za-z+/]+[=]{0,3} ([^@][email protected][^@]+)#"
  34. @christopherhein .Status & .Scale Sub-resources # ... CRD .spec
 subresources:


    scale:
 labelSelectorPath: .status.labelSelector
 specReplicasPath: .spec.deployments
 statusReplicasPath: .status.deployments
 status: {}
  35. @christopherhein Even more… • Pruning unknown fields • Defaulting •

    Multiple versions • Client side validation • Additional column printers for kubectl • CRDs graduated to GA in 1.16
  36. @christopherhein They’re the bees knees

  37. @christopherhein How do they work?

  38. @christopherhein Controllers

  39. @christopherhein Controllers (didn’t we learn about this?)

  40. @christopherhein Building a custom controller • They can run as

    a Pod • Leader election for High availability • Communicate with the API Server • Manage external resource OR internal resources • In-charge of reconciling Kubernetes objects including CRD types
  41. @christopherhein Best Practices

  42. @christopherhein .Metadata.OwnerReferences # ... CRD .metadata
 ownerReferences:
 - apiVersion: ecr.awsctrl.io/v1alpha1


    controller: true
 blockOwnerDeletion: true
 kind: Repository
 name: awsctrl-repository
 uid: 01411f5c-665c-4849-aebc-c838289d603
  43. @christopherhein Incremental reconciliation # ... ctrl.Reconcile
 if ok := utils.ContainsFinalizer(obj.ObjectMeta,

    utils.Name); !ok {
 log.Info("Adding Stack Finalizer")
 if err := r.addCFNFinalizer(ctx, &instance); err != nil {
 return ctrl.Result{}, err
 }
 return ctrl.Result{}, nil
 }
  44. @christopherhein Dynamic configuration # ... ctrl.Reconcile
 var config selfv1alpha1.Config
 if

    err := r.Get(ctx, req.NamespacedName, &config); err != nil {
 return ctrl.Result{}, err
 }
 
 log.Info("Configuring the Client")
 if err := r.Client.Configure(&config.Spec.API); err != nil {
 return ctrl.Result{}, err
 }
  45. @christopherhein Avoid split brain where possible o, err := cfn.CreateStack(c,

    obj)
 if err != nil {
 return err
 } $ aws cloudformation update-stack \
 --stack-name "<name>" \
 # ... other params
  46. @christopherhein Use Namespaces Resources # ... .spec
 scope: Namespaced •

    Reduces security risks with lower RBAC permissions • Helps NaaS to work • Most resources don't need cluster scope*
  47. @christopherhein Use Kubernetes v1.Events # ... ctrl.Reconcile
 recorder.Event(obj, "Normal", "CreateStack",

    "User initiated")
 
 annos := map[string]string{"retried": "true"}
 recoder.AnnotatedEventf(obj, annos, "Normal", "DeleteStack", "")
  48. @christopherhein Report .Status # ... r.UpdateStatus
 stackstatus := cloudformationv1alpha1.StackStatus{
 StatusMeta:

    metav1alpha1.StatusMeta{
 StackID: stackID,
 Status: status,
 Message: &message,
 ObservedGeneration: instanceCopy.Generation,
 },
 }
 objCopy.Status = stackstatus
 r.Status().Update(ctx, &objCopy)
  49. @christopherhein Use multiple CRDs* apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 metadata:
 name:

    repositories.ecr.awsctrl.io
 spec:
 group: ecr.awsctrl.io
 names:
 kind: Repository
 listKind: RepositoryList
 plural: repositories
 singular: repository apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 metadata:
 name: stacks.cloudformation.awsctrl.io
 spec:
 group: cloudformation.awsctrl.io
 names:
 kind: Stack
 listKind: StackList
 plural: stacks
 singular: stack
  50. @christopherhein Use an SDK/Framework $ kubebuilder init --domain awsctrl.io
 $

    kubebuilder create api \
 --group ecr \
 --version v1alpha1 \
 --kind Repository $ operator-sdk new app-operator --repo awsctrl.io
 $ operator-sdk add api --api-version ecr.awsctrl.io/v1alpha1 --kind Repository
 $ operator-sdk add controller \
 --api-version ecr.awsctrl.io/v1alpha1 \
 --kind Repository
  51. @christopherhein Examples

  52. @christopherhein source: operatorhub.io

  53. Thanks! Chris Hein CNCF Ambassador [email protected] @christopherhein