Controllers and Service Catalog; The Power of CRDs

Custom Resource Definitions (CRDs) make it easy for you to add new types of primitives that can be managed by the internal control loop of Kubernetes. In this presentation, Chris Hein will provide an in-depth look at how he has been using CRDs to model external services and some of the benefits and pitfalls of using this style. We'll begin by diving into what CRDs are and how they can be used. Why he choose to model cloud resources using CRDs as opposed to Service Catalog. Continuing into a review of what it's like to build using the control loop and we'll end by discussing golang & Kubernetes code generation and how you can speed up your development by using these tools.

Transcript

1. Controllers & Service Catalog; The Power of CRDs Chris Hein CNCF Ambassador hello@chrishein.com @christopherhein
2. @christopherhein Purpose of this talk
3. @christopherhein I want you to build controllers…
4. @christopherhein Why?
5. @christopherhein because you can
6. @christopherhein …and because they can help you.
7. @christopherhein Some Basics…
8. @christopherhein What is Kubernetes?
9. @christopherhein {api}
10. @christopherhein Show me $ kubectl get pods -v=7 GET https://apiserver/api/v1/namespaces/default/pods?limit=500 Request Headers: 
 Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json 
 User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216 Response Status: 200 OK in 145 milliseconds
 Response Headers:
 Content-Type: application/json
 Content-Length: 1909
 Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634 
 Response Body: ...
11. @christopherhein What does it expose?
12. @christopherhein Kubernetes objects
13. @christopherhein bindings, componentstatuses, configmaps, endpoints, events, limitranges, namespaces, nodes, persistentvolumeclaims, persistentvolumes, pods, podtemplates, replicationcontrollers, resourcequotas, secrets, serviceaccounts, services, mutatingwebhookconfigurations.admissionregistration.k8s.io, validatingwebhookconfigurations.admissionregistration.k8s.io, customresourcedefinitions.apiextensions.k8s.io, apiservices.apiregistration.k8s.io, controllerrevisions.apps, daemonsets.apps, deployments.apps, replicasets.apps, statefulsets.apps, tokenreviews.authentication.k8s.io, localsubjectaccessreviews.authorization.k8s.io, selfsubjectaccessreviews.authorization.k8s.io, selfsubjectrulesreviews.authorization.k8s.io, subjectaccessreviews.authorization.k8s.io, horizontalpodautoscalers.autoscaling, cronjobs.batch, jobs.batch, certificatesigningrequests.certificates.k8s.io, leases.coordination.k8s.io, events.events.k8s.io, daemonsets.extensions, deployments.extensions, ingresses.extensions, networkpolicies.extensions, podsecuritypolicies.extensions, replicasets.extensions, ingresses.networking.k8s.io, networkpolicies.networking.k8s.io, runtimeclasses.node.k8s.io, poddisruptionbudgets.policy, podsecuritypolicies.policy, clusterrolebindings.rbac.authorization.k8s.io, clusterroles.rbac.authorization.k8s.io, rolebindings.rbac.authorization.k8s.io, roles.rbac.authorization.k8s.io, priorityclasses.scheduling.k8s.io, csidrivers.storage.k8s.io, csinodes.storage.k8s.io, storageclasses.storage.k8s.io, volumeattachments.storage.k8s.io
14. @christopherhein How do they work?
15. @christopherhein State Feedback Reconcile
16. @christopherhein …but how?
17. @christopherhein Controllers
18. @christopherhein attachdetach, bootstrapsigner, clusterrole- aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, garbagecollector, job, namespace, horizontalpodautoscaling, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv- protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after- finished bootstrapsigner, tokencleaner
19. @christopherhein State Feedback Reconcile
20. @christopherhein But this talk is about extending…
21. @christopherhein We’re in luck…
22. @christopherhein Kubernetes was built to be extensible
23. @christopherhein API Aggregation
24. @christopherhein Service Catalog
25. @christopherhein Service Catalog Features • Standardized specification - https://openservicebrokerapi.org • Allows many “Service Brokers” to be configured • Manage external services and set up Application binding from k8s • Two Kinds to use - ServiceInstance & ServiceBinding*
26. @christopherhein Service Catalog Example apiVersion: servicecatalog.k8s.io/v1beta1
 kind: ServiceInstance
 metadata:
 name: example-mysql-database-instance
 namespace: default
 spec:
 clusterServiceClassExternalName: mysql-5-7-database
 clusterServicePlanExternalName: database
27. @christopherhein Kubernetes Interfaces • CSI - Container Storage Interface - EFS, EBS, ScaleIO… • CRI - Container Runtime Interface - cri-o, Docker, rkt… • CNI - Container Network Interface - Calico, Cilium, Weave… • DPI - Device Plugin Interface - AMD GPU, KubeVirt … • SMI* - Service Mesh Interface - Linkerd, Maesh, Istio…
28. @christopherhein and my personal favorite…
29. @christopherhein Custom Resource Definitions (CRDs)
30. @christopherhein Why you ask?
31. @christopherhein Creates RESTFUL routes in the API Server $ kubectl get authorizedkeys christopherhein-key -o yaml apiVersion: node.hein.dev/v1alpha1
 kind: AuthorizedKey
 metadata:
 name: christopherhein-key
 selfLink: /apis/node.hein.dev/v1alpha1/authorizedkeys/christopherhein-key
 data:
 key: ssh-rsa ...
32. @christopherhein Uses YAML and kubectl to create apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
 name: authorizedkeys.node.hein.dev
 spec:
 group: node.hein.dev
 names:
 kind: AuthorizedKey
 plural: authorizedkeys
 versions:
 - name: v1alpha1
 served: true
 storage: true
33. @christopherhein Validations built-in # ... CRD .spec
 validation:
 openAPIV3Schema:
 properties:
 spec:
 properties:
 key:
 type: string
 pattern: "#ssh-rsa AAAA[0-9A-Za-z+/]+[=]{0,3} ([^@]+@[^@]+)#"
34. @christopherhein .Status & .Scale Sub-resources # ... CRD .spec
 subresources:
 scale:
 labelSelectorPath: .status.labelSelector
 specReplicasPath: .spec.deployments
 statusReplicasPath: .status.deployments
 status: {}
35. @christopherhein Even more… • Pruning unknown fields • Defaulting • Multiple versions • Client side validation • Additional column printers for kubectl • CRDs graduated to GA in 1.16
36. @christopherhein They’re the bees knees
37. @christopherhein How do they work?
38. @christopherhein Controllers
39. @christopherhein Controllers (didn’t we learn about this?)
40. @christopherhein Building a custom controller • They can run as a Pod • Leader election for High availability • Communicate with the API Server • Manage external resource OR internal resources • In-charge of reconciling Kubernetes objects including CRD types
41. @christopherhein Best Practices
42. @christopherhein .Metadata.OwnerReferences # ... CRD .metadata
 ownerReferences:
 - apiVersion: ecr.awsctrl.io/v1alpha1
 controller: true
 blockOwnerDeletion: true
 kind: Repository
 name: awsctrl-repository
 uid: 01411f5c-665c-4849-aebc-c838289d603
43. @christopherhein Incremental reconciliation # ... ctrl.Reconcile
 if ok := utils.ContainsFinalizer(obj.ObjectMeta, utils.Name); !ok {
 log.Info("Adding Stack Finalizer")
 if err := r.addCFNFinalizer(ctx, &instance); err != nil {
 return ctrl.Result{}, err
 }
 return ctrl.Result{}, nil
 }
44. @christopherhein Dynamic configuration # ... ctrl.Reconcile
 var config selfv1alpha1.Config
 if err := r.Get(ctx, req.NamespacedName, &config); err != nil {
 return ctrl.Result{}, err
 }
 
 log.Info("Configuring the Client")
 if err := r.Client.Configure(&config.Spec.API); err != nil {
 return ctrl.Result{}, err
 }
45. @christopherhein Avoid split brain where possible o, err := cfn.CreateStack(c, obj)
 if err != nil {
 return err
 } $ aws cloudformation update-stack \
 --stack-name "<name>" \
 # ... other params
46. @christopherhein Use Namespaces Resources # ... .spec
 scope: Namespaced • Reduces security risks with lower RBAC permissions • Helps NaaS to work • Most resources don't need cluster scope*
47. @christopherhein Use Kubernetes v1.Events # ... ctrl.Reconcile
 recorder.Event(obj, "Normal", "CreateStack", "User initiated")
 
 annos := map[string]string{"retried": "true"}
 recoder.AnnotatedEventf(obj, annos, "Normal", "DeleteStack", "")
48. @christopherhein Report .Status # ... r.UpdateStatus
 stackstatus := cloudformationv1alpha1.StackStatus{
 StatusMeta: metav1alpha1.StatusMeta{
 StackID: stackID,
 Status: status,
 Message: &message,
 ObservedGeneration: instanceCopy.Generation,
 },
 }
 objCopy.Status = stackstatus
 r.Status().Update(ctx, &objCopy)
49. @christopherhein Use multiple CRDs* apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 metadata:
 name: repositories.ecr.awsctrl.io
 spec:
 group: ecr.awsctrl.io
 names:
 kind: Repository
 listKind: RepositoryList
 plural: repositories
 singular: repository apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 metadata:
 name: stacks.cloudformation.awsctrl.io
 spec:
 group: cloudformation.awsctrl.io
 names:
 kind: Stack
 listKind: StackList
 plural: stacks
 singular: stack
50. @christopherhein Use an SDK/Framework $ kubebuilder init --domain awsctrl.io
 $ kubebuilder create api \
 --group ecr \
 --version v1alpha1 \
 --kind Repository $ operator-sdk new app-operator --repo awsctrl.io
 $ operator-sdk add api --api-version ecr.awsctrl.io/v1alpha1 --kind Repository
 $ operator-sdk add controller \
 --api-version ecr.awsctrl.io/v1alpha1 \
 --kind Repository
51. @christopherhein Examples
52. @christopherhein source: operatorhub.io
53. Thanks! Chris Hein CNCF Ambassador hello@chrishein.com @christopherhein