Extending Kubernetes

If you have use-cases to extend Kubernetes, what are the means to do so? This Talk explains the existing extension points on a high-level, with reference to a demo project (available on GitHub) showing how to implement custom resource definitions (CRDs) and custom controllers utilizing the Kubebuilder framework.

Originally presented at the "Kubernetes / Cloud Native Meetup Hamburg" in November 2018.

Transcript

1. Extending Kubernetes Photo by Dominik Schröder on Unsplash
2. About me • Senior Software Engineer @ ePages • > 20 years development experience • > 3 years running on & operating Kubernetes • Contact d.jablonski@epages.com @dirkjablonski djablonski
3. Extending Kubernetes Photo by Evan Dennis on Unsplash Why?
4. Automate recurring tasks
5. Enforce Custom Policies
6. Enable new Use-Cases
7. EXTENSION POINTS Photo by John Carlisle on Unsplash
8. Custom Resource Definitions (CRDs) apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: foos.samplecontroller.k8s.io spec: group: samplecontroller.k8s.io version: v1alpha1 names: kind: Foo plural: foos scope: Namespaced • Kubernetes objects defining custom object types • Once registered, allow to create objects of that type • Left alone, are just fancy storage
9. Custom Controller • Just “plain” applications • deployed within your cluster • interacting with the API server • Usage examples: • Register services/deployments with monitoring annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9102'
10. Controller Event Model Image by Thomas Stringer on medium.com
11. CRDs + Custom Controller • Combined, release the full power of CRDs • Custom Controller reacts on CRD object lifecycle • Example Use-Cases: • Routing information in Service Meshes • Certificate Management
12. Operators • Technically the same as Custom Controller + CRD • Used for full applications • Include operational knowledge • Can support smart operations, e.g. scaling • Example Use-Cases: • etcd Operator • Prometheus Operator • Vault Operator • …
13. Admission Controllers • Intercept requests to K8s API prior to persisting • Can be mutating or validating (or both) • Are compiled into the API- server • Therefore very limited usage • Example Use-Cases: • AlwaysPullImages • DefaultStorageClass • DenyEscalatingExec
14. Admission Control Webhooks • Similar to admission controllers, but registered dynamically • HTTP callbacks which receive admission requests (prior to requests being persisted) • Are called by Admission Controllers (need to be activated) • Also have two different types: Mutating Admission Webhook • Used to change requests, e.g. to enforce custom defaults Validating Admission Webhook • Used to enforce custom admission policies
15. Request Handling Lifecycle HTTP Handler Auth Mutating Admission Object Schema Validation Validating Admission Persisted To etcd Mutating Webhooks Vaildating Webhooks
16. AdmissionRequest • Request contains information about the requested operation type AdmissionRequest struct { UID types.UID Kind metav1.GroupVersionKind Resource metav1.GroupVersionResource SubResource string Name string Namespace string Operation Operation UserInfo authentication.UserInfo Object runtime.Object OldObject runtime.Object DryRun *bool }
17. AdmissionResponse • Contains your response to the requested operation (vote or patch for request) type AdmissionResponse struct { UID types.UID Allowed bool Result *metav1.Status Patch []byte PatchType *PatchType AuditAnnotations map[string]string }
18. Webhook Registration apiVersions: - v1 resources: - pods failurePolicy: Ignore clientConfig: service: namespace: default name: webhook caBundle: apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingAdmissionHookConfiguration metadata: name: myConfig webhooks: - name: pod-policy.example.io rules: - operations: - CREATE apiGroups: - ""
19. Aggregated API Servers • Offer additional API objects • Register an APIService “claiming” specific path • Need to be enabled in Kubernetes API-Server • Limited usefulness for individual use cases • Example Use-Case: Service Catalog
20. DEMO TIME Photo by NordWood Themes on Unsplash
21. Motivation • Application with Microservice architecture • Lots of services needing a database • Re-using database instances, but different logical databases • Admin-access to shared DB instance not available to devs
22. k8s-db-controller Idea: • Let microservices declare their need for a logical database (i.e. schema): • CustomResourceDefinition “DatabaseSchema” • Microservices need generated access data • Generate Secret • Let a Custom Controller do the work!
23. Tooling: Kubebuilder • Framework for building Kubernetes APIs using CRDs • Generates project structure & Go code (plus some other resources) • Spares you from writing a lot of boilerplate code • Also supports Admission Webhooks
24. Action
25. SUMMARY Photo by Max Langelott on Unsplash
26. Most useful extension points (imho) CRDs + Custom Controllers for • automating recurring tasks • advanced use-cases Admission Webhooks for • enforcing custom policies • automating recurring tasks
27. Not too hard to learn & implement • Sophisticated tooling support (Kubebuilder et al.) • Good documentation • Loads of existing third party resources (blog posts etc.)
28. RESOURCES Photo by Max Langelott on Unsplash
29. This talk: • Slides • Demo project on GitHub Software • Kubebuilder - SDK for building Kubernetes APIs using CRDs Books: • Managing Kubernetes • Kubernetes in Action
30. Kubernetes Documentation: • Custom Resources • Extend the Kubernetes API with CustomResourceDefinitions • Dynamic Admission Control • Extending the Kubernetes API with the aggregation layer Blogposts etc.: • Extending Kubernetes: Create Controllers for Core and Custom Resources • Getting Started with Kubernetes Validating Admission Webhooks the FaaS Way • Diving into Kubernetes MutatingAdmissionWebhook
31. QUESTIONS
32. THANK YOU!