Extending Kubernetes

What’s a Kubernetes Service
Building blocks for a service
Configuration Plane
Custom Resource Definition

Transcript

1. Architecting Kubernetes Services
2. • Service extends Kubernetes with additional functionality • K8s at core is an orchestration platform build to manage your containerized loads • Once you have your application running on k8s, then arrives the need for features like scaling, monitoring, migration, backup, etc. • K8s has an active community contributing to it, but they cannot build everything on their own. • That’s where people start building external services. Istio is such a service! • At Trilio we have built a service which handles Backup, Restore, Disaster Recovery and Migration for any workloads. • We have taken a Kubernetes native approach while building our service and this talk references our learning from the same What’s a Kubernetes Service
3. Building blocks for a service • Configuration Plane • Management Plane • API Server • Task Queue + Message Broker • Scheduler • RBAC • CLI • Packaging
4. • Any service comes up with certain functionality • User of the service needs a way to use/configure that functionality • A well defined way of doing this is to use API’s • K8s has it’s inbuilt API layer, but it also allows you to extend it using Custom Resource Definition (CRD) • You may or may not require to define a CRD for your service as there is a rich set of API available in k8s for your use. • We will focus this discussion considering CRD as our configuration plane Configuration Plane
5. •As mentioned earlier CRD’s are a way to extend k8s API •In k8s every resource is a first class citizen. •CRD’s also become a first class citizen. Once you define a CRD, you get all the features that you get for a inbuilt k8s API resource •You can do CRUD operation using kubectl / API calls to kube-apiserver •Define relationships between CR’s using Object Reference or Owner Reference Custom Resource Definition
6. apiversion: kafka.strimzi.io/v1beta1 kind: kafkatopic metadata: name: my-topic labels: strimzi.io/cluster: "my-cluster" spec: partitions: 3 replicas: 1 Sample CRD
7. •If your are using CRD, there is no need to define your own API server layer •kube-apiserver serves as your API layer •K8s already has a caching layer using Shared Informer so that you don’t have to hit the etcd for each of your request •Use Admission Webhooks (Validation and Mutation) to enforce any constraints or to add additional info to your CR’s •All your sorting, filtering, pagination requirements are already handled by kube-apiserver kube-apiserver is your API server
8. •Controllers can reconcile on one or many CRD’s •Controller is responsible for making the current state of a resource closer to the desired state •Every edit to your CR creates an event which is available for your controller to consume. That is called reconciliation •It is recommended to maintain a state field in your CRD spec, so that you can put your reconcile logic based on the current and the desired state Controllers are your Management Plane
10. •K8s comes with a shared informer which is a queue shared across your cluster •Every resource object or CR that you create/edit gets queued into the shared informer •Controllers act as your broker which will pick up objects from the queue based on the type of resource and the state of the resource •If your controller goes down for some reason, your queue remains intact •If your cluster goes down, queue can be rebuilt •Controller acquires a mutation lock on a queue object as soon as it starts processing it. So in case you are running multiple replicas of your controller, there is a guarantee that the object will be processed only once, making your transaction idempotent No need for Task Queue or Message Broker
11. • You may have a use case where you want to schedule any operation • Scheduler is an inbuilt resource in k8s, you don’t need to define your own scheduler logic • Works on cron schema • Using the Job resource you can define the business logic that you need to perform when the schedule is triggered K8s has inbuilt scheduler
12. Need access control, k8s has got RBAC • K8s has a matrued RBAC engine • You can define your RBAC policies at the resource, namespace and cluster level • RBAC policies for user and controller may need to differ. It can be segregated through Service Account
13. CRD’s are available through kubectl • Kubectl command works same for all types of resources including your Custom Resources • Kubectl command supports CRUD operations on each of them • Everything that you can perform through kube-apiserver can be performed through kubectl • Customization the kubectl command output is also possible • So in essence you don’t need to write your own CLI
14. Packaging - Helm • Helm has evolved as the de facto for packaging and distributing your application • Helm basically comprises of charts and values • Charts are your default configurations and values are used as the replacement of specific placeholders within your charts • You can manage your releases using combination of charts and values • Helm can be used to package and distribute your CRD’s and their controllers
15. •Helm is ok for deploying your application, but you need one layer on top to manage your application •Consider a scenario where a deployment created by your helm chart is deleted or a scenario where you want to scale your application on need basis •You can do it manually. But manual handling is pain and error prone •That’s where operator comes into picture •Operator is nothing but a controller intended to manage the state of your application •Like other controllers operator are driven by CRD •You define a CR for your application and operator will work to manage the state of your application based on the CR configuration Packaging - Operator
17. •Kubebuilder is a framework for building Kubernetes APIs using CRD •Building Kubernetes tools and APIs involves making a lot of decisions and writing a lot of boilerplate. •In order to facilitate easily building Kubernetes APIs, Kubebuilder provides a collection of Kubernetes development tools to minimize toil. •Using Kubebuilder you get boilerplate code for • Creating CRD • Creating Controller for the CRD • Creating Validation and Mutation Webhooks • Test by running against a cluster Kubebuilder makes this all easy
18. • K8s is a much matured platform now • With the thriving ecosystem there are a lot of integrations available • With the ability of extending k8s API through CRD there is much you can achieve without involving external components • So if you are building a k8s service then its better to build it k8s native. Conclusion