Automating Kubernetes Cluster Operations with Operators

At Giant Swarm, we manage Kubernetes clusters for customers 24/7, both on-premises and in the cloud. That means we do not just set something up and hand it over, but we actually take care that it’s operational and up-to-date at all times. This talk explains how Giant Swarm are using Operators to codify all operational tasks of managing Kubernetes cluster and distributed applications on top. The operators manage PKI infrastructures, networks, VMs and storage both on-premises and in the cloud.

Transcript

1. Automating Kubernetes Cluster Operations with Operators by Timo Derstappen
2. Running Kubernetes in Kubernetes
3. Are you in desired state?
4. Overview 1. Background / Motivation 2. Architecture 3. Operators 4. Outlook
5. Background At Giant Swarm we manage Kubernetes clusters for customers 24/7. Both on bare metal and in the cloud.
6. Distributed Team People are in 6 different european countries Looking for more platform engineers and SREs
7. Kubernetes as a Service (KaaS) Own DC in Frankfurt, Germany Full access to clusters
9. On-premises KaaS Current focus is to manage Kubernetes in the DCs or AWS accounts of the customers itself.
10. Open Source Our tooling is open-source and we are working with the community to improve kubernetes operations.
11. Multiple Clusters "Soft" multi-tenancy not enough in enterprise context PCI-compliance, privacy laws, etc. Different environments: separation of dev, test, prod Test new K8s versions
12. People must come to things in their own time, in their own way, for their own reasons, or they never truly come at all. - Dee Hock
13. RBAC, NetworkPolicies FTW We encourage to use the features and integrate. Small clusters make little sense Support while processes need to be adapted Kubernetes matures - trust over time
14. Introducing: Giantnetes
15. Motivation Rather obvious. If you provide others with Kubernetes because you think the building blocks are right, you just have to use it too.
16. De nition G8s: Giantnetes K8s: Kubernetes == Guest cluster
17. Overview 1. Background / Motivation 2. Architecture 3. Operators 4. Outlook
20. Immutability Container linux (Hosts and Guests) Continuous pipeline (Container build, registry, kubernetes)
21. Reproducable builds and deployments at any point in time Architect ( ) Draughtsman ( ) https://github.com/giantswarm/architect https://github.com/giantswarm/draughtsman
22. Every manual change is ephemeral
23. Mean time to recovery!
24. Networking Flannel/VPC network between guest cluster nodes Calico between containers Network policy controller
29. Certi cates Each cluster has its own root ca Using PKI backends in Vault Certificates are rotated every day
30. Certi cates #2 Kubernetes components use mutual TLS Client certificates are used for RBAC
31. Overview 1. Background / Motivation 2. Architecture 3. Operators 4. Outlook
32. Why are we doing this again? Fully automate deployment of kubernetes Continuously manage desired state of the clusters
33. Iterations of the platform 1. K8s clusters via systemd units over fleet 2. K8s manifests to create K8s clusters used as templates 3. Writing operators
34. Operators Codify all operational tasks Manage desired state
35. Third-Party Resources Custom Resource De nitions Easily extend the Kubernetes API
36. Operate all the things aws-operator kvm-operator cert-operator flannel-operator ingress-operator
37. AWS operator Create k8s clusters on AWS based on a TPR
44. The TPRs de ne Kubernetes configuration/version Ingress configuration/version Network configuration Master/Node configurations Certificates
45. The Cert TPR de nes TTL Common Name SANs Cluster / Component
46. Overview 1. Background / Motivation 2. Architecture 3. Operators 4. Outlook
47. Micro Operators Operators should have single responsibilities. Keep them simple and maintainable
48. Operatorkit Our services are based upon microkit. We've also created operatorkit to reduce boilerplate in our operators and collect them in a library. https://github.com/giantswarm/operatorkit
49. Self-hosted First step Having Giantnetes and all guest clusters running self-hosted will further ease the lifecycle management of the clusters.
51. Thank you. Timo Derstappen CTO of Giant Swarm @teemow