Automating Kubernetes Cluster Operations with Operators

AWS Community Tech Summit Cologne

At Giant Swarm, we manage Kubernetes clusters for customers 24/7, both on-premises and in the cloud. That means we do not just set something up and hand it over, but we actually take care that it’s operational and up-to-date at all times.
In this talk Timo explains how Giant Swarm are using Operators to codify all operational tasks of managing Kubernetes cluster and distributed applications on top. The operators manage PKI infrastructures, networks, VMs and storage both on-premises and in the cloud. There have been a lots of challenges and learnings in the past year and Timo would like to share them with you.

Transcript

1. Automating Kubernetes Cluster Operations with Operators by Timo Derstappen
2. Running Kubernetes in Kubernetes
3. Are you in desired state?
4. Overview 1. Background / Motivation 2. Overview 3. Operators 4. Outlook
5. Background At Giant Swarm we manage Kubernetes clusters for customers 24/7. Both on bare metal and in the cloud.
6. Distributed Team People are in 8 different european countries Always looking for more awesome engineers
7. Kubernetes as a Service (KaaS) Deployment within our own AWS account. Get a cluster within a few minutes! Full root access to clusters
9. On-premises KaaS Current focus is to manage Kubernetes in the DCs or AWS accounts of the customers itself.
10. Open Source Our tooling is open-source and we are working with the community to improve kubernetes operations.
11. Multiple Clusters "Soft" multi-tenancy not enough in enterprise context PCI-compliance, privacy laws, etc. Different environments: separation of dev, test, prod Test new K8s versions
12. People must come to things in their own time, in their own way, for their own reasons, or they never truly come at all. - Dee Hock
13. RBAC, NetworkPolicies FTW We encourage people to use these features. Small clusters make little sense Support while processes need to be adapted Kubernetes matures - trust over time
14. Introducing: Giantnetes
15. Motivation Rather obvious. If you provide others with Kubernetes because you think the building blocks are right, you just want to have it too.
16. De nition G8s: Giantnetes K8s: Kubernetes == Guest cluster
17. Overview 1. Background / Motivation 2. Overview 3. Operators 4. Outlook
20. Immutability Container linux (Hosts and Guests) Continuous pipeline (Container build, registry, kubernetes)
21. Reproducable builds and deployments at any point in time Architect ( ) Draughtsman ( ) https://github.com/giantswarm/architect https://github.com/giantswarm/draughtsman
23. Every manual change is ephemeral
24. Mean time to recovery!
25. Networking Separate VPC for each cluster Peered with the VPC of the control plane Calico between containers Network policy controller
26. Certi cates Each cluster has its own root ca Using PKI backends in Vault
27. Certi cates #2 Kubernetes components use mutual TLS Client certificates are used for RBAC
28. Overview 1. Background / Motivation 2. Overview 3. Operators 4. Outlook
29. Why are we doing this again? Fully automate deployment of kubernetes Continuously manage desired state of the clusters
30. Iterations of the platform 1. K8s clusters via systemd units over fleet 2. K8s manifests to create K8s clusters used as templates 3. Operators manage a clusters lifecycle continuously
31. Operators Codify all operational tasks Manage desired state
32. Third-Party Resources Custom Resource De nitions Easily extend the Kubernetes API
33. Operate all the things aws-operator kvm-operator cert-operator flannel-operator ingress-operator
34. AWS operator Create k8s clusters on AWS based on a TPR
41. The TPRs de ne Kubernetes configuration/version Ingress configuration/version Network configuration Master/Node configurations Certificates
42. The Cert TPR de nes TTL Common Name SANs Cluster / Component
43. Overview 1. Background / Motivation 2. Overview 3. Operators 4. Outlook
44. Micro Operators Operators should have single responsibilities. Keep them simple and maintainable
45. Operatorkit Our services are based upon microkit. We've also created operatorkit to reduce boilerplate in our operators and collect them in a library. https://github.com/giantswarm/operatorkit
46. Self-hosted Having the control plane and all tenant clusters running self-hosted will further ease the lifecycle management of the clusters.
48. Thank you. Timo Derstappen CTO of Giant Swarm @teemow