SLAC2024: 1001 Ways to Shoot Yourself in the Foot with On-Prem Kubernetes

Containers and associated orchestration platforms such as Docker and Kubernetes have become an integral part of the modern world of IT operations - provided you are running in the public cloud. But the further you move away from the public cloud, the more you enter a world of uncertainty and pain. With a bare metal setup, you are often trapped in a thicket of provisioning and deployment, storage and networking. In this talk, I share the lessons we learned while building our own on-premise Kubernetes platform - including all the pitfalls we stumbled upon and how to successfully run Kubernetes on your own metal.

Transcript

1. May 6th, 2024 MARTIN HELMICH @mittwald 1001 WAYS to SHOOT YOURSELF IN THE FOOT with ON - PREM KUBERNETES CC - BY Anja Vrečko
2. MARTIN HELMICH Head of Architecture & 
 Developer Relations Lecturer, Software Engineering & Cloud Computing Sci-Fi-Nerd, Metalhead, 
 Amateur Woodworker
4. PREVIOUSLY ON MARTINS ADVENTURES WITH KUBERNETES SKIP RECAP ( SLAC 2018 ) 
 https://speakerdeck.com/martinhelmich/slac18-php-in-the-container-cloud
5. SKIP RECAP
7. Content Warning: The following PowerPoint presentation contains material that may be harmful or traumatizing to some audiences.
8. OUR ILEAGE AY ARY Y M M V CAUTION : This presentation summarizes our OWN LEARNINGS from running Kubernetes on-premise. Not all learnings might apply to your own use case.
9. SOURCE OPEN CAUTION : All learnings in this presentation were learned from using OPEN SOURCE components ONLY (because that is what we prefer). Commercial alternatives may exist.
10. NODE BASIC KUBERNETES PRINCIPLES NODE NODE CLUSTER YEET USER POD POD POD POD ( SLIGHTLY SIMPLIFIED ) POD
11. ETCD API SERVER CONTROLLER MANAGER SCHEDULER KUBE PROXY KUBELET POD POD POD POD POD CONTOL PLANE NODE BASIC KUBERNETES ARCHITECTURE
12. > ./kube-apiserver \ 
 --bind-address=0.0.0.0 \ 
 --apiserver-count=3 \ 
 --authorization-mode=Node,RBAC \ 
 --etcd-servers=http://10.0.0.1:2379,http://10.0.0.2:2379,http//... 
 --[one million more flags] 
 
 > ./kube-scheduler --config=/etc/kubernetes/config/kube-scheduler.yaml 
 > ./kube-controller-manager \ 
 --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig 
 --[even more flags]
13. https://github.com/kelseyhightower/kubernetes-the-hard-way
14. NETWORKING CLUSTER 
 AUTOSCALING ETCD “THE CONTROL PLANE IS NOT THAT COMPLICATED” “IT’S ALL JUST YAML FILES”
15. NETWORKING
17. KUBE PROXY POD POD POD POD POD NODE POD POD POD POD POD NODE OUTSIDE WORLD INTER - POD INTER - NODE KUBE PROXY INGRESS
18. POD POD INTER - POD MANUAL ROUTING CALICO CILIUM FLANNEL WEAVE AWS VPC AZURE CNI RECOMMENDATION : Choose a CNI provider that fits your use case. Apart from the cloud- provider specific solutions (like AWS VPC ) , these all usually work both on-premise and in the cloud. We went with Calico, and consider migrating to Cilium in the future.
19. KUBE PROXY POD POD POD POD POD NODE POD POD POD POD POD NODE OUTSIDE WORLD INTER - POD INTER - NODE KUBE PROXY INGRESS
20. POD OUTSIDE WORLD KUBE PROXY INGRESS ??? A LOAD BALANCER service provides a publicly routed IP address to a SERVICE object
21. apiVersion: v1 kind: Service metadata: name: example-service spec: type: LoadBalancer selector: app: example ports: - port: 80 targetPort: 8080 What ACTUALLY HAPPENS when you create a LoadBalancer service? 🤔
22. WAIT, IT’S ALL JUST IPTABLES? ALWAYS HAS BEEN RECOMMENDATION : Use IPVS (instead of iptables) in large clusters. Read more
23. ETCD API SERVER CONTROLLER MANAGER SCHEDULER KUBE PROXY KUBELET POD POD POD POD POD CONTOL PLANE NODE BASIC KUBERNETES ARCHITECTURE
24. ETCD API SERVER CONTROLLER MANAGER SCHEDULER CONTOL PLANE BASIC KUBERNETES ARCHITECTURE CLOUD 
 CONTROLLER 
 MANAGER CLOUD PROVIDER CLUSTER EXTERNAL NETWORKING RESOURCES Read more
25. NETWORKING CLUSTER 
 AUTOSCALING ETCD
26. TORAGE
27. RECOMMENDATION : In the public cloud, it’s often easier to limit yourself to stateless deployments, and use database resources offered by the cloud provider (like AWS RDS ) . On-prem, you need to deploy your stateful workloads SOMEWHERE and need to think about providing STORAGE for these workloads. EXAMPLES : - Databases - User uploads ( TYPO3 fileadmin/, Wordpress wp-content/) - Web Hosting use-case: Entire applications that are deployed via direct upload
28. ETCD API SERVER CONTROLLER MANAGER SCHEDULER CONTOL PLANE BASIC KUBERNETES ARCHITECTURE CLOUD PROVIDER CLOUD 
 CONTROLLER 
 MANAGER
29. ETCD API SERVER CONTROLLER MANAGER SCHEDULER CONTOL PLANE BASIC KUBERNETES ARCHITECTURE CLOUD PROVIDER CLOUD 
 CONTROLLER 
 MANAGER CSI DRIVER Read more
30. eloneo https://pixabay.com/photos/hard-drive- technology-computers-1265259/ OBJECT STORAGE BLOCK STORAGE FILE STORAGE N ETW O RK ED LO C A L EPH EM ERA L N ETW O RK ED LO C A L EPH EM ERA L
32. eloneo https://pixabay.com/photos/hard-drive- technology-computers-1265259/ MIND YOUR WORKLOAD RECOMMENDATION : Consider the actual performance and access mode requirements of your application workload. Cloud-native solutions (e.g. using object storage instead of a shared file system) are preferable, but require support by the application.
33. NETWORKING CLUSTER 
 AUTOSCALING ETCD
34. ETCD
35. ETCD
36. etcd on networked block storage etcd on local SSD RECOMMENDATION : Storage performance quickly becomes a bottleneck for ETCD in larger clusters. Use local storage whenever possible.
37. NETWORKING CLUSTER 
 AUTOSCALING ETCD
38. PROVISIONING CLUSTERS
39. ON 
 PREM YES YES NO OPEN SOURCE NO HOW - TO 
 CLOUD
40. OPEN - SOURCE CLUSTER PROVISIONING CLUSTER API GARDENER OPENSTACK MAGNUM
41. Gardener BARE METAL SERVERS
42. Gardener BARE METAL SERVERS
44. Lukas Tennie https://unsplash.com/photos/a-close-up-of-a-watch-face-with-the-gears-missing-3dyDozzCORw COMPLEXITY
45. Gardener ORGANIZATIONAL BOUNDARY BARE METAL SERVERS TRADE - OFF : Vertical organisational boundaries through your tech stack may be necessary to manage complexity, but increase communication overhead (which is annoying in failure scenarios and may increase TTR )
46. Ralph_PH, CC - BY K 
 I 
 S 
 S EEP 
 T 
 IMPLE, 
 TUPID.
47. Erika Wittlieb https://pixabay.com/photos/lemonade-stand-lemonade-summer-2483297/ WHAT IS YOUR PRODUCT? WHAT DO YOU NEED TO RUN IT? (ACTUALLY)
48. Gardener BARE METAL SERVERS
49. BARE METAL SERVERS RECOMMENDATION : ( YMMV ) Eliminate components in your stack that you do not actually need for offering your service. In our case, we decided to ultimately move away from OpenStack and complex cluster provisioning solutions like Gardener.
50. BARE METAL SERVERS Metallb REMINDER : YMMV OUR APPROACH : KUBERNETES - NATIVE EVERYTHING STORAGE VIRTUALIZATION NETWORKING metal-stack RECOMMENDATION : Keep to a single ecosystem (in our case: Kubernetes) to keep complexity manageable. RECOMMENDATION : Established, but battle- proven tools (looking at you, kubeadm) may be sufficient to get the job done.
51. OUTLOOK https://metal-stack.io/ https://github.com/onmetal There are a few projects (all in early states of maturity) that also try to automate deploying Kubernetes on bare metal.
52. ACCEPT COMPLEXITY WHERE YOU NEED IT ( AND MANAGE ACCORDINGLY ) KEEP IT SIMPLE WHERE YOU DON'T KNOW YOUR PRODUCT
53. https://github.com/martin-helmich https://www.mittwald.de 
 https://www.martin-helmich.de https://www.linkedin.com/in/martinhelmich