KubeCon 2017 - Kernels and Distros

Our KubeCon 2017 talk. We posit that "Kubernetes Distributions" are A Real Thing, and the the project would be better overall if we start to think about how to manage parts of it more like the Linux kernel and other parts more like a Linux distribution.

Transcript

1. Google Cloud Platform Kubernetes: Kernels & Distros KubeCon 2017, Austin December 7, 2017 Tim Hockin <thockin@google.com> @thockin Michael Rubin <mrubin@google.com> @matchstick (c) Google LLC
2. Google Cloud Platform Where is Kubernetes today?
3. Google Cloud Platform A fairly large project (~3M LOC) Still growing Not a monolithic program A set of cooperating microservices Kubernetes is ...
4. Google Cloud Platform apiserver scheduler controller manager kube-proxy kubelet etcd container runtime
5. Google Cloud Platform kube-proxy kubelet container runtime elasticsearch heapster influxdb fluentd ingress controller kube-dns cloud controller manager apiserver scheduler controller manager etcd CNI driver CSI driver
6. Google Cloud Platform Extensibility - Enabling out-of-tree Drivers • Network (CNI) • Storage (flex, CSI) • Device (e.g. GPU) • Cloud providers Container runtimes Operators / controllers Add-ons • e.g. logging, monitoring
7. Google Cloud Platform Some assembly required
8. Google Cloud Platform Some assembly required Must Find Components!
9. Google Cloud Platform Some assembly required Must Find Components! Download Components!
10. Google Cloud Platform Some assembly required Must Find Components! Download Components! Version Skew!
11. Google Cloud Platform Some assembly required Must Find Components! Download Components! Version Skew! Test Hell!
12. Google Cloud Platform Some assembly required Must Find Components! Download Components! Version Skew! Test Hell! Unusable to most users
13. Google Cloud Platform Comparative analysis
14. Google Cloud Platform Linux is many projects Decentralized & layered: • Installers • Kernel • Bootup • Shells • Tools • Programming Languages • GUIs Roughly “kernel” & “distro”
15. Google Cloud Platform One big program Lives in its own git repo • No tools, add-ons, etc. Releases fairly frequently • 8-10 weeks Has only X.Y versions • Bugs in X.Y get fixed in X.Y+1 • X.Y.Z patch releases managed by community The kernel - Not useful alone
16. Google Cloud Platform Everything else is developed separately DIY systems are not tenable ca. 1992, the concept of “Linux distros” emerged EVERYONE uses a distro Distributions
17. Google Cloud Platform Distros serve different needs Most users don’t care about kernel version, just distro version Generally release slowly • Quarters to years Emphasize their differences • Technical & opinions Distributions
18. Google Cloud Platform Distributions Distro Value Add Platform support (drivers & extensions) Support, security, & testing Packaging & component lifecycle Simple installation
19. Google Cloud Platform Kubernetes: Kernel or Distribution?
20. Google Cloud Platform Releases quarterly Includes enough to run most clouds • But not all of them • May not be true in the future Contains a bunch of drivers • But not all of them • May not be true in the future Is Kubernetes a kernel or a distro? ?
21. Google Cloud Platform Integrates some 3rd party add-ons • But not many • Does not usually carry patches X.Y.Z releases and medium-term support Lives in a small number of git repos • Highly coupled components Is Kubernetes a kernel or a distro? ?
22. Google Cloud Platform Platform support (drivers & extensions) Support, security, & testing Packaging & component lifecycle Simple installation Kubernetes upstream as a distro Distro Value Add
23. Google Cloud Platform Other Kubernetes distros Distro Value Add There are ALREADY more than 30 Kubernetes distributions! • Clouds • Enterprise vendors • Higher level platforms • Bespoke Platform support (drivers & extensions) Support, security, & testing Packaging & component lifecycle Simple installation
24. Google Cloud Platform * conformance program Fragmentation risk: conformance
25. Google Cloud Platform Distros were inevitable.
26. Google Cloud Platform Distros were inevitable. How do we want to organize our project and community?
27. Google Cloud Platform Others will make distros • No coordination or consistency • We will have no say Result: • Fragmentation & politicization • Many options, confusing UX • Over time, converge on 3-4 distros? Option #1: Ignore it
28. Google Cloud Platform Needs huge non-eng effort Major distraction from k8s Others will still do their own • Opinions Result: Probably failure, see option #1 Option #2: One distro to rule them all
29. Google Cloud Platform Formalize what we already do Focus on correctness and stability Others will still do their own • But hopefully based on ours? Result: • Clean up our thinking / processes • Define tools, standards, etc. • Derived distros benefit from staying close Option 3: Find the middle-ground
30. Google Cloud Platform Concrete Ideas
31. Google Cloud Platform Pick ONE installer and make it great • Which one? Contentious! Or define a manifest format that installers consume? • Think kernel config process and result Installers 39% Please wait...
32. Google Cloud Platform Formalize “add-ons” • Central repository • Ownership • Management mechanism • Tooling ◦ kubectl apt-get update? Start with cluster/addons/... • How to track upstreams? Add-ons
33. Google Cloud Platform Bound and extract “the kernel” • Still multiple binaries • No installer, add-ons, cloud providers Manage it as a single component Release it tick-tock style • Features vs. stability Kernel != distro Manage the kernel OSS K8s
34. Google Cloud Platform Fork all code into our space • Only ship things we control • Carry patches IFF needed Push everything to one repository • No hunting all over the internet Manage the distro Kernel Installers Container Images Addons Components
35. Google Cloud Platform Distinguish component version from package version • e.g. 1.2.3-4 = “4th build of 1.2.3” Release distro every 6-12 months Base deprecation policies on distro releases Manage the distro
36. Google Cloud Platform A new role - Distro Hero! These are just some ideas To pull this off, we need a community • Different skills • Different focus https://commons.wikimedia.org/wiki/File:Placeholder_couple_superhero.png