KubeCon 2017 - Kernels and Distros

Slide 1

Slide 1 text

Google Cloud Platform Kubernetes: Kernels & Distros KubeCon 2017, Austin December 7, 2017 Tim Hockin @thockin Michael Rubin @matchstick (c) Google LLC

Slide 2

Slide 2 text

Google Cloud Platform Where is Kubernetes today?

Slide 3

Slide 3 text

Google Cloud Platform A fairly large project (~3M LOC) Still growing Not a monolithic program A set of cooperating microservices Kubernetes is ...

Slide 4

Slide 4 text

Google Cloud Platform apiserver scheduler controller manager kube-proxy kubelet etcd container runtime

Slide 5

Slide 5 text

Google Cloud Platform kube-proxy kubelet container runtime elasticsearch heapster influxdb fluentd ingress controller kube-dns cloud controller manager apiserver scheduler controller manager etcd CNI driver CSI driver

Slide 6

Slide 6 text

Google Cloud Platform Extensibility - Enabling out-of-tree Drivers ● Network (CNI) ● Storage (flex, CSI) ● Device (e.g. GPU) ● Cloud providers Container runtimes Operators / controllers Add-ons ● e.g. logging, monitoring

Slide 7

Slide 7 text

Google Cloud Platform Some assembly required

Slide 8

Slide 8 text

Google Cloud Platform Some assembly required Must Find Components!

Slide 9

Slide 9 text

Google Cloud Platform Some assembly required Must Find Components! Download Components!

Slide 10

Slide 10 text

Google Cloud Platform Some assembly required Must Find Components! Download Components! Version Skew!

Slide 11

Slide 11 text

Google Cloud Platform Some assembly required Must Find Components! Download Components! Version Skew! Test Hell!

Slide 12

Slide 12 text

Google Cloud Platform Some assembly required Must Find Components! Download Components! Version Skew! Test Hell! Unusable to most users

Slide 13

Slide 13 text

Google Cloud Platform Comparative analysis

Slide 14

Slide 14 text

Google Cloud Platform Linux is many projects Decentralized & layered: ● Installers ● Kernel ● Bootup ● Shells ● Tools ● Programming Languages ● GUIs Roughly “kernel” & “distro”

Slide 15

Slide 15 text

Google Cloud Platform One big program Lives in its own git repo ● No tools, add-ons, etc. Releases fairly frequently ● 8-10 weeks Has only X.Y versions ● Bugs in X.Y get fixed in X.Y+1 ● X.Y.Z patch releases managed by community The kernel - Not useful alone

Slide 16

Slide 16 text

Google Cloud Platform Everything else is developed separately DIY systems are not tenable ca. 1992, the concept of “Linux distros” emerged EVERYONE uses a distro Distributions

Slide 17

Slide 17 text

Google Cloud Platform Distros serve different needs Most users don’t care about kernel version, just distro version Generally release slowly ● Quarters to years Emphasize their differences ● Technical & opinions Distributions

Slide 18

Slide 18 text

Google Cloud Platform Distributions Distro Value Add Platform support (drivers & extensions) Support, security, & testing Packaging & component lifecycle Simple installation

Slide 19

Slide 19 text

Google Cloud Platform Kubernetes: Kernel or Distribution?

Slide 20

Slide 20 text

Google Cloud Platform Releases quarterly Includes enough to run most clouds ● But not all of them ● May not be true in the future Contains a bunch of drivers ● But not all of them ● May not be true in the future Is Kubernetes a kernel or a distro? ?

Slide 21

Slide 21 text

Google Cloud Platform Integrates some 3rd party add-ons ● But not many ● Does not usually carry patches X.Y.Z releases and medium-term support Lives in a small number of git repos ● Highly coupled components Is Kubernetes a kernel or a distro? ?

Slide 22

Slide 22 text

Google Cloud Platform Platform support (drivers & extensions) Support, security, & testing Packaging & component lifecycle Simple installation Kubernetes upstream as a distro Distro Value Add

Slide 23

Slide 23 text

Google Cloud Platform Other Kubernetes distros Distro Value Add There are ALREADY more than 30 Kubernetes distributions! ● Clouds ● Enterprise vendors ● Higher level platforms ● Bespoke Platform support (drivers & extensions) Support, security, & testing Packaging & component lifecycle Simple installation

Slide 24

Slide 24 text

Google Cloud Platform * conformance program Fragmentation risk: conformance

Slide 25

Slide 25 text

Google Cloud Platform Distros were inevitable.

Slide 26

Slide 26 text

Google Cloud Platform Distros were inevitable. How do we want to organize our project and community?

Slide 27

Slide 27 text

Google Cloud Platform Others will make distros ● No coordination or consistency ● We will have no say Result: ● Fragmentation & politicization ● Many options, confusing UX ● Over time, converge on 3-4 distros? Option #1: Ignore it

Slide 28

Slide 28 text

Google Cloud Platform Needs huge non-eng effort Major distraction from k8s Others will still do their own ● Opinions Result: Probably failure, see option #1 Option #2: One distro to rule them all

Slide 29

Slide 29 text

Google Cloud Platform Formalize what we already do Focus on correctness and stability Others will still do their own ● But hopefully based on ours? Result: ● Clean up our thinking / processes ● Define tools, standards, etc. ● Derived distros benefit from staying close Option 3: Find the middle-ground

Slide 30

Slide 30 text

Google Cloud Platform Concrete Ideas

Slide 31

Slide 31 text

Google Cloud Platform Pick ONE installer and make it great ● Which one? Contentious! Or define a manifest format that installers consume? ● Think kernel config process and result Installers 39% Please wait...

Slide 32

Slide 32 text

Google Cloud Platform Formalize “add-ons” ● Central repository ● Ownership ● Management mechanism ● Tooling ○ kubectl apt-get update? Start with cluster/addons/... ● How to track upstreams? Add-ons

Slide 33

Slide 33 text

Google Cloud Platform Bound and extract “the kernel” ● Still multiple binaries ● No installer, add-ons, cloud providers Manage it as a single component Release it tick-tock style ● Features vs. stability Kernel != distro Manage the kernel OSS K8s

Slide 34

Slide 34 text

Google Cloud Platform Fork all code into our space ● Only ship things we control ● Carry patches IFF needed Push everything to one repository ● No hunting all over the internet Manage the distro Kernel Installers Container Images Addons Components

Slide 35

Slide 35 text

Google Cloud Platform Distinguish component version from package version ● e.g. 1.2.3-4 = “4th build of 1.2.3” Release distro every 6-12 months Base deprecation policies on distro releases Manage the distro

Slide 36

Slide 36 text

Google Cloud Platform A new role - Distro Hero! These are just some ideas To pull this off, we need a community ● Different skills ● Different focus https://commons.wikimedia.org/wiki/File:Placeholder_couple_superhero.png