KubeCon 2017 - Kernels and Distros

Our KubeCon 2017 talk. We posit that "Kubernetes Distributions" are A Real Thing, and the the project would be better overall if we start to think about how to manage parts of it more like the Linux kernel and other parts more like a Linux distribution.

Transcript

1. Google Cloud Platform
   Kubernetes: Kernels & Distros
   KubeCon 2017, Austin
   December 7, 2017
   Tim Hockin @thockin
   Michael Rubin @matchstick
   (c) Google LLC
   

   View Slide

2. Google Cloud Platform
   Where is Kubernetes today?
   

   View Slide

3. Google Cloud Platform
   A fairly large project (~3M LOC)
   Still growing
   Not a monolithic program
   A set of cooperating
   microservices
   Kubernetes is ...
   

   View Slide

4. Google Cloud Platform
   apiserver
   scheduler
   controller
   manager
   kube-proxy
   kubelet
   etcd
   container
   runtime
   

   View Slide

5. Google Cloud Platform
   kube-proxy
   kubelet
   container
   runtime
   elasticsearch
   heapster
   influxdb
   fluentd
   ingress
   controller
   kube-dns
   cloud
   controller
   manager
   apiserver
   scheduler
   controller
   manager
   etcd
   CNI driver
   CSI driver
   

   View Slide

6. Google Cloud Platform
   Extensibility - Enabling out-of-tree
   Drivers
   ● Network (CNI)
   ● Storage (flex, CSI)
   ● Device (e.g. GPU)
   ● Cloud providers
   Container runtimes
   Operators / controllers
   Add-ons
   ● e.g. logging, monitoring
   

   View Slide

7. Google Cloud Platform
   Some assembly
   required
   

   View Slide

8. Google Cloud Platform
   Some assembly
   required
   Must Find
   Components!
   

   View Slide

9. Google Cloud Platform
   Some assembly
   required
   Must Find
   Components! Download
   Components!
   

   View Slide

10. Google Cloud Platform
    Some assembly
    required
    Must Find
    Components! Download
    Components!
    Version
    Skew!
    

    View Slide

11. Google Cloud Platform
    Some assembly
    required
    Must Find
    Components! Download
    Components!
    Version
    Skew! Test Hell!
    

    View Slide

12. Google Cloud Platform
    Some assembly
    required
    Must Find
    Components! Download
    Components!
    Version
    Skew! Test Hell!
    Unusable to
    most users
    

    View Slide

13. Google Cloud Platform
    Comparative analysis
    

    View Slide

14. Google Cloud Platform
    Linux is many projects
    Decentralized & layered:
    ● Installers
    ● Kernel
    ● Bootup
    ● Shells
    ● Tools
    ● Programming Languages
    ● GUIs
    Roughly “kernel” & “distro”
    

    View Slide

15. Google Cloud Platform
    One big program
    Lives in its own git repo
    ● No tools, add-ons, etc.
    Releases fairly frequently
    ● 8-10 weeks
    Has only X.Y versions
    ● Bugs in X.Y get fixed in X.Y+1
    ● X.Y.Z patch releases managed
    by community
    The kernel - Not useful alone
    

    View Slide

16. Google Cloud Platform
    Everything else is developed
    separately
    DIY systems are not tenable
    ca. 1992, the concept of
    “Linux distros” emerged
    EVERYONE uses a distro
    Distributions
    

    View Slide

17. Google Cloud Platform
    Distros serve different needs
    Most users don’t care about kernel
    version, just distro version
    Generally release slowly
    ● Quarters to years
    Emphasize their differences
    ● Technical & opinions
    Distributions
    

    View Slide

18. Google Cloud Platform
    Distributions
    Distro Value Add
    Platform support
    (drivers & extensions)
    Support,
    security, & testing
    Packaging & component
    lifecycle
    Simple installation
    

    View Slide

19. Google Cloud Platform
    Kubernetes: Kernel or Distribution?
    

    View Slide

20. Google Cloud Platform
    Releases quarterly
    Includes enough to run most clouds
    ● But not all of them
    ● May not be true in the future
    Contains a bunch of drivers
    ● But not all of them
    ● May not be true in the future
    Is Kubernetes a kernel or a distro?
    ?
    

    View Slide

21. Google Cloud Platform
    Integrates some 3rd party add-ons
    ● But not many
    ● Does not usually carry patches
    X.Y.Z releases and medium-term
    support
    Lives in a small number of git repos
    ● Highly coupled components
    Is Kubernetes a kernel or a distro?
    ?
    

    View Slide

22. Google Cloud Platform
    Platform support
    (drivers & extensions)
    Support,
    security, & testing
    Packaging & component
    lifecycle
    Simple installation
    Kubernetes upstream as a distro
    Distro Value Add
    

    View Slide

23. Google Cloud Platform
    Other Kubernetes distros
    Distro Value Add There are ALREADY more than
    30 Kubernetes distributions!
    ● Clouds
    ● Enterprise vendors
    ● Higher level platforms
    ● Bespoke
    Platform support
    (drivers & extensions)
    Support,
    security, & testing
    Packaging & component
    lifecycle
    Simple installation
    

    View Slide

24. Google Cloud Platform
    * conformance program
    Fragmentation risk: conformance
    

    View Slide

25. Google Cloud Platform
    Distros were inevitable.
    

    View Slide

26. Google Cloud Platform
    Distros were inevitable.
    How do we want to
    organize our project and
    community?
    

    View Slide

27. Google Cloud Platform
    Others will make distros
    ● No coordination or consistency
    ● We will have no say
    Result:
    ● Fragmentation & politicization
    ● Many options, confusing UX
    ● Over time, converge on 3-4
    distros?
    Option #1: Ignore it
    

    View Slide

28. Google Cloud Platform
    Needs huge non-eng effort
    Major distraction from k8s
    Others will still do their own
    ● Opinions
    Result: Probably failure, see
    option #1
    Option #2: One distro to rule them all
    

    View Slide

29. Google Cloud Platform
    Formalize what we already do
    Focus on correctness and stability
    Others will still do their own
    ● But hopefully based on ours?
    Result:
    ● Clean up our thinking / processes
    ● Define tools, standards, etc.
    ● Derived distros benefit from staying
    close
    Option 3: Find the middle-ground
    

    View Slide

30. Google Cloud Platform
    Concrete Ideas
    

    View Slide

31. Google Cloud Platform
    Pick ONE installer and make it
    great
    ● Which one? Contentious!
    Or define a manifest format
    that installers consume?
    ● Think kernel config process and
    result
    Installers
    39%
    Please wait...
    

    View Slide

32. Google Cloud Platform
    Formalize “add-ons”
    ● Central repository
    ● Ownership
    ● Management mechanism
    ● Tooling
    ○ kubectl apt-get update?
    Start with cluster/addons/...
    ● How to track upstreams?
    Add-ons
    

    View Slide

33. Google Cloud Platform
    Bound and extract “the kernel”
    ● Still multiple binaries
    ● No installer, add-ons, cloud providers
    Manage it as a single component
    Release it tick-tock style
    ● Features vs. stability
    Kernel != distro
    Manage the kernel
    OSS
    K8s
    

    View Slide

34. Google Cloud Platform
    Fork all code into our space
    ● Only ship things we control
    ● Carry patches IFF needed
    Push everything to one repository
    ● No hunting all over the internet
    Manage the distro
    Kernel
    Installers
    Container
    Images
    Addons
    Components
    

    View Slide

35. Google Cloud Platform
    Distinguish component version
    from package version
    ● e.g. 1.2.3-4 = “4th build of 1.2.3”
    Release distro every 6-12 months
    Base deprecation policies on
    distro releases
    Manage the distro
    

    View Slide

36. Google Cloud Platform
    A new role - Distro Hero!
    These are just some ideas
    To pull this off, we need a
    community
    ● Different skills
    ● Different focus
    https://commons.wikimedia.org/wiki/File:Placeholder_couple_superhero.png
    

    View Slide