Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes Extensibility

Kubernetes Extensibility

A look at some of the ways Kubernetes is extensible and modular.

Tim Hockin

May 09, 2016
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Google Cloud Platform
    Building Longevity into Kubernetes
    CoreOS Fest
    May 9th, 2016
    Tim Hockin
    Senior Staff SW Engineer, Google
    @thockin

    View Slide

  2. Google Cloud Platform
    Google has been developing
    and using containers to
    manage our applications for
    over 12 years.
    Images by Connie Zhou

    View Slide

  3. Google Cloud Platform
    That’s a lot of time to accumulate
    features...

    View Slide

  4. Google Cloud Platform
    That’s a lot of time to accumulate
    features...
    The OSS world is far more diverse
    than Google internally...

    View Slide

  5. Google Cloud Platform
    That’s a lot of time to accumulate
    features...
    The OSS world is far more diverse
    than Google internally...
    We get to rebuild it all, from
    scratch, in less than half the
    time...

    View Slide

  6. Google Cloud Platform
    The chances of satisfying
    everyone?

    View Slide

  7. Google Cloud Platform

    View Slide

  8. Google Cloud Platform
    Modularity
    Extensibility
    Pluggability

    View Slide

  9. Google Cloud Platform
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    kubelet docker
    cloud
    provider
    etcd

    View Slide

  10. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    docker
    kubelet

    View Slide

  11. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    kube-dns
    fluentd
    docker
    kubelet

    View Slide

  12. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet

    View Slide

  13. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet

    View Slide

  14. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    influxdb

    View Slide

  15. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    influxdb

    View Slide

  16. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    cloud
    provider
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb

    View Slide

  17. Google Cloud Platform
    You get the point

    View Slide

  18. Google Cloud Platform
    Modularity
    Loose coupling is a goal everywhere
    • simpler
    • more composable
    • more extensible
    Isolate risk with interchangeable parts
    Sometimes accused of having too many
    parts -- but most are trivially replaced!
    Result: A more robust, adaptable system

    View Slide

  19. Google Cloud Platform
    kubelet
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    docker
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider

    View Slide

  20. Google Cloud Platform
    Linked as a library (for now)
    • needs to be extracted for easier
    customization
    Supports major public clouds, and
    some private clouds
    Interfaces for things like availability
    zones, VMs, load-balancers
    Used by controllers, admission control,
    Cloud Provider
    ?

    View Slide

  21. Google Cloud Platform
    users master nodes
    apiserver
    scheduler controller
    manager
    kube-proxy
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    kubelet docker

    View Slide

  22. Google Cloud Platform
    Not all containers are Docker containers
    Runtime API abstracts containers and
    implements pods
    • Linked as a library, but out-of-
    process plugin is in devel
    Docker is just the first implementation
    In devel:
    • rkt (CoreOS)
    • hyper_ (Hyper.sh)
    Container Runtime
    Runtime API

    View Slide

  23. Google Cloud Platform
    users master nodes
    apiserver
    controller
    manager
    kube-proxy
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    kubelet docker
    scheduler

    View Slide

  24. Google Cloud Platform
    A scheduler is just a program that uses
    the Kubernetes API
    Anyone can write their own
    • e.g. specialized affinity or isolation
    • customized for your needs
    • handle only Pods you care about
    Pods can “opt in” to alternative
    schedulers with an annotation (alpha)
    • default scheduler will ignore them
    Multi-Scheduler
    Scheduler
    Scheduler Scheduler
    ?

    View Slide

  25. Google Cloud Platform
    users master nodes
    apiserver
    controller
    manager
    kube-proxy
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    scheduler
    kubelet

    View Slide

  26. Google Cloud Platform
    Network Plugins
    Introduced in Kubernetes v1.0
    • still considered experimental
    Uses CNI (CoreOS/appc) in v1.1
    • simple exec interface
    • not using Docker libnetwork,
    • can simply defer to Docker
    Cluster admins can customize their installs
    • DHCP, MACVLAN, Flannel, custom...
    net
    Plugin
    Plugin
    Plugin

    View Slide

  27. Google Cloud Platform
    Storage Plugins
    Introduced in Kubernetes v1.0
    • ~20 plugins in-tree so far
    Linked as a library
    • APIs: attach, mount, recycle, provision
    • not all plugins implement all facets
    • “flex” plugin for out-of-process plugins
    • can (soon) pass-through to Docker
    Usable directly (from a Pod) or through
    PersistentVolumes abstraction
    Plugin
    Plugin
    Plugin

    View Slide

  28. Google Cloud Platform
    users master nodes
    apiserver
    controller
    manager
    kube-proxy
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    dashboard
    influxdb
    cloud
    provider
    docker
    scheduler
    kubelet
    l7-lb-controller

    View Slide

  29. Google Cloud Platform
    Ingress (L7 LB)
    Services are L3/L4 (IP + port)
    Many apps are HTTP/HTTPS
    Ingress maps incoming traffic to backend
    services
    • by HTTP host headers
    • by HTTP URL paths
    HAProxy, NGINX, AWS and GCE
    implementations in progress
    Now with SSL!
    BETA in Kubernetes v1.2
    Client
    URL Map

    View Slide

  30. Google Cloud Platform
    users master nodes
    controller
    manager
    kube-proxy
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    scheduler
    kubelet
    apiserver

    View Slide

  31. Google Cloud Platform
    3rd Party & API Groups
    ThirdPartyResource:
    • store your objects in our API server
    • CRUD only
    • creates a new Kind in the API
    • try ideas or build simple extensions
    Federated API servers: coming soon
    • store your objects in your own API server
    • register with our API, we delegate
    • more control of validation, errors, etc.
    • creates a new Kind in the API
    Pods
    Services
    ReplicaSets
    Your3PR
    Your
    API
    YourAPI
    API Server

    View Slide

  32. Google Cloud Platform
    users master nodes
    controller
    manager
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    scheduler
    kubelet
    apiserver
    kube-proxy

    View Slide

  33. Google Cloud Platform
    Services & kube-proxy
    Services are an abstraction: a stable IP that
    fronts N pods
    Default implementation is iptables, but that is just
    one way to do it
    Other implementations are equally valid
    • “real” load balancers
    • VIP
    • SDN-centric solutions
    • ...let’s get creative!
    Client
    Service IP

    View Slide

  34. Google Cloud Platform
    Past-me predicts that
    future-me is out of time

    View Slide

  35. Google Cloud Platform
    users master nodes
    controller
    manager
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    scheduler
    kubelet
    kube-proxy
    apiserver
    Authorization
    Authentication
    Admission

    View Slide

  36. Google Cloud Platform
    users master nodes
    controller
    manager
    etcd
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    kubelet
    kube-proxy
    apiserver
    scheduler
    Predicates
    Priorities
    Extenders

    View Slide

  37. Google Cloud Platform
    users master nodes
    controller
    manager
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    kubelet
    kube-proxy
    scheduler
    API storage
    etcd
    apiserver

    View Slide

  38. Google Cloud Platform
    users master nodes
    controller
    manager
    kube-dns
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    kube-proxy
    scheduler
    etcd
    apiserver kubelet
    Credentials

    View Slide

  39. Google Cloud Platform
    users master nodes
    controller
    manager
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    influxdb
    cloud
    provider
    docker
    kube-proxy
    scheduler
    etcd
    apiserver kubelet
    kube-dns
    Replaceable

    View Slide

  40. Google Cloud Platform
    users master nodes
    controller
    manager
    fluentd
    elasticsearch
    docker
    kubelet heapster
    l7-lb-controller
    dashboard
    cloud
    provider
    docker
    kube-proxy
    scheduler
    etcd
    apiserver kubelet
    kube-dns
    Replaceable
    influxdb
    Replaceable

    View Slide

  41. 41
    41
    Kubernetes is Open
    https://kubernetes.io
    Code: github.com/kubernetes/kubernetes
    Chat: slack.k8s.io
    Twitter: @kubernetesio
    open community
    open design
    open source
    open to ideas

    View Slide