Kubernetes: An Extensible Platform

569f10721398d92f5033097ac6d9132c?s=47 Tim Hockin
November 15, 2018

Kubernetes: An Extensible Platform

A Survey of some of the extension mechanisms in Kubernetes.

569f10721398d92f5033097ac6d9132c?s=128

Tim Hockin

November 15, 2018
Tweet

Transcript

  1. Google Cloud Platform logo (c) Google LLC Kubernetes: An Extensible

    Platform Warsaw Cloud Native / Kubernetes Meetup November, 2018 Tim Hockin Principle Software Engineer, Google @thockin
  2. Google Cloud Platform Kubernetes is ... ? Kubernetes is a...

  3. Google Cloud Platform Kubernetes is: an abstraction layer for cloud

    infrastructure Kubernetes is a...
  4. Google Cloud Platform Kubernetes is: a framework for declarative APIs

    and distrubuted control Kubernetes is a...
  5. Google Cloud Platform Extensibility means different things an abstraction layer

    for infrastructure a framework for declarative APIs and distributed control Infrastructure Extensibility API Extensibility
  6. Google Cloud Platform The goals of extensibility Infrastructure Extensibility API

    Extensibility Support portability Support customization Autonomy Autonomy Scalable project growth Encourage new uses
  7. Google Cloud Platform A major focus of the last 2

    years of development From infrastructure to APIs, over a dozen extension points I have WAY more material than time! For later reading: https://goo.gl/2qz8jW Let’s talk about extensibility
  8. Google Cloud Platform Infrastructure Extensibility

  9. Google Cloud Platform Allow Kubernetes run on new infrastructure (think

    IaaS) • Networking • Storage • Devices (e.g. GPUs) • Clouds Allow cluster operators to replace critical components • Container runtime • Auth{n,z} • Scheduling What does it mean?
  10. Google Cloud Platform Networks are like snowflakes There is no

    “one size fits all” for almost anything networking related We need a way for users to customize how Kubernetes consumes networking infra Networking plugins
  11. Google Cloud Platform CNI - Container Network Interface • Started

    by CoreOS, now CNCF • Exec interface with stdin/stdout/env API Widely used, including by other projects (e.g. Mesos) Significant vendor uptake (Calico, Weave, Flannel, Contiv) Networking plugins
  12. Google Cloud Platform So many storage technologies - physical and

    virtual, block and filesystem • Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ... Vendors want their products to integrate well with Kubernetes Storage plugins
  13. Google Cloud Platform Old v1: built-in “plugins” (aka “send Tim

    a PR”) Old v2: Volume “flex” plugins via “exec” New: CSI - Container Storage Interface • Collaboration: Google, Mesosphere, Docker, Cloud Foundry • gRPC plugins, with Kubernetes-specific adaptors • Spec 1.0 soon Plan to transition most in-tree plugins to CSI Storage plugins
  14. Google Cloud Platform GPUs and other “accelerator” hardware is becoming

    very common Part of the larger resource model in Kubernetes gRPC plugins from day 1 Focused on GPUs for now Device plugins
  15. Google Cloud Platform Docker was baked-in, but people wanted to

    try new and interesting ideas • Container-ish: rkt, Containerd, CRI-O • VM-ish: Kata containers, Hyper.sh, gVisor CRI - gRPC based plugins for Kubernetes Docker-specific code was everywhere - making it a plugin made the code better: win-win! Container runtime plugins
  16. Google Cloud Platform Plugins: lessons learned • • → •

    → •
  17. Google Cloud Platform Controllers: A different form of extensibility observe

    diff act
  18. Google Cloud Platform THE fundamental design pattern in Kubernetes Examples:

    scheduler, kubelet, deployments, kube-proxy, cloud providers, load balancers, volume provisioners, auto-scalers, DNS, ... Allows automation & extension of almost any existing API Controllers
  19. Google Cloud Platform Kubernetes is designed to leverage clouds (including

    private clouds) Built-in cloud-provider API (i.e. send us a PR) is hooked into many core control loops 8 implementations (and huge LOC count), so moving out-of-tree, to separate controllers Cloud providers
  20. Google Cloud Platform The API is a VIP (more or

    less) and virtual load-balancer We ship a default implementation (kube-proxy), but that can be replaced Controller: watch the API server for Services and Endpoints, program the underlying network Services
  21. Google Cloud Platform • Secret management (KMS) • HTTP load-balancing

    (Ingress) • NetworkPolicy • DNS • Scheduler extenders & whole schedulers • Auth plugins ...and that’s JUST the infrastructure (i.e. boring) parts ...and more!
  22. Google Cloud Platform API Extensibility

  23. Google Cloud Platform Allow new types of resources to be

    added • Without changing code or recompiling the system! • With a result that feels “built in” Add custom policy hooks • To built-in and extension APIs • Policy is arbitrary and very customer specific => APIs that add and modify the APIs What does it mean?
  24. Google Cloud Platform Different users / apps need different levels

    of customization Kubernetes provides 2 ways to extend the API • Custom Resource Definitions (CRD) • Extension API Servers (EAS) Both allow dynamic creation of new API “kinds” • Feel native: support kubectl, discovery API, etc. Trade-off: simplicity vs. flexibility Levels of customization
  25. Google Cloud Platform Add new kinds of “workload patterns” •

    Customized deployments or stateful apps Add new abstractions • Build a custom PaaS that exposes JUST what you need Add entirely new concepts - it doesn’t have to be Kubernetes! What can I do with these?
  26. Google Cloud Platform Istio: A service mesh and network control

    plane KNative: A PaaS toolkit Kubernetes Metrics API: Adapters for monitoring systems Operators: Software robots to manage complex apps Docker Stacks: A higher-level abstraction Examples
  27. Google Cloud Platform $ cat docker-compose.yml version: "3.3" services: redis:

    image: redis:alpine ports: - 6379 networks: - frontend deploy: replicas: 1 networks: frontend:
  28. Google Cloud Platform $ docker stack deploy --compose-file docker-compose.yml stackdemo

    Waiting for the stack to be stable and running... - Service redis has one container running Stack stackdemo is stable and running
  29. Google Cloud Platform $ kubectl config current-context docker-for-desktop

  30. Google Cloud Platform $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s
  31. Google Cloud Platform $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 1 1 1 1 2s
  32. Google Cloud Platform compose resource service resource deployment resource Higher

    level of abstraction Lower level of abstraction
  33. Google Cloud Platform Kubernetes APIs Service API Deployment API service

    resource deployment resource dockerd hypothetical /stacks compose resource docker cli
  34. Google Cloud Platform $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 1 1 1 1 2s $ kubectl get stacks NAME AGE stackdemo 39s
  35. Google Cloud Platform $ $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001
  36. Google Cloud Platform $ kubectl get stacks -s localhost:8001 NAME

    AGE stackdemo 1m $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 I0613 10:13:27.322416 82905 proxy_server.go:138] Filter accepting GET /apis/compose.docker.com/v1beta2/namespaces/d efault/stacks localhost
  37. Google Cloud Platform Kubernetes APIs Service API Deployment API service

    resource deployment resource dockerd hypothetical /stacks compose resource Stacks API service resource docker cli
  38. Google Cloud Platform Kubernetes APIs Service API Deployment API service

    resource deployment resource dockerd hypothetical /stacks compose resource kubectl cli Stacks API service resource
  39. Google Cloud Platform $ kubectl get apiservices.apiregistration.k8s.io NAME AGE v1.

    29d v1.apps 29d ... v1beta2.compose.docker.com 29d v2beta1.autoscaling 29d
  40. Google Cloud Platform $ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com Name: v1beta2.compose.docker.com

    ... API Version: apiregistration.k8s.io/v1beta1 Kind: APIService Metadata: ... Spec: ... Service: Name: compose-api Namespace: docker Status: Conditions: Message: all checks passed
  41. Google Cloud Platform $ kubectl get services -n docker NAME

    TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d
  42. Google Cloud Platform $ kubectl get services -n docker NAME

    TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d $ kubectl get deployments -n docker NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE compose 1 1 1 1 29d compose-api 1 1 1 1 29d
  43. Google Cloud Platform EAP: • Write and run your own

    apiserver • Run your own storage • The “main” apiserver is a proxy for your resources • Validation and defaulting built-in CRD: • Write a schema (or not!) for your resource • The “main” apiserver stores and serves your resources • Validation and defaulting as admission webhooks EAP vs. CRD
  44. Google Cloud Platform EAS Forked LoC: 0 Storage: provided Components:

    1 Popularity: 100s Multiversioning: not yet Policy: dynamic only Customizability: good Forked LoC: 5000* Storage: you manage Components: 3 Popularity: 10s Multiversioning: yes Policy: code or dynamic Customizability: better * http://github.com/sample-apiserver CRD
  45. Google Cloud Platform Extensions ecosystem Devices 5 public plugins Storage

    10 public plugins Networking >20 public plugins Custom APIs >400 Github Projects with custom APIs
  46. Google Cloud Platform Admission: After authn/z but before storing the

    change. Affects writes, not reads. Webhook: The API Server calls your URL, synchronously Run in cluster (via service) or outside (e.g. serverless) Admission webhooks
  47. Google Cloud Platform Composable: anyone (modulo ACLs) can add them

    to any resources Make all the changes (mutating) before doing all the checks (validating) MutatingWebhookConfiguration ValidatingWebhookConfiguration Admission webhooks
  48. Google Cloud Platform Kelsey Hightower example: • reject pods that

    set environment variables https://github.com/kelseyhightower/denyenv-validating-admission-webhook CRD authors: • add complex validation logic Cluster admins: • enforce arbitrary company policy Validating
  49. Google Cloud Platform Istio: • inject a sidecar container into

    pods Service Catalog: • inject credentials into pods VerticalPodAutoscaler: • set ideal resource requests Mutating
  50. Google Cloud Platform Kubernetes is serious about extensibility We didn’t

    even cover half of the mechanisms! Extending Kubernetes is easy (and encouraged) Conclusion
  51. Google Cloud Platform Dziękuję