Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dockercon_2018_-_Kubernetes_Extensibility.pdf

569f10721398d92f5033097ac6d9132c?s=47 Tim Hockin
June 16, 2018
1.1k

 Dockercon_2018_-_Kubernetes_Extensibility.pdf

569f10721398d92f5033097ac6d9132c?s=128

Tim Hockin

June 16, 2018
Tweet

Transcript

  1. Eric Tune & Tim Hockin Google Kubernetes Extensibility

  2. Kubernetes is a container management system

  3. Kubernetes is a container management system platform

  4. What is Kubernetes? ...an abstraction layer over infrastructure ...a framework

    for declarative APIs and distributed control Infrastructure Extensibility API Extensibility
  5. Extensibility Goals Infrastructure Extensibility API Extensibility Support portability Support customization

    Autonomy Autonomy Scalable growth of project Encourage new uses
  6. A major focus of the last 2 years of development

    From infrastructure to APIs, we have over a dozen extension points We have WAY more material than time! https://goo.gl/2qz8jW Kubernetes & Extensibility
  7. Infrastructure Extensibility

  8. Networks are like snowflakes There is no “one size fits

    all” for almost anything networking related We needed a way for users to customize how Kubernetes consumes networking infra Network Plugins
  9. Old: built-in “plugins” (aka “send Tim a PR”) New: CNI

    - Container Network Interface • Started by CoreOS, now CNCF with community • “exec” interface with stdin/stdout/env API Widely used, also by other projects (e.g. Mesos) Underpins the default impl in Kubernetes Network Plugins (present)
  10. Proposal open for a gRPC based API which covers more

    than just interfaces and IPAM Tighter coupling with Service API seems valuable Proposals open for multi-IP and multi-network Network Plugins (future)
  11. Many storage technologies - physical and virtual, block and file

    • Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ... Many vendors want their products to support Kubernetes Storage Plugins
  12. Old: built-in “plugins” (aka “send Tim a PR”) Old: Volume

    “flex” plugins via “exec” New: CSI - Container Storage Interface • Collaboration: Google, Mesosphere, Docker, Cloud Foundry • gRPC spec, with Kubernetes-specific adaptors • In development now, alpha in Kubernetes 1.10 Plan to transition most in-tree plugins to CSI Storage Plugins (present)
  13. GPUs and other “accelerator” hardware is becoming very common Part

    of the larger resource model in Kubernetes gRPC based plugins Beta in Kubernetes 1.10 Device Plugins
  14. Docker was baked-in, but people wanted to try new and

    interesting ideas • rkt, Containerd, CRI-O • Kata containers, Hyper.sh, gVisor Making it a plugin made the code better: win-win! CRI - gRPC based plugins Container Runtimes
  15. • Stateful, daemon plugins • Upgradeable in-cluster plugins • Evolution:

    exec → RPC • Evolution: loose spec → tight • Containerized plugins FTW Lessons Learned gRPC Plugins Runtimes (CRI) Storage (CSI) Devices Key Management Networking (proposed)
  16. Controllers observe diff act

  17. Controllers THE fundamental design pattern in Kubernetes Examples: scheduler, kubelet,

    deployments, kube-proxy, cloud providers, load balancers, volume provisioners, auto-scalers, ... Allows automation & extension of almost any existing API
  18. resource resource resource Higher level of abstraction Lower level of

    abstraction
  19. Kubernetes is designed to leverage clouds Built-in cloud-provider API (i.e.

    send me a PR) is hooked into many core control loops Now 8 implementations (and huge LOC count), so moving out-of-tree Cloud Providers
  20. The API is a VIP (more or less) and virtual

    LB We ship a default implementation (kube-proxy), but that can be replaced Controller: watch the API server for Services and Endpoints, program $NETWORK Services
  21. But Wait, There’s More! • Secret management (KMS) • HTTP

    load-balancing (Ingress) • NetworkPolicy • DNS • Scheduler extenders & whole schedulers • ...and that’s JUST the infrastructure (i.e. boring) parts
  22. API Extensibility

  23. • Add new types of resources to your cluster •

    Add custom policy hooks ◦ to custom and built-in APIs • "APIs that add and modify APIs" API Extensibility
  24. • In Mac Edge, Windows Edge, and EE 2.0 •

    Supports API Extensions. • Certified Kubernetes • Docker Stacks uses API Extensions Kubernetes for Docker
  25. Exploring Stacks Follow along at https://goo.gl/JT7v8Z

  26. Exploring Stacks https://goo.gl/JT7v8Z $ cat docker-compose.yml version: "3.3" services: redis:

    image: redis:alpine ports: - 6379 networks: - frontend deploy: replicas: 1 networks: frontend:
  27. Exploring Stacks https://goo.gl/JT7v8Z $ docker stack deploy --compose-file docker-compose.yml stackdemo

    Waiting for the stack to be stable and running... - Service redis has one container running Stack stackdemo is stable and running
  28. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl config current-context docker-for-desktop

  29. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s
  30. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 1 1 1 1 2s
  31. compose resource service resource deployment resource Higher level of abstraction

    Lower level of abstraction
  32. Kubernetes API Server Service API compose resource Deployment API service

    resource deployment resource
  33. Kubernetes APIs Service API Deployment API service resource deployment resource

    something custom compose resource
  34. Kubernetes APIs Service API Deployment API service resource deployment resource

    dockerd hypothetical /stacks compose resource
  35. Kubernetes APIs Service API Deployment API service resource deployment resource

    dockerd hypothetical /stacks compose resource docker cli
  36. Exploring Stacks API https://goo.gl/JT7v8Z # last time... $ docker stack

    deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running
  37. Exploring Stacks API https://goo.gl/JT7v8Z № last time... $ docker stack

    deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running $ kubectl get stacks NAME AGE stackdemo 39s
  38. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001
  39. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001 $
  40. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001 $ kubectl get stacks -s localhost:8001
  41. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001 I0613 10:13:27.322416 82905 proxy_server.go:138] Filter accepting GET /apis/compose.docker.com/v1beta2/name spaces/default/stacks localhost $ kubectl get stacks -s localhost:8001 NAME AGE stackdemo 1m
  42. Kubernetes APIs Service API Deployment API service resource deployment resource

    kubectl compose.docker.com API compose resource
  43. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get apiservices.apiregistration.k8s.io NAME AGE

    v1. 29d v1.apps 29d ... v1beta2.compose.docker.com 29d v2beta1.autoscaling 29d
  44. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com Name:

    v1beta2.compose.docker.com ... API Version: apiregistration.k8s.io/v1beta1 Kind: APIService Metadata: ... Spec: ... Service: Name: compose-api Namespace: docker Status: Conditions: Message: all checks passed
  45. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d
  46. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d $ kubectl get deployments -n docker NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE compose 1 1 1 1 29d compose-api 1 1 1 1 29d
  47. Kubernetes APIs Service API API Registration API Kubernetes Cluster Deployment

    API Compose-API
  48. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API
  49. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API
  50. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API
  51. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API Compose
  52. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API Compose redis docker CLI
  53. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API Compose redis redis redis redis
  54. • Users ◦ Already have a client installed ◦ Already

    know how to find, trust it (TLS) and auth to the API • Controllers ◦ Can efficiently watch your resources • Admins ◦ Can separate your resources by Namespace ◦ Can authorize and audit log access to your resources Why Use an API Extension?
  55. API Aggregation & Extension API Servers (EAS) Extension API Server

    (EAS) API resource Controller
  56. API Aggregation & Extension API Servers (EAS) Extension API Server

    (EAS) API resource Controller
  57. Extension API Server (EAS) API Aggregation & Extension API Servers

    (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
  58. Extension API Server (EAS) API Aggregation & Extension API Servers

    (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
  59. Extension API Server (EAS) Extension API Server (EAS) Extension API

    Server (EAS) API resource Controller
  60. API resource Controller Custom Resource Definitions

  61. EAS Forked LoC: 0 Storage: provided Components: 1 Popularity: 100s

    Multiversioning: not yet Customizability: good CRD Forked LoC: 5000* Storage: you manage Components: 3 Popularity: 10s Multiversioning: yes Customizability: better * http://github.com/sample-apiserver
  62. Extension Ecosystem Devices 5 public plugins Storage 10 public plugins

    Networking >20 public plugins Custom APIs >400 Github Projects with custom APIs
  63. Extension Ecosystem • 4 Serverless frameworks • 6 PaaSes •

    10 CI/CD systems • 14 different database controllers • 4 popular ML toolkits
  64. Adding Types to the API • Extension API Servers •

    Custom Resource Definitions Adding Policy to the API • ValidatingAdmissionWebhooks • MutatingAdmissionWebhooks API Extensions
  65. Admission: After authn/z but before storing the change. Affects mutations,

    not reads. Webhooks: The API Server calls your URL, synchronously Run in cluster via service or outside, e.g. serverless. Admission Webhooks
  66. Old thinking: Better to make narrow specific interfaces, like ImagePolicyWebhook,

    for specific use cases. Can make easier to use. Overly general extensions may limit future optimization. Admission Webhooks
  67. New thinking: Many custom resoures. Cluster owners need to write

    policy for core resources and for custom resources written by 3rd parties. Need to compose policies written by different parties. Admission Webhooks
  68. Composability. Make all the changes before doing all the checks.

    MutatingAdmissionWebhooks - then- ValidatingAdmissionWebhooks Admission Webhooks
  69. Kelsey Hightower: - reject pods that set environment variables https://github.com/kelseyhightower/denyenv-validating-admission-webhook

    CRD Authors : - add complex validation Validating Admission Webhooks
  70. Istio: inject sidecar into all the pods Service Catalog: inject

    credentials into Mutating Admission Webhooks
  71. - Mutate the pod template of a deployment - Install

    a flaky webhook matching all resources. Bad Ideas
  72. •Kubernetes for Docker: • Super easy way to try Kubernetes

    •API Extensions: • Use them. Author them. On Docker. For Kubernetes. •Try it: • https://goo.gl/JT7v8Z Conclusion
  73. v Questions? Learn more: https://goo.gl/JT7v8Z Thanks!