Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dockercon_2018_-_Kubernetes_Extensibility.pdf

Tim Hockin
June 16, 2018
1.2k

 Dockercon_2018_-_Kubernetes_Extensibility.pdf

Tim Hockin

June 16, 2018
Tweet

Transcript

  1. What is Kubernetes? ...an abstraction layer over infrastructure ...a framework

    for declarative APIs and distributed control Infrastructure Extensibility API Extensibility
  2. A major focus of the last 2 years of development

    From infrastructure to APIs, we have over a dozen extension points We have WAY more material than time! https://goo.gl/2qz8jW Kubernetes & Extensibility
  3. Networks are like snowflakes There is no “one size fits

    all” for almost anything networking related We needed a way for users to customize how Kubernetes consumes networking infra Network Plugins
  4. Old: built-in “plugins” (aka “send Tim a PR”) New: CNI

    - Container Network Interface • Started by CoreOS, now CNCF with community • “exec” interface with stdin/stdout/env API Widely used, also by other projects (e.g. Mesos) Underpins the default impl in Kubernetes Network Plugins (present)
  5. Proposal open for a gRPC based API which covers more

    than just interfaces and IPAM Tighter coupling with Service API seems valuable Proposals open for multi-IP and multi-network Network Plugins (future)
  6. Many storage technologies - physical and virtual, block and file

    • Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ... Many vendors want their products to support Kubernetes Storage Plugins
  7. Old: built-in “plugins” (aka “send Tim a PR”) Old: Volume

    “flex” plugins via “exec” New: CSI - Container Storage Interface • Collaboration: Google, Mesosphere, Docker, Cloud Foundry • gRPC spec, with Kubernetes-specific adaptors • In development now, alpha in Kubernetes 1.10 Plan to transition most in-tree plugins to CSI Storage Plugins (present)
  8. GPUs and other “accelerator” hardware is becoming very common Part

    of the larger resource model in Kubernetes gRPC based plugins Beta in Kubernetes 1.10 Device Plugins
  9. Docker was baked-in, but people wanted to try new and

    interesting ideas • rkt, Containerd, CRI-O • Kata containers, Hyper.sh, gVisor Making it a plugin made the code better: win-win! CRI - gRPC based plugins Container Runtimes
  10. • Stateful, daemon plugins • Upgradeable in-cluster plugins • Evolution:

    exec → RPC • Evolution: loose spec → tight • Containerized plugins FTW Lessons Learned gRPC Plugins Runtimes (CRI) Storage (CSI) Devices Key Management Networking (proposed)
  11. Controllers THE fundamental design pattern in Kubernetes Examples: scheduler, kubelet,

    deployments, kube-proxy, cloud providers, load balancers, volume provisioners, auto-scalers, ... Allows automation & extension of almost any existing API
  12. Kubernetes is designed to leverage clouds Built-in cloud-provider API (i.e.

    send me a PR) is hooked into many core control loops Now 8 implementations (and huge LOC count), so moving out-of-tree Cloud Providers
  13. The API is a VIP (more or less) and virtual

    LB We ship a default implementation (kube-proxy), but that can be replaced Controller: watch the API server for Services and Endpoints, program $NETWORK Services
  14. But Wait, There’s More! • Secret management (KMS) • HTTP

    load-balancing (Ingress) • NetworkPolicy • DNS • Scheduler extenders & whole schedulers • ...and that’s JUST the infrastructure (i.e. boring) parts
  15. • Add new types of resources to your cluster •

    Add custom policy hooks ◦ to custom and built-in APIs • "APIs that add and modify APIs" API Extensibility
  16. • In Mac Edge, Windows Edge, and EE 2.0 •

    Supports API Extensions. • Certified Kubernetes • Docker Stacks uses API Extensions Kubernetes for Docker
  17. Exploring Stacks https://goo.gl/JT7v8Z $ cat docker-compose.yml version: "3.3" services: redis:

    image: redis:alpine ports: - 6379 networks: - frontend deploy: replicas: 1 networks: frontend:
  18. Exploring Stacks https://goo.gl/JT7v8Z $ docker stack deploy --compose-file docker-compose.yml stackdemo

    Waiting for the stack to be stable and running... - Service redis has one container running Stack stackdemo is stable and running
  19. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s
  20. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP

    EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 1 1 1 1 2s
  21. Kubernetes APIs Service API Deployment API service resource deployment resource

    dockerd hypothetical /stacks compose resource docker cli
  22. Exploring Stacks API https://goo.gl/JT7v8Z # last time... $ docker stack

    deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running
  23. Exploring Stacks API https://goo.gl/JT7v8Z № last time... $ docker stack

    deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running $ kubectl get stacks NAME AGE stackdemo 39s
  24. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001 $ kubectl get stacks -s localhost:8001
  25. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting

    to serve on 127.0.0.1:8001 I0613 10:13:27.322416 82905 proxy_server.go:138] Filter accepting GET /apis/compose.docker.com/v1beta2/name spaces/default/stacks localhost $ kubectl get stacks -s localhost:8001 NAME AGE stackdemo 1m
  26. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get apiservices.apiregistration.k8s.io NAME AGE

    v1. 29d v1.apps 29d ... v1beta2.compose.docker.com 29d v2beta1.autoscaling 29d
  27. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com Name:

    v1beta2.compose.docker.com ... API Version: apiregistration.k8s.io/v1beta1 Kind: APIService Metadata: ... Spec: ... Service: Name: compose-api Namespace: docker Status: Conditions: Message: all checks passed
  28. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d
  29. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker

    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d $ kubectl get deployments -n docker NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE compose 1 1 1 1 29d compose-api 1 1 1 1 29d
  30. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API
  31. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API
  32. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API Compose
  33. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API Compose redis docker CLI
  34. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes

    Cluster Deployment API Compose-API compose.docker.com API Compose redis redis redis redis
  35. • Users ◦ Already have a client installed ◦ Already

    know how to find, trust it (TLS) and auth to the API • Controllers ◦ Can efficiently watch your resources • Admins ◦ Can separate your resources by Namespace ◦ Can authorize and audit log access to your resources Why Use an API Extension?
  36. Extension API Server (EAS) API Aggregation & Extension API Servers

    (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
  37. Extension API Server (EAS) API Aggregation & Extension API Servers

    (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
  38. EAS Forked LoC: 0 Storage: provided Components: 1 Popularity: 100s

    Multiversioning: not yet Customizability: good CRD Forked LoC: 5000* Storage: you manage Components: 3 Popularity: 10s Multiversioning: yes Customizability: better * http://github.com/sample-apiserver
  39. Extension Ecosystem Devices 5 public plugins Storage 10 public plugins

    Networking >20 public plugins Custom APIs >400 Github Projects with custom APIs
  40. Extension Ecosystem • 4 Serverless frameworks • 6 PaaSes •

    10 CI/CD systems • 14 different database controllers • 4 popular ML toolkits
  41. Adding Types to the API • Extension API Servers •

    Custom Resource Definitions Adding Policy to the API • ValidatingAdmissionWebhooks • MutatingAdmissionWebhooks API Extensions
  42. Admission: After authn/z but before storing the change. Affects mutations,

    not reads. Webhooks: The API Server calls your URL, synchronously Run in cluster via service or outside, e.g. serverless. Admission Webhooks
  43. Old thinking: Better to make narrow specific interfaces, like ImagePolicyWebhook,

    for specific use cases. Can make easier to use. Overly general extensions may limit future optimization. Admission Webhooks
  44. New thinking: Many custom resoures. Cluster owners need to write

    policy for core resources and for custom resources written by 3rd parties. Need to compose policies written by different parties. Admission Webhooks
  45. Composability. Make all the changes before doing all the checks.

    MutatingAdmissionWebhooks - then- ValidatingAdmissionWebhooks Admission Webhooks
  46. Istio: inject sidecar into all the pods Service Catalog: inject

    credentials into Mutating Admission Webhooks
  47. - Mutate the pod template of a deployment - Install

    a flaky webhook matching all resources. Bad Ideas
  48. •Kubernetes for Docker: • Super easy way to try Kubernetes

    •API Extensions: • Use them. Author them. On Docker. For Kubernetes. •Try it: • https://goo.gl/JT7v8Z Conclusion