Dockercon_2018_-_Kubernetes_Extensibility.pdf

Slide 1

Slide 1 text

Eric Tune & Tim Hockin Google Kubernetes Extensibility

Slide 2

Slide 2 text

Kubernetes is a container management system

Slide 3

Slide 3 text

Kubernetes is a container management system platform

Slide 4

Slide 4 text

What is Kubernetes? ...an abstraction layer over infrastructure ...a framework for declarative APIs and distributed control Infrastructure Extensibility API Extensibility

Slide 5

Slide 5 text

Extensibility Goals Infrastructure Extensibility API Extensibility Support portability Support customization Autonomy Autonomy Scalable growth of project Encourage new uses

Slide 6

Slide 6 text

A major focus of the last 2 years of development From infrastructure to APIs, we have over a dozen extension points We have WAY more material than time! https://goo.gl/2qz8jW Kubernetes & Extensibility

Slide 7

Slide 7 text

Infrastructure Extensibility

Slide 8

Slide 8 text

Networks are like snowflakes There is no “one size fits all” for almost anything networking related We needed a way for users to customize how Kubernetes consumes networking infra Network Plugins

Slide 9

Slide 9 text

Old: built-in “plugins” (aka “send Tim a PR”) New: CNI - Container Network Interface ● Started by CoreOS, now CNCF with community ● “exec” interface with stdin/stdout/env API Widely used, also by other projects (e.g. Mesos) Underpins the default impl in Kubernetes Network Plugins (present)

Slide 10

Slide 10 text

Proposal open for a gRPC based API which covers more than just interfaces and IPAM Tighter coupling with Service API seems valuable Proposals open for multi-IP and multi-network Network Plugins (future)

Slide 11

Slide 11 text

Many storage technologies - physical and virtual, block and file ● Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ... Many vendors want their products to support Kubernetes Storage Plugins

Slide 12

Slide 12 text

Old: built-in “plugins” (aka “send Tim a PR”) Old: Volume “flex” plugins via “exec” New: CSI - Container Storage Interface ● Collaboration: Google, Mesosphere, Docker, Cloud Foundry ● gRPC spec, with Kubernetes-specific adaptors ● In development now, alpha in Kubernetes 1.10 Plan to transition most in-tree plugins to CSI Storage Plugins (present)

Slide 13

Slide 13 text

GPUs and other “accelerator” hardware is becoming very common Part of the larger resource model in Kubernetes gRPC based plugins Beta in Kubernetes 1.10 Device Plugins

Slide 14

Slide 14 text

Docker was baked-in, but people wanted to try new and interesting ideas ● rkt, Containerd, CRI-O ● Kata containers, Hyper.sh, gVisor Making it a plugin made the code better: win-win! CRI - gRPC based plugins Container Runtimes

Slide 15

Slide 15 text

● Stateful, daemon plugins ● Upgradeable in-cluster plugins ● Evolution: exec → RPC ● Evolution: loose spec → tight ● Containerized plugins FTW Lessons Learned gRPC Plugins Runtimes (CRI) Storage (CSI) Devices Key Management Networking (proposed)

Slide 16

Slide 16 text

Controllers observe diff act

Slide 17

Slide 17 text

Controllers THE fundamental design pattern in Kubernetes Examples: scheduler, kubelet, deployments, kube-proxy, cloud providers, load balancers, volume provisioners, auto-scalers, ... Allows automation & extension of almost any existing API

Slide 18

Slide 18 text

resource resource resource Higher level of abstraction Lower level of abstraction

Slide 19

Slide 19 text

Kubernetes is designed to leverage clouds Built-in cloud-provider API (i.e. send me a PR) is hooked into many core control loops Now 8 implementations (and huge LOC count), so moving out-of-tree Cloud Providers

Slide 20

Slide 20 text

The API is a VIP (more or less) and virtual LB We ship a default implementation (kube-proxy), but that can be replaced Controller: watch the API server for Services and Endpoints, program $NETWORK Services

Slide 21

Slide 21 text

But Wait, There’s More! ● Secret management (KMS) ● HTTP load-balancing (Ingress) ● NetworkPolicy ● DNS ● Scheduler extenders & whole schedulers ● ...and that’s JUST the infrastructure (i.e. boring) parts

Slide 22

Slide 22 text

API Extensibility

Slide 23

Slide 23 text

● Add new types of resources to your cluster ● Add custom policy hooks ○ to custom and built-in APIs ● "APIs that add and modify APIs" API Extensibility

Slide 24

Slide 24 text

● In Mac Edge, Windows Edge, and EE 2.0 ● Supports API Extensions. ● Certified Kubernetes ● Docker Stacks uses API Extensions Kubernetes for Docker

Slide 25

Slide 25 text

Exploring Stacks Follow along at https://goo.gl/JT7v8Z

Slide 26

Slide 26 text

Exploring Stacks https://goo.gl/JT7v8Z $ cat docker-compose.yml version: "3.3" services: redis: image: redis:alpine ports: - 6379 networks: - frontend deploy: replicas: 1 networks: frontend:

Slide 27

Slide 27 text

Exploring Stacks https://goo.gl/JT7v8Z $ docker stack deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service redis has one container running Stack stackdemo is stable and running

Slide 28

Slide 28 text

Exploring Stacks https://goo.gl/JT7v8Z $ kubectl config current-context docker-for-desktop

Slide 29

Slide 29 text

Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 29d redis ClusterIP None 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 6379:31248/TCP 1s

Slide 30

Slide 30 text

Slide 31

Slide 31 text

compose resource service resource deployment resource Higher level of abstraction Lower level of abstraction

Slide 32

Slide 32 text

Kubernetes API Server Service API compose resource Deployment API service resource deployment resource

Slide 33

Slide 33 text

Kubernetes APIs Service API Deployment API service resource deployment resource something custom compose resource

Slide 34

Slide 34 text

Kubernetes APIs Service API Deployment API service resource deployment resource dockerd hypothetical /stacks compose resource

Slide 35

Slide 35 text

Kubernetes APIs Service API Deployment API service resource deployment resource dockerd hypothetical /stacks compose resource docker cli

Slide 36

Slide 36 text

Exploring Stacks API https://goo.gl/JT7v8Z # last time... $ docker stack deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running

Slide 37

Slide 37 text

Exploring Stacks API https://goo.gl/JT7v8Z № last time... $ docker stack deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running $ kubectl get stacks NAME AGE stackdemo 39s

Slide 38

Slide 38 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001

Slide 39

Slide 39 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 $

Slide 40

Slide 40 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 $ kubectl get stacks -s localhost:8001

Slide 41

Slide 41 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 I0613 10:13:27.322416 82905 proxy_server.go:138] Filter accepting GET /apis/compose.docker.com/v1beta2/name spaces/default/stacks localhost $ kubectl get stacks -s localhost:8001 NAME AGE stackdemo 1m

Slide 42

Slide 42 text

Kubernetes APIs Service API Deployment API service resource deployment resource kubectl compose.docker.com API compose resource

Slide 43

Slide 43 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get apiservices.apiregistration.k8s.io NAME AGE v1. 29d v1.apps 29d ... v1beta2.compose.docker.com 29d v2beta1.autoscaling 29d

Slide 44

Slide 44 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com Name: v1beta2.compose.docker.com ... API Version: apiregistration.k8s.io/v1beta1 Kind: APIService Metadata: ... Spec: ... Service: Name: compose-api Namespace: docker Status: Conditions: Message: all checks passed

Slide 45

Slide 45 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 443/TCP 17d

Slide 46

Slide 46 text

Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 443/TCP 17d $ kubectl get deployments -n docker NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE compose 1 1 1 1 29d compose-api 1 1 1 1 29d

Slide 47

Slide 47 text

Kubernetes APIs Service API API Registration API Kubernetes Cluster Deployment API Compose-API

Slide 48

Slide 48 text

Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API

Slide 49

Slide 49 text

Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API

Slide 50

Slide 50 text

Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API

Slide 51

Slide 51 text

Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API Compose

Slide 52

Slide 52 text

Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API Compose redis docker CLI

Slide 53

Slide 53 text

Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API Compose redis redis redis redis

Slide 54

Slide 54 text

● Users ○ Already have a client installed ○ Already know how to find, trust it (TLS) and auth to the API ● Controllers ○ Can efficiently watch your resources ● Admins ○ Can separate your resources by Namespace ○ Can authorize and audit log access to your resources Why Use an API Extension?

Slide 55

Slide 55 text

API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) API resource Controller

Slide 56

Slide 56 text

API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) API resource Controller

Slide 57

Slide 57 text

Extension API Server (EAS) API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller

Slide 58

Slide 58 text

Extension API Server (EAS) API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller

Slide 59

Slide 59 text

Extension API Server (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller

Slide 60

Slide 60 text

API resource Controller Custom Resource Definitions

Slide 61

Slide 61 text

EAS Forked LoC: 0 Storage: provided Components: 1 Popularity: 100s Multiversioning: not yet Customizability: good CRD Forked LoC: 5000* Storage: you manage Components: 3 Popularity: 10s Multiversioning: yes Customizability: better * http://github.com/sample-apiserver

Slide 62

Slide 62 text

Extension Ecosystem Devices 5 public plugins Storage 10 public plugins Networking >20 public plugins Custom APIs >400 Github Projects with custom APIs

Slide 63

Slide 63 text

Extension Ecosystem ● 4 Serverless frameworks ● 6 PaaSes ● 10 CI/CD systems ● 14 different database controllers ● 4 popular ML toolkits

Slide 64

Slide 64 text

Adding Types to the API ● Extension API Servers ● Custom Resource Definitions Adding Policy to the API ● ValidatingAdmissionWebhooks ● MutatingAdmissionWebhooks API Extensions

Slide 65

Slide 65 text

Admission: After authn/z but before storing the change. Affects mutations, not reads. Webhooks: The API Server calls your URL, synchronously Run in cluster via service or outside, e.g. serverless. Admission Webhooks

Slide 66

Slide 66 text

Old thinking: Better to make narrow specific interfaces, like ImagePolicyWebhook, for specific use cases. Can make easier to use. Overly general extensions may limit future optimization. Admission Webhooks

Slide 67

Slide 67 text

New thinking: Many custom resoures. Cluster owners need to write policy for core resources and for custom resources written by 3rd parties. Need to compose policies written by different parties. Admission Webhooks

Slide 68

Slide 68 text

Composability. Make all the changes before doing all the checks. MutatingAdmissionWebhooks - then- ValidatingAdmissionWebhooks Admission Webhooks

Slide 69

Slide 69 text

Kelsey Hightower: - reject pods that set environment variables https://github.com/kelseyhightower/denyenv-validating-admission-webhook CRD Authors : - add complex validation Validating Admission Webhooks

Slide 70

Slide 70 text

Istio: inject sidecar into all the pods Service Catalog: inject credentials into Mutating Admission Webhooks

Slide 71

Slide 71 text

- Mutate the pod template of a deployment - Install a flaky webhook matching all resources. Bad Ideas

Slide 72

Slide 72 text

•Kubernetes for Docker: • Super easy way to try Kubernetes •API Extensions: • Use them. Author them. On Docker. For Kubernetes. •Try it: • https://goo.gl/JT7v8Z Conclusion

Slide 73

Slide 73 text

v Questions? Learn more: https://goo.gl/JT7v8Z Thanks!