Kubernetes: The Latest and Greatest

Kubernetes is a open-source system, developed by Google, for automating the deployment, management, and scaling of containerized applications across a distributed cluster. Many companies are standardizing on Kubernetes as a technology for container orchestration including Red Hat, eBay, and of course Google. Kubernetes aims to provide better ways of managing related, distributed components across varied infrastructure. In this session, we'll take a detailed look at Kubernetes, the open source orchestration system for containers, and explore the key terms, concepts, and vocabulary introduced by the project. We'll also explore key features and improvement in Kubernetes 1.2 & 1.3 and how to use them in practice.

Transcript

1. Ian Lewis Developer Advocate, Google Cloud Platform Kubernetes: The Latest and Greatest #containercon Japan
2. Confidential & Proprietary Google Cloud Platform 2 Ian Lewis Developer Advocate - Google Cloud Platform Tokyo, Japan +Ian Lewis @IanMLewis
3. Agenda • Kubernetes Architecture • Concepts, Lots of Concepts • What’s new in 1.3
4. Confidential & Proprietary Google Cloud Platform 4 Why did we build this thing?
5. For the last 15 years Google has been building the world’s fastest, most powerful infrastructure.
7. Cloud Technology Innovations 2012 2013 MapReduce Spanner/F1 2003 2006 2007 2010 2011 GFS Omega Colossus Cloud Storage Dremel BigQuery Big Table Cloud Datastore Paxos impl. 2004 Cloud Bigtable
8. Copyright 2015 Google Inc Google has been running all our services in Containers for over 10 years. We start over 2 billion containers every week. Images by Connie Zhou
9. http://research.google.com/pubs/pub43438.html
10. Image by Connie Zhou
11. job hello_world = { runtime = { cell = 'ic' } // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks } 10000 Developer View
12. web browsers BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Binary Developer View What just happened?
13. Developer View
14. Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Image by Connie Zhou Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!
15. Confidential & Proprietary Google Cloud Platform 15 Kubernetes
16. Enter Kubernetes Greek for “Helmsman”; also the root of the word “Governor” • Container orchestrator • Runs containers • Supports multiple cloud and bare- metal environments • Inspired and informed by Google’s experiences and internal systems • Open source, written in Go Manage applications, not machines
17. Google Cloud Platform UI CLI API users master nodes etcd scheduler controllers master The 10000 Foot View kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet
18. web browsers BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Manifest Master Kubelet Kubelet Kubelet Binary Developer View Docker Image Docker Hub/Private Repo What just happened? Kubelet
19. Confidential & Proprietary Google Cloud Platform 19 Pods
20. Google Cloud Platform Pods (TODO) docker … --net=container: id --ipc=container:id -- pid=container:id https://github. com/docker/docker/issue s/10163 IPC Network PID Hostname cgroup Web Server cgroup File Puller localhost
21. Google Cloud Platform Arbitrary metadata Attached to any API object Generally represent identity Queryable by selectors • think SQL ‘select ... where ...’ The only grouping mechanism • pods under a ReplicationController • pods in a Service • capabilities of a node (constraints) Labels
22. Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE Selectors
23. Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp Selectors
24. Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = FE Selectors
25. Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = BE Selectors
26. Google Cloud Platform Selectors App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = prod
27. Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = test Selectors
28. Confidential & Proprietary Google Cloud Platform 28 Configuration
29. Google Cloud Platform ConfigMaps Goal: manage app configuration • ...without making overly-brittle container images 12-factor says config comes from the environment • Kubernetes is the environment Manage config via the Kubernetes API Inject config as a virtual volume into your Pods • late-binding, live-updated (atomic) • also available as env vars Status: GA in Kubernetes v1.2 node API Pod Config Map
30. Google Cloud Platform Secrets Goal: grant a pod access to a secured something • don’t put secrets in the container image! 12-factor says config comes from the environment • Kubernetes is the environment Manage secrets via the Kubernetes API Inject secrets as virtual volumes into your Pods • late-binding, tmpfs - never touches disk • also available as env vars node API Pod Secret
31. Confidential & Proprietary Google Cloud Platform 31 Volumes
32. Google Cloud Platform PersistentVolumes A higher-level storage abstraction • insulation from any one cloud environment Admin provisions them, users claim them • NEW: auto-provisioning (alpha in v1.2) Independent lifetime from consumers • lives until user is done with it • can be handed-off between pods Dynamically “scheduled” and managed, like nodes and pods Claim
33. Google Cloud Platform PersistentVolumes Cluster Admin
34. Google Cloud Platform PersistentVolumes Provision Cluster Admin PersistentVolumes
35. Google Cloud Platform PersistentVolumes User Cluster Admin PersistentVolumes
36. Google Cloud Platform PersistentVolumes User PVClaim Create Cluster Admin PersistentVolumes
37. Google Cloud Platform PersistentVolumes User PVClaim Binder Cluster Admin PersistentVolumes
38. Google Cloud Platform PersistentVolumes User PVClaim Pod Create Cluster Admin PersistentVolumes
39. Google Cloud Platform PersistentVolumes User PVClaim Pod Cluster Admin PersistentVolumes *
40. Google Cloud Platform PersistentVolumes User PVClaim Pod Delete * Cluster Admin PersistentVolumes *
41. Google Cloud Platform PersistentVolumes User PVClaim Cluster Admin PersistentVolumes *
42. Google Cloud Platform PersistentVolumes User PVClaim Pod Create Cluster Admin PersistentVolumes *
43. Google Cloud Platform PersistentVolumes User PVClaim Pod Cluster Admin PersistentVolumes *
44. Google Cloud Platform PersistentVolumes User PVClaim Pod Delete Cluster Admin PersistentVolumes *
45. Google Cloud Platform PersistentVolumes User PVClaim Delete Cluster Admin PersistentVolumes *
46. Google Cloud Platform PersistentVolumes User Recycler Cluster Admin PersistentVolumes
47. Confidential & Proprietary Google Cloud Platform 47 Controllers: Deployments & ReplicaSets
48. Google Cloud Platform The 10000 Foot View etcd scheduler controller-manager apiserver kubelet docker kube-proxy iptables 1. User creates Deployment 2. API server saves info to etcd 3. CM finds Deployment and creates ReplicaSet, which creates Pods (unscheduled) 4. Scheduler schedules pods. 5. Kubelet sees pod scheduled to it and tells docker to run the container. 6. Docker pulls and runs the container. 1 2 6 3 5 4 Docker Hub / GCR
49. Google Cloud Platform observe diff act Deployments & ReplicaSets
50. Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
51. Google Cloud Platform Deployments ReplicaSet - replicas: 4 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
52. Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
53. Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
54. Google Cloud Platform Rolling Updates ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl apply ...
55. Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Rolling Updates ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v2 Deployment - name: MyApp
56. Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
57. Google Cloud Platform ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
58. Google Cloud Platform ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
59. Google Cloud Platform ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
60. Google Cloud Platform ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
61. Google Cloud Platform ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
62. Confidential & Proprietary Google Cloud Platform 62 Services
63. Google confidential │ Do not distribute Services A group of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Gets a stable virtual IP and port • sometimes called the service portal • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables when backends change Hides complexity - ideal for non-native apps Virtual IP Client
64. Google Cloud Platform External Services Services VIPs are only available inside the cluster Need to receive traffic from “the outside world” Service “type” • NodePort: expose on a port on every node • LoadBalancer: provision a cloud load-balancer DiY load-balancer solutions • socat (for nodePort remapping) • haproxy • nginx Ingress (L7 LB)
65. Google Cloud Platform Ingress (L7) Many apps are HTTP/HTTPS Services are L4 (IP + port) Ingress maps incoming traffic to backend services • by HTTP host headers • by HTTP URL paths HAProxy, NGINX, AWS and GCE implementations in progress Now with SSL! Status: BETA in Kubernetes v1.2 Client URL Map
66. Confidential & Proprietary Google Cloud Platform 66 Health Checks
67. Google confidential │ Do not distribute Health Checks Makes sure containers are healthy. Includes liveness and readiness probes. Liveness Probes • Checks if a container is alive. Hasn’t crashed or deadlocked. Readiness Probes • Checks to make sure a container is ready to serve traffic.
68. Google confidential │ Do not distribute Liveness Checks if a container is alive. Hasn’t crashed or deadlocked. If a container fails a liveness check too many times it will be restarted.
69. Google confidential │ Do not distribute Readiness Checks if a container is ready to serve traffic. Typically, an app will check to make sure it’s dependencies are available. If a container fails a readiness check it many times it’s pod is removed from service endpoints. This makes sure services only use pods that can serve traffic. Virtual IP Client
70. Google Cloud Platform Graceful Termination Goal: Give pods time to clean up • finish in-flight operations • log state • flush to disk • 30 seconds by default Catch SIGTERM, cleanup, exit ASAP Pod status “Terminating” Declarative: ‘DELETE’ appears as an object field in the API
71. Confidential & Proprietary Google Cloud Platform 71 DaemonSets
72. Google Cloud Platform DaemonSets Problem: how to run a Pod on every node? • or a subset of nodes Similar to ReplicaSet • principle: do one thing, don’t overload “Which nodes?” is a selector Use familiar tools and patterns Status: BETA in Kubernetes v1.2 Pod
73. Confidential & Proprietary Google Cloud Platform 73 Jobs
74. Google Cloud Platform Jobs Run-to-completion, as opposed to run-forever • Express parallelism vs. required completions • Workflow: restart on failure • Build/test: don’t restart on failure Aggregates success/failure counts Built for batch and big-data work Status: GA in Kubernetes v1.2 ...
75. Confidential & Proprietary Google Cloud Platform 75 Namespaces
76. Google Cloud Platform Namespaces Problem: I have too much stuff! • name collisions in the API • poor isolation between users • don’t want to expose things like Secrets Solution: Slice up the cluster • create new Namespaces as needed • per-user, per-app, per-department, etc. • part of the API - NOT private machines • most API objects are namespaced • part of the REST URL path • Namespaces are just another API object • One-step cleanup - delete the Namespace • Obvious hook for policy enforcement (e.g. quota)
77. Google Cloud Platform Resource Isolation Principles: • Apps must not be able to affect each other’s performance • if so it is an isolation failure • Repeated runs of the same app should see ~equal behavior • QoS levels drives resource decisions in (soft) real-time • Correct in all cases, optimal in some • reduce unreliable components • SLOs are the lingua franca
78. Google Cloud Platform Pros: • Sharing - users don’t worry about interference (aka the noisy neighbor problem) • Predictable - allows us to offer strong SLAs to apps Cons: • Stranding - arbitrary slices mean some resources get lost • Confusing - how do I know how much I need? • analog: what size VM should I use? • smart auto-scaling is needed! • Expensive - you pay for certainty In reality this is a multi-dimensional bin-packing problem: CPU, memory, disk space, IO bandwidth, network bandwidth, ... Strong isolation
79. Google Cloud Platform Requests and Limits Request: • how much of a resource you are asking to use, with a strong guarantee of availability • CPU (seconds/second) • RAM (bytes) • scheduler will not over-commit requests Limit: • max amount of a resource you can access Repercussions: • Usage > Request: resources might be available • Usage > Limit: throttled or killed
80. Google Cloud Platform Quality of Service Defined in terms of Request and Limit Guaranteed: highest protection • request > 0 && limit == request Burstable: medium protection • request > 0 && limit > request Best Effort: lowest protection • request == 0 What does “protection” mean? • OOM score • CPU scheduling
81. Google Cloud Platform ResourceQuota Admission control: apply limits in aggregate Per-namespace: ensure no user/app/department abuses the cluster Reminiscent of disk quota by design Applies to each type of resource • CPU and memory for now Disallows pods without resources
82. Google Cloud Platform LimitRange Admission control: limit the limits • min and max • ratio of limit/request Default values for unspecified limits Per-namespace Together with ResourceQuota gives cluster admins powerful tools
83. Confidential & Proprietary Google Cloud Platform 83 Kubernetes 1.3
84. Kubernetes 1.3 • Supports up to 60,000 containers on 2000 nodes • PetSet (alpha) • Federation • Cross cluster service discovery • New in 1.2: • Deployments • ConfigMaps
85. Google Cloud Platform PetSets (working name) Goal: enable clustered software on Kubernetes • mysql, redis, zookeeper, ... Clustered apps need “identity” and sequencing guarantees • stable hostname, available in DNS • an ordinal index • stable storage: linked to the ordinal & hostname • discovery of peers for quorum • startup/teardown ordering Status: ALPHA in Kubernetes v1.3
86. Google Container Engine - Inspired by a decade within Google - Reimagines Cluster computing - Designed for a multi-cloud world Photo by Connie Zhou
87. Google Container Engine New service for cluster-based compute • Provisioned cluster in seconds. Fully configured. • Fine-grained control over cluster. • Designed for multi-cloud. Runs Kubernetes. Releases • Now GA!! • No additional cost for up to 5 nodes Resources • Google Container Engine: http://cloud.google.com/container-engine • Kubernetes: http://kubernetes.io
88. Thank You