Share
Embed
Kubernetes: The Latest and Greatest
Kubernetes is a open-source system, developed by Google, for automating the deployment, management, and scaling of containerized applications across a distributed cluster. Many companies are standardizing on Kubernetes as a technology for container orchestration including Red Hat, eBay, and of course Google. Kubernetes aims to provide better ways of managing related, distributed components across varied infrastructure. In this session, we'll take a detailed look at Kubernetes, the open source orchestration system for containers, and explore the key terms, concepts, and vocabulary introduced by the project. We'll also explore key features and improvement in Kubernetes 1.2 & 1.3 and how to use them in practice.
Ian Lewis
July 15, 2016
Transcript
88 slides
-
Ian Lewis Developer Advocate, Google Cloud Platform Kubernetes: The Latest
and Greatest #containercon Japan -
Confidential & Proprietary Google Cloud Platform 2 Ian Lewis Developer
Advocate - Google Cloud Platform Tokyo, Japan +Ian Lewis @IanMLewis -
Agenda • Kubernetes Architecture • Concepts, Lots of Concepts •
What’s new in 1.3 -
Confidential & Proprietary Google Cloud Platform 4 Why did we
build this thing? -
For the last 15 years Google has been building the
world’s fastest, most powerful infrastructure. -
Cloud Technology Innovations 2012 2013 MapReduce Spanner/F1 2003 2006 2007
2010 2011 GFS Omega Colossus Cloud Storage Dremel BigQuery Big Table Cloud Datastore Paxos impl. 2004 Cloud Bigtable -
Copyright 2015 Google Inc Google has been running all our
services in Containers for over 10 years. We start over 2 billion containers every week. Images by Connie Zhou -
job hello_world = { runtime = { cell = 'ic'
} // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks } 10000 Developer View -
web browsers BorgMaster link shard UI shard BorgMaster link shard
UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Binary Developer View What just happened? -
Hello world! Hello world! Hello world! Hello world! Hello world!
Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Image by Connie Zhou Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! -
Enter Kubernetes Greek for “Helmsman”; also the root of the
word “Governor” • Container orchestrator • Runs containers • Supports multiple cloud and bare- metal environments • Inspired and informed by Google’s experiences and internal systems • Open source, written in Go Manage applications, not machines -
Google Cloud Platform UI CLI API users master nodes etcd
scheduler controllers master The 10000 Foot View kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet -
web browsers BorgMaster link shard UI shard BorgMaster link shard
UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Manifest Master Kubelet Kubelet Kubelet Binary Developer View Docker Image Docker Hub/Private Repo What just happened? Kubelet -
Google Cloud Platform Pods (TODO) docker … --net=container: id --ipc=container:id
-- pid=container:id https://github. com/docker/docker/issue s/10163 IPC Network PID Hostname cgroup Web Server cgroup File Puller localhost -
Google Cloud Platform Arbitrary metadata Attached to any API object
Generally represent identity Queryable by selectors • think SQL ‘select ... where ...’ The only grouping mechanism • pods under a ReplicationController • pods in a Service • capabilities of a node (constraints) Labels -
Google Cloud Platform App: MyApp Phase: prod Role: FE App:
MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE Selectors -
Google Cloud Platform App: MyApp Phase: prod Role: FE App:
MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp Selectors -
Google Cloud Platform App: MyApp Phase: prod Role: FE App:
MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = FE Selectors -
Google Cloud Platform App: MyApp Phase: prod Role: FE App:
MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = BE Selectors -
Google Cloud Platform Selectors App: MyApp Phase: prod Role: FE
App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = prod -
Google Cloud Platform App: MyApp Phase: prod Role: FE App:
MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = test Selectors -
Google Cloud Platform ConfigMaps Goal: manage app configuration • ...without
making overly-brittle container images 12-factor says config comes from the environment • Kubernetes is the environment Manage config via the Kubernetes API Inject config as a virtual volume into your Pods • late-binding, live-updated (atomic) • also available as env vars Status: GA in Kubernetes v1.2 node API Pod Config Map -
Google Cloud Platform Secrets Goal: grant a pod access to
a secured something • don’t put secrets in the container image! 12-factor says config comes from the environment • Kubernetes is the environment Manage secrets via the Kubernetes API Inject secrets as virtual volumes into your Pods • late-binding, tmpfs - never touches disk • also available as env vars node API Pod Secret -
Google Cloud Platform PersistentVolumes A higher-level storage abstraction • insulation
from any one cloud environment Admin provisions them, users claim them • NEW: auto-provisioning (alpha in v1.2) Independent lifetime from consumers • lives until user is done with it • can be handed-off between pods Dynamically “scheduled” and managed, like nodes and pods Claim -
Google Cloud Platform PersistentVolumes User PVClaim Pod Delete * Cluster
Admin PersistentVolumes * -
Google Cloud Platform PersistentVolumes User PVClaim Pod Create Cluster Admin
PersistentVolumes * -
Google Cloud Platform PersistentVolumes User PVClaim Pod Delete Cluster Admin
PersistentVolumes * -
Google Cloud Platform The 10000 Foot View etcd scheduler controller-manager
apiserver kubelet docker kube-proxy iptables 1. User creates Deployment 2. API server saves info to etcd 3. CM finds Deployment and creates ReplicaSet, which creates Pods (unscheduled) 4. Scheduler schedules pods. 5. Kubelet sees pod scheduled to it and tells docker to run the container. 6. Docker pulls and runs the container. 1 2 6 3 5 4 Docker Hub / GCR -
Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector:
- app: MyApp - version: v1 Deployment - name: MyApp kubectl create ... -
Google Cloud Platform Deployments ReplicaSet - replicas: 4 - selector:
- app: MyApp - version: v1 Deployment - name: MyApp kubectl create ... -
Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector:
- app: MyApp - version: v1 Deployment - name: MyApp kubectl create ... -
Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector:
- app: MyApp - version: v1 Deployment - name: MyApp kubectl create ... -
Google Cloud Platform Rolling Updates ReplicaSet - replicas: 3 -
selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl apply ... -
Google Cloud Platform ReplicaSet - replicas: 3 - selector: -
app: MyApp - version: v1 Rolling Updates ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v2 Deployment - name: MyApp -
Google Cloud Platform ReplicaSet - replicas: 3 - selector: -
app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp -
Google Cloud Platform ReplicaSet - replicas: 2 - selector: -
app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp -
Google Cloud Platform ReplicaSet - replicas: 2 - selector: -
app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp -
Google Cloud Platform ReplicaSet - replicas: 1 - selector: -
app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp -
Google Cloud Platform ReplicaSet - replicas: 1 - selector: -
app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp -
Google Cloud Platform ReplicaSet - replicas: 0 - selector: -
app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp -
Google confidential │ Do not distribute Services A group of
pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Gets a stable virtual IP and port • sometimes called the service portal • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables when backends change Hides complexity - ideal for non-native apps Virtual IP Client -
Google Cloud Platform External Services Services VIPs are only available
inside the cluster Need to receive traffic from “the outside world” Service “type” • NodePort: expose on a port on every node • LoadBalancer: provision a cloud load-balancer DiY load-balancer solutions • socat (for nodePort remapping) • haproxy • nginx Ingress (L7 LB) -
Google Cloud Platform Ingress (L7) Many apps are HTTP/HTTPS Services
are L4 (IP + port) Ingress maps incoming traffic to backend services • by HTTP host headers • by HTTP URL paths HAProxy, NGINX, AWS and GCE implementations in progress Now with SSL! Status: BETA in Kubernetes v1.2 Client URL Map -
Google confidential │ Do not distribute Health Checks Makes sure
containers are healthy. Includes liveness and readiness probes. Liveness Probes • Checks if a container is alive. Hasn’t crashed or deadlocked. Readiness Probes • Checks to make sure a container is ready to serve traffic. -
Google confidential │ Do not distribute Liveness Checks if a
container is alive. Hasn’t crashed or deadlocked. If a container fails a liveness check too many times it will be restarted. -
Google confidential │ Do not distribute Readiness Checks if a
container is ready to serve traffic. Typically, an app will check to make sure it’s dependencies are available. If a container fails a readiness check it many times it’s pod is removed from service endpoints. This makes sure services only use pods that can serve traffic. Virtual IP Client -
Google Cloud Platform Graceful Termination Goal: Give pods time to
clean up • finish in-flight operations • log state • flush to disk • 30 seconds by default Catch SIGTERM, cleanup, exit ASAP Pod status “Terminating” Declarative: ‘DELETE’ appears as an object field in the API -
Google Cloud Platform DaemonSets Problem: how to run a Pod
on every node? • or a subset of nodes Similar to ReplicaSet • principle: do one thing, don’t overload “Which nodes?” is a selector Use familiar tools and patterns Status: BETA in Kubernetes v1.2 Pod -
Google Cloud Platform Jobs Run-to-completion, as opposed to run-forever •
Express parallelism vs. required completions • Workflow: restart on failure • Build/test: don’t restart on failure Aggregates success/failure counts Built for batch and big-data work Status: GA in Kubernetes v1.2 ... -
Google Cloud Platform Namespaces Problem: I have too much stuff!
• name collisions in the API • poor isolation between users • don’t want to expose things like Secrets Solution: Slice up the cluster • create new Namespaces as needed • per-user, per-app, per-department, etc. • part of the API - NOT private machines • most API objects are namespaced • part of the REST URL path • Namespaces are just another API object • One-step cleanup - delete the Namespace • Obvious hook for policy enforcement (e.g. quota) -
Google Cloud Platform Resource Isolation Principles: • Apps must not
be able to affect each other’s performance • if so it is an isolation failure • Repeated runs of the same app should see ~equal behavior • QoS levels drives resource decisions in (soft) real-time • Correct in all cases, optimal in some • reduce unreliable components • SLOs are the lingua franca -
Google Cloud Platform Pros: • Sharing - users don’t worry
about interference (aka the noisy neighbor problem) • Predictable - allows us to offer strong SLAs to apps Cons: • Stranding - arbitrary slices mean some resources get lost • Confusing - how do I know how much I need? • analog: what size VM should I use? • smart auto-scaling is needed! • Expensive - you pay for certainty In reality this is a multi-dimensional bin-packing problem: CPU, memory, disk space, IO bandwidth, network bandwidth, ... Strong isolation -
Google Cloud Platform Requests and Limits Request: • how much
of a resource you are asking to use, with a strong guarantee of availability • CPU (seconds/second) • RAM (bytes) • scheduler will not over-commit requests Limit: • max amount of a resource you can access Repercussions: • Usage > Request: resources might be available • Usage > Limit: throttled or killed -
Google Cloud Platform Quality of Service Defined in terms of
Request and Limit Guaranteed: highest protection • request > 0 && limit == request Burstable: medium protection • request > 0 && limit > request Best Effort: lowest protection • request == 0 What does “protection” mean? • OOM score • CPU scheduling -
Google Cloud Platform ResourceQuota Admission control: apply limits in aggregate
Per-namespace: ensure no user/app/department abuses the cluster Reminiscent of disk quota by design Applies to each type of resource • CPU and memory for now Disallows pods without resources -
Google Cloud Platform LimitRange Admission control: limit the limits •
min and max • ratio of limit/request Default values for unspecified limits Per-namespace Together with ResourceQuota gives cluster admins powerful tools -
Kubernetes 1.3 • Supports up to 60,000 containers on 2000
nodes • PetSet (alpha) • Federation • Cross cluster service discovery • New in 1.2: • Deployments • ConfigMaps -
Google Cloud Platform PetSets (working name) Goal: enable clustered software
on Kubernetes • mysql, redis, zookeeper, ... Clustered apps need “identity” and sequencing guarantees • stable hostname, available in DNS • an ordinal index • stable storage: linked to the ordinal & hostname • discovery of peers for quorum • startup/teardown ordering Status: ALPHA in Kubernetes v1.3 -
Google Container Engine - Inspired by a decade within Google
- Reimagines Cluster computing - Designed for a multi-cloud world Photo by Connie Zhou -
Google Container Engine New service for cluster-based compute • Provisioned
cluster in seconds. Fully configured. • Fine-grained control over cluster. • Designed for multi-cloud. Runs Kubernetes. Releases • Now GA!! • No additional cost for up to 5 nodes Resources • Google Container Engine: http://cloud.google.com/container-engine • Kubernetes: http://kubernetes.io