Google confidential │ Do not distribute Google confidential │ Do not distribute Kubernetes and Container Management Container Camp 2015 Tim Hockin Senior Staff Software Engineer @thockin

Google confidential │ Do not distribute Google has been developing and using containers to manage our applications for over 10 years. Images by Connie Zhou

Google confidential │ Do not distribute Everything at Google runs in containers: • Gmail, Web Search, Maps, ... • MapReduce, batch, ... • GFS, Colossus, ... • Even GCE itself: VMs run in containers

Google confidential │ Do not distribute Everything at Google runs in containers: • Gmail, Web Search, Maps, ... • MapReduce, batch, ... • GFS, Colossus, ... • Even GCE itself: VMs run in containers We launch over 2 billion containers per week.

Google confidential │ Do not distribute But it’s so different! • Deployment • Management, monitoring • Isolation (very complicated!) • Updates • Discovery • Scaling, replication, sets A fundamentally different way of managing applications requires different tooling and abstractions Images by Connie Zhou

Google confidential │ Do not distribute Enter Kubernetes Greek for “Helmsman”; also the root of the word “Governor” and “cybernetic” • Container orchestrator • Runs and manages containers • Supports multiple cloud and bare-metal environments • Inspired and informed by Google’s experiences and internal systems • 100% Open source, written in Go Manage applications, not machines

Google confidential │ Do not distribute Pets vs. Cattle

Google confidential │ Do not distribute A 50000 foot view

Google confidential │ Do not distribute users master nodes A 50000 foot view CLI API UI apiserver kubelet kubelet kubelet scheduler

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler Run X Replicas = 2 Memory = 4Gi CPU = 2.5

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler SUCCESS UID=8675309

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler Which nodes for X ?

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler Run X Run X

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler Registry pull X pull X

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler Status X Status X X X

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler X X GET X

Google confidential │ Do not distribute A 50000 foot view apiserver kubelet kubelet kubelet scheduler X X Status X

Google confidential │ Do not distribute Container clusters: A story in two parts

Google confidential │ Do not distribute All you really care about Run X Master Container Cluster X X Status X

Google confidential │ Do not distribute Container clusters: A story in two parts 1. Setting up a cluster • Choose a cloud: GCE, AWS, Azure, Rackspace, on-premises, ... • Choose a node OS: CoreOS, Atomic, RHEL, Debian, CentOS, Ubuntu, ... • Provision machines: Boot VMs, install and run kube components, ... • Configure networking: IP ranges for Pods, Services, SDN, ... • Start cluster services: DNS, logging, monitoring, ... • Manage nodes: kernel upgrades, OS updates, hardware failures... Not the easy or fun part, but unavoidable This is where things like Google Container Engine (GKE) really help

Google confidential │ Do not distribute 2. Using a cluster • Run Pods & Containers • Replication controllers • Services • Volumes • Secrets This is the fun part! A distinct set of problems from cluster setup and management Don’t make developers deal with cluster administration! Accelerate development by focusing on the applications, not the cluster Container clusters: A story in two parts

Google confidential │ Do not distribute Networking

Google confidential │ Do not distribute 10.1.1.0/24 172.16.1.1 172.16.1.2 Docker networking 10.1.2.0/24 172.16.1.1 10.1.3.0/24 172.16.1.1

Google confidential │ Do not distribute 10.1.1.0/24 172.16.1.1 172.16.1.2 Docker networking 10.1.2.0/24 172.16.1.1 10.1.3.0/24 172.16.1.1 NAT NAT NAT NAT NAT

Google confidential │ Do not distribute Kubernetes networking Pod IPs are routable • docker default is private IP Pods can reach each other without NAT • even across nodes No brokering of port numbers • too complex, why bother? This is a fundamental requirement • several SDN solutions exist

Google confidential │ Do not distribute 10.1.1.0/24 10.1.1.93 10.1.1.113 10.1.2.0/24 10.1.2.118 10.1.3.0/24 10.1.3.129 Kubernetes networking

Google confidential │ Do not distribute Concept: Pods Small group of containers & volumes Tightly coupled The atom of scheduling & placement in Kubernetes Shared namespace • share IP address & localhost • share IPC Mortal • can die, cannot be reborn Example: data puller & web server Pod File Puller Web Server Volume Consumers Content Manager

Google confidential │ Do not distribute Concept: Services A group of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” for now Gets a stable virtual IP and port • called the service portal • also a DNS name VIP is captured by kube-proxy • watches the service constituency • updates when backends change Hides complexity - ideal for non-native apps Portal (VIP) Client

Google confidential │ Do not distribute WATCH Services, Endpoints Services kube-proxy apiserver

Google confidential │ Do not distribute Services kube-proxy Pod - Name = “pod1” - Labels = {“App”: “Nifty”} - Port = 9376 apiserver POST pods WATCH Services, Endpoints

Google confidential │ Do not distribute Services kube-proxy apiserver pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 run pods Pod - Name = “pod1” - Labels = {“App”: “Nifty”} - Port = 9376 WATCH Services, Endpoints

Google confidential │ Do not distribute POST service pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 apiserver WATCH Services, Endpoints

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints new service!

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X (random) Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints new endpoints!

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables Client redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 connect to 10.9.8.7:80

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables Client redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 connect to 10.9.8.7:80

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux iptables Client Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 connect to localhost:X

Google confidential │ Do not distribute pod1 10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables Client Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 proxy for client

Google confidential │ Do not distribute Concept: Volumes Very similar to Docker’s concept Pod scoped storage Share the pod’s lifetime & fate Support many types of volume plugins • Empty directory • Host path • Git repository • GCE Persistent Disk • AWS Elastic Block Store • iSCSI Pod Container Container Git GitHub Host Host’s FS GCE GCE PD Empty • NFS • GlusterFS • Ceph File and RBD • Cinder • ...

Google confidential │ Do not distribute User owned Admin owned New: Persistent Volumes A higher-level abstraction - insulation from any one cloud environment Admin provisions them, users claim them Independent lifetime and fate Can be handed-off between pods and lives until user is done with it Dynamically “scheduled” and managed, like nodes and pods Pod ClaimRef PVClaim PersistentVolume GCE PD AWS ELB NFS iSCSI

Google confidential │ Do not distribute Docker, Rocket, LXC, Oh my! Currently built on Docker Work is in progress to abstract that (a bit) into a Runtime abstraction Interest in Rocket and LXC support Rocket support is in flight (we like plugins) Dynamically “scheduled” and managed, like nodes and pods Runtime Docker Rocket LXC? Kubelet

Google confidential │ Do not distribute What else is in new? • Network plugins • Secrets • Graceful termination • Quota • More volumes • Downward API • More platforms • Performance • Scalability • High availability masters • Scheduling • Cluster federation • Multi-cloud • Easier setup

Google confidential │ Do not distribute Kubernetes status & plans Open sourced in June, 2014 • won the 2014 BlackDuck “rookie of the year” award Google Container Engine (GKE) • hosted Kubernetes - don’t think about cluster setup Red Hat: OpenShift 3 • open PaaS on Kubernetes CoreOS: Tectonic • ready-to-run Kubernetes - don’t think about cluster setup Mirantis: Murano • Kubernetes and OpenStack Driving towards a 1.0 release in O(months) Roadmap: • https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/roadmap.md

Google confidential │ Do not distribute The Goal: Shake things up Containers are a new way of working Requires new concepts and new tools Google has a lot of experience... ...but we are listening to the users Workload portability is important!

Google confidential │ Do not distribute Kubernetes is Open - open community - open design - open to ideas - open source http://kubernetes.io https://github.com/GoogleCloudPlatform/kubernetes irc.freenode.net #google-containers @kubernetesio