Kubernetes at Scale @ DevOps Meetup Singapore

Slide 1

Slide 1 text

Kubernetes & Google Container Engine Overview DevOps Meetup Singapore

Slide 2

Slide 2 text

Confidential & Proprietary Google Cloud Platform 2 Ian Lewis Developer Advocate - Google Cloud Platform Tokyo, Japan @IanMLewis

Slide 3

Slide 3 text

For the last 15 years Google has been building the world’s fastest, most powerful infrastructure.

Slide 4

Slide 4 text

Google’s World Spanning Backbone Network

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Building what’s next 6 33 Countries 70 Edge Locations The most of any Cloud Provider Google-Grade Networking

Slide 7

Slide 7 text

Monitoring Mobile Development Compute Network Big Data Storage

Slide 8

Slide 8 text

2012 2015 MapReduce Spanner 2003 2006 2010 2011 GFS Borg Colossus Dremel Bigtable Chubby 2004

Slide 9

Slide 9 text

Slide 10

Slide 10 text

http://research.google.com/pubs/pub43438.html

Slide 11

Slide 11 text

Image by Connie Zhou

Slide 12

Slide 12 text

job hello_world = { runtime = { cell = 'ic' } // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks } 10000 Developer View

Slide 13

Slide 13 text

Developer View

Slide 14

Slide 14 text

web browsers BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Binary Developer View What just happened?

Slide 15

Slide 15 text

Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Image by Connie Zhou Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!

Slide 16

Slide 16 text

Container Image Dependencies Application Code Containers encapsulate application code and all dependencies. Applications can be depend less on the infrastructure where it runs. • In traditional IT environments, applications needed specific infrastructure. Dependencies needed to be installed beforehand. • Containers incorporate applications and their dependencies so deployment to development, test, and production can be made easier. • Don’t need to be dependent on on-premise, private or public cloud environments. What are Containers?

Slide 17

Slide 17 text

Fast Simple and Fast compared to VMs. Can be started in just a few milliseconds. Portable Can be run in a many environments. Efficiency Low overhead. Resources use by containers can be limited. Why Containers?

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Copyright 2015 Google Inc Container Management Node Node Cluster Node ??? ● How to deploy to multiple nodes? ● How to deal with node failures? ● How to deal with container failures? ● How do you update your applications?

Slide 20

Slide 20 text

Kubernetes κυβερνήτης: Greek for “pilot” or “helmsman of a ship” the open source cluster manager from Google

Slide 21

Slide 21 text

CNCF(Cloud Native Computing Foundation)

Slide 22

Slide 22 text

Google Cloud Platform Goal: Avoid vendor lock-in Runs in many environments, including “bare metal” and “your laptop” The API and the implementation are 100% open The whole system is modular and replaceable Workload portability

Slide 23

Slide 23 text

Google Cloud Platform Goal: Write once, run anywhere* Don’t force apps to know about concepts that are cloud-provider-specific Examples of this: ● Network model ● Ingress ● Service load-balancers ● PersistentVolumes * approximately Workload portability

Slide 24

Slide 24 text

Google Cloud Platform Goal: Avoid coupling Don’t force apps to know about concepts that are Kubernetes-specific Examples of this: ● Namespaces ● Services / DNS ● Downward API ● Secrets / ConfigMaps Workload portability

Slide 25

Slide 25 text

Google Cloud Platform Result: Portability Build your apps on-prem, lift-and-shift into cloud when you are ready Don’t get stuck with a platform that doesn’t work for you Put your app on wheels and move it whenever and wherever you need Workload portability

Slide 26

Slide 26 text

Container Engine Google Cloud Platform

Slide 27

Slide 27 text

Google Cloud Platform Small group of containers & volumes Tightly coupled The atom of scheduling & placement Shared namespace • share IP address & localhost • share IPC, etc. Managed lifecycle • bound to a node, restart in place • can die, cannot be reborn with same ID Example: data puller & web server Consumers Content Manager File Puller Web Server Volume Pod Pods

Slide 28

Slide 28 text

Google Cloud Platform Docker Containers IPC Network PID Hostname Mount nginx IPC Network PID Hostname Mount nginx IPC Network PID Hostname Mount nginx

Slide 29

Slide 29 text

Google Cloud Platform IPC Network PID Hostname Mounts nginx IPC Network PID Hostname Mount git pull IPC Network PID Hostname Mount nginx Docker Containers

Slide 30

Slide 30 text

Google Cloud Platform IPC Network PID Hostname Mounts nginx IPC Network PID Hostname Mount git pull IPC Network PID Hostname Mount nginx Docker Containers VOLUME VOLUME Host Volume

Slide 31

Slide 31 text

Google Cloud Platform Host NIC Network IPC Network PID Hostname Mounts nginx IPC Network PID Hostname Mount git pull IPC Network PID Hostname Mount nginx Docker Containers NAT NAT

Slide 32

Slide 32 text

Google Cloud Platform 172.16.1.1 172.16.1.2 172.16.1.1 172.16.1.1 NAT NAT NAT NAT NAT Docker networking

Slide 33

Slide 33 text

Google Cloud Platform A: 172.16.1.1 3306 B: 172.16.1.2 80 9376 11878 SNAT SNAT C: 172.16.1.1 8000 Port mapping

Slide 34

Slide 34 text

Google Cloud Platform Pods & Docker? confd nginx HUP W RITE READ etcd CHANGE nginx.conf app app app IP Address LB

Slide 35

Slide 35 text

Google Cloud Platform Container Container Pods & Docker? confd nginx HUP W RITE READ etcd CHANGE ? ? ? ?

Slide 36

Slide 36 text

Google Cloud Platform Pods & Docker? Container nginx confd foreman

Slide 37

Slide 37 text

Google Cloud Platform Container foreman Pods & Docker? nginx confd

Slide 38

Slide 38 text

Google Cloud Platform Container foreman Pods & Docker? Everything’s A-OK!! nginx confd Crash-Restart Loop

Slide 39

Slide 39 text

Google Cloud Platform IPC Network Pods docker … --net=container:id --ipc=container:id Hostname cgroup Web Server Pod cgroup File Puller localhost

Slide 40

Slide 40 text

Google Cloud Platform Pods (TODO) docker … --net=container:id --ipc=container:id --pid=container:id https://github.com/docker /docker/issues/10163 IPC Network PID Hostname cgroup Web Server cgroup File Puller localhost

Slide 41

Slide 41 text

Google Cloud Platform IPs are cluster-scoped • vs docker default private IP Pods can reach each other directly • even across nodes No brokering of port numbers • too complex, why bother? This is a fundamental requirement • can be L3 routed • can be underlayed (cloud) • can be overlayed (SDN) Kubernetes networking

Slide 42

Slide 42 text

Google Cloud Platform 10.1.1.0/24 10.1.1.1 10.1.1.2 10.1.2.0/24 10.1.2.1 10.1.3.0/24 10.1.3.1 Kubernetes networking

Slide 43

Slide 43 text

Google Cloud Platform Goal: manage app configuration • ...without making overly-brittle container images 12-factor says config comes from the environment • Kubernetes is the environment Manage config via the Kubernetes API Inject config as a virtual volume into your Pods • late-binding, live-updated (atomic) • also available as env vars Status: GA in Kubernetes v1.2 node API Pod Config Map ConfigMaps

Slide 44

Slide 44 text

Google Cloud Platform Goal: grant a pod access to a secured something • don’t put secrets in the container image! 12-factor says config comes from the environment • Kubernetes is the environment Manage secrets via the Kubernetes API Inject secrets as virtual volumes into your Pods • late-binding, tmpfs - never touches disk • also available as env vars node API Pod Secret Secrets

Slide 45

Slide 45 text

Google Cloud Platform A higher-level storage abstraction • insulation from any one cloud environment Admin provisions them, users claim them • NEW: auto-provisioning (alpha in v1.2) Independent lifetime from consumers • lives until user is done with it • can be handed-off between pods Dynamically “scheduled” and managed, like nodes and pods Claim PersistentVolumes

Slide 46

Slide 46 text

Google Cloud Platform Cluster Admin PersistentVolumes

Slide 47

Slide 47 text

Google Cloud Platform Provision Cluster Admin PersistentVolumes PersistentVolumes

Slide 48

Slide 48 text

Google Cloud Platform User Cluster Admin PersistentVolumes PersistentVolumes

Slide 49

Slide 49 text

Google Cloud Platform User PVClaim Create Cluster Admin PersistentVolumes PersistentVolumes

Slide 50

Slide 50 text

Google Cloud Platform User PVClaim Binder Cluster Admin PersistentVolumes PersistentVolumes

Slide 51

Slide 51 text

Google Cloud Platform User PVClaim Pod Create Cluster Admin PersistentVolumes PersistentVolumes

Slide 52

Slide 52 text

Google Cloud Platform User PVClaim Pod Cluster Admin PersistentVolumes * PersistentVolumes

Slide 53

Slide 53 text

Google Cloud Platform User PVClaim Pod Delete * Cluster Admin PersistentVolumes * PersistentVolumes

Slide 54

Slide 54 text

Google Cloud Platform User PVClaim Cluster Admin PersistentVolumes * PersistentVolumes

Slide 55

Slide 55 text

Google Cloud Platform User PVClaim Pod Create Cluster Admin PersistentVolumes * PersistentVolumes

Slide 56

Slide 56 text

Google Cloud Platform User PVClaim Pod Cluster Admin PersistentVolumes * PersistentVolumes

Slide 57

Slide 57 text

Google Cloud Platform User PVClaim Pod Delete Cluster Admin PersistentVolumes * PersistentVolumes

Slide 58

Slide 58 text

Google Cloud Platform User PVClaim Delete Cluster Admin PersistentVolumes * PersistentVolumes

Slide 59

Slide 59 text

Google Cloud Platform User Recycler Cluster Admin PersistentVolumes PersistentVolumes

Slide 60

Slide 60 text

Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...

Slide 61

Slide 61 text

Google Cloud Platform Deployments ReplicaSet - replicas: 4 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...

Slide 62

Slide 62 text

Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...

Slide 63

Slide 63 text

Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...

Slide 64

Slide 64 text

Google Cloud Platform Rolling Updates ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl apply ...

Slide 65

Slide 65 text

Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Rolling Updates ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v2 Deployment - name: MyApp

Slide 66

Slide 66 text

Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp

Slide 67

Slide 67 text

Google Cloud Platform ReplicaSet - replicas: 2 - selector: - app: nginx - ver: 1.10 ReplicaSet - replicas: 1 - selector: - app: nginx - ver: 1.11 Deployment - app: nginx Rolling Updates

Slide 68

Slide 68 text

Google Cloud Platform ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp

Slide 69

Slide 69 text

Google Cloud Platform ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp

Slide 70

Slide 70 text

Google Cloud Platform ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp

Slide 71

Slide 71 text

Google Cloud Platform ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp

Slide 72

Slide 72 text

Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp

Slide 73

Slide 73 text

Google confidential │ Do not distribute Services A group of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Gets a stable virtual IP and port • sometimes called the service portal • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables when backends change Hides complexity - ideal for non-native apps Virtual IP Client

Slide 74

Slide 74 text

Google Cloud Platform Arbitrary metadata Attached to any API object Generally represent identity Queryable by selectors • think SQL ‘select ... where ...’ The only grouping mechanism • pods under a ReplicationController • pods in a Service • capabilities of a node (constraints) Labels

Slide 75

Slide 75 text

Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE Selectors

Slide 76

Slide 76 text

Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp Selectors

Slide 77

Slide 77 text

Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = FE Selectors

Slide 78

Slide 78 text

Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = BE Selectors

Slide 79

Slide 79 text

Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = prod Selectors

Slide 80

Slide 80 text

Google Cloud Platform App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = test Selectors

Slide 81

Slide 81 text

Google Cloud Platform Run-to-completion, as opposed to run-forever • Express parallelism vs. required completions • Workflow: restart on failure • Build/test: don’t restart on failure Aggregates success/failure counts Built for batch and big-data work Status: GA in Kubernetes v1.2 ... Jobs

Slide 82

Slide 82 text

Google Cloud Platform Problem: I have too much stuff! • name collisions in the API • poor isolation between users • don’t want to expose things like Secrets Solution: Slice up the cluster • create new Namespaces as needed • per-user, per-app, per-department, etc. • part of the API - NOT private machines • most API objects are namespaced • part of the REST URL path • Namespaces are just another API object • One-step cleanup - delete the Namespace • Obvious hook for policy enforcement (e.g. quota) Namespaces