Kubernetes: Applying Lessons from a Decade of Containers @ iTHome Cloud & Datacenter Expo

Transcript

1. Ian Lewis Developer Advocate, Google Cloud Platform Kubernetes Applying Lessons from a Decade of Containers
2. Confidential & Proprietary Google Cloud Platform 2 Ian Lewis Developer Advocate - Google Cloud Platform Tokyo, Japan +Ian Lewis @IanMLewis
3. Agenda • Current Practice • Containers • Kubernetes • Wrap Up
4. Confidential & Proprietary Google Cloud Platform 4 Current Practice
5. Confidential & Proprietary Google Cloud Platform 5 Database Batch processing Cache Webservers Webservers Webservers Webservers
6. 6 No isolation No namespacing Common libs Highly coupled apps and OS Shared Machines 6 kernel libs app app app app
7. 7 Some isolation Inefficient Still highly coupled to the guest OS Hard to manage Virtual Machines 7 app libs kernel libs app app kernel app libs libs kernel kernel
8. Confidential & Proprietary Google Cloud Platform 8 Database Batch processing Cache Webservers Webservers Webservers Webservers
9. Confidential & Proprietary Google Cloud Platform 9 Database Batch processing Cache Webservers Webservers Webservers Webservers Chef?
10. Confidential & Proprietary Google Cloud Platform 10 Database Batch processing Cache Webservers Webservers Webservers Webservers Chef? Puppet?
11. Confidential & Proprietary Google Cloud Platform 11 Database Batch processing Cache Webservers Webservers Webservers Webservers Chef? Ansible? Puppet?
12. Confidential & Proprietary Google Cloud Platform 12 Database Batch processing Cache Webservers Webservers Webservers Webservers Chef? Ansible? Puppet? Versioning?
13. Confidential & Proprietary Google Cloud Platform 13 Database Batch processing Cache Webservers Webservers Webservers Webservers Chef? Ansible? Deployment? Puppet? Versioning?
14. Confidential & Proprietary Google Cloud Platform 14 Containers
15. For the last 15 years Google has been building the world’s fastest, most powerful infrastructure.
17. Monitoring Mobile Development Compute Network Big Data Storage
18. Cloud Technology Innovations 2012 2013 MapReduce Spanner/F1 2003 2006 2007 2010 2011 GFS Omega Colossus Cloud Storage Dremel BigQuery Big Table Cloud Datastore Paxos impl. 2004 Cloud Bigtable
19. Copyright 2015 Google Inc Google has been running all our services in Containers for over 10 years. We start over 2 billion containers every week. Images by Connie Zhou
20. http://research.google.com/pubs/pub43438.html
21. Image by Connie Zhou
22. job hello_world = { runtime = { cell = 'ic' } // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks } 10000 Developer View
23. Developer View
24. web browsers BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Binary Developer View What just happened?
25. Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Image by Connie Zhou Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!
26. Container Image Dependencies Application Code Containers encapsulate application code and all dependencies. Applications can be depend less on the infrastructure where it runs. • In traditional IT environments, applications needed specific infrastructure. Dependencies needed to be installed beforehand. • Containers incorporate applications and their dependencies so deployment to development, test, and production can be made easier. • Don’t need to be dependent on on-premise, private or public cloud environments. What are Containers?
27. Fast Simple and Fast compared to VMs. Can be started in just a few milliseconds. Portable Can be run in a many environments. Efficiency Low overhead. Resources use by containers can be limited. Why Containers?
28. 28 Containers 28 libs app kernel libs app libs app libs app
29. Confidential & Proprietary Google Cloud Platform 29 Compute Continuum More agility More Flexibility IaaS Containers PaaS • VMs • Install anything • Manage yourself • Docker/rkt etc. • Install nearly anything • Easier deployment • App Engine/Heroku etc. • Limited libraries & APIs • Simple deployment
30. Google confidential │ Do not distribute But it’s all so different! • Deployment • Management, monitoring • Isolation (very complicated!) • Updates • Discovery • Scaling, replication, sets A fundamentally different way of managing applications requires different tooling and abstractions Images by Connie Zhou
32. Copyright 2015 Google Inc Container Management Node Node Cluster Node ??? • How to deploy to multiple nodes? • How to deal with node failures? • How to deal with container failures? • How do you update your applications? • How can your containers discover and communicate with each other?
33. Confidential & Proprietary Google Cloud Platform 33 Kubernetes
34. Enter Kubernetes Greek for “Helmsman”; also the root of the word “Governor” • Container orchestrator • Runs containers • Supports multiple cloud and bare- metal environments • Inspired and informed by Google’s experiences and internal systems • Open source, written in Go Manage applications, not machines
35. CNCF(Cloud Native Computing Foundation)
36. Confidential & Proprietary Google Cloud Platform 36 Community • Current stable version: 1.2 • 780+ contributors • over 14,000 github stars Project Partners
37. Google Cloud Platform UI CLI API users master nodes etcd scheduler controllers master The 10000 Foot View kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet
38. Google Cloud Platform The 10000 Foot View etcd scheduler controller-manager apiserver kubelet docker kube-proxy iptables 1. User creates Deployment 2. API server saves info to etcd 3. CM finds Deployment and creates ReplicaSet, which creates Pods (unscheduled) 4. Scheduler schedules pods. 5. Kubelet sees pod scheduled to it and tells docker to run the container. 6. Docker pulls and runs the container. 1 2 6 3 5 4 Docker Hub / GCR
39. Google Cloud Platform IPC Network PID Hostname Mount nginx IPC Network PID Hostname Mount nginx IPC Network PID Hostname Mount nginx Docker Containers
40. Google Cloud Platform IPC Network PID Hostname Mounts nginx IPC Network PID Hostname Mount git pull IPC Network PID Hostname Mount nginx Docker Containers
41. Google Cloud Platform IPC Network PID Hostname IPC Network PID Hostname Mount git pull IPC Network PID Hostname Mount nginx VOLUME nginx Pods
42. Google Cloud Platform IPC Network docker … --net=container: id --ipc=container:id Hostname cgroup Web Server Pod cgroup File Puller localhost Pods
43. Google confidential │ Do not distribute Pods Small group of containers & volumes Tightly coupled The atom of scheduling & placement in Kubernetes Shared namespace • share IP address & localhost • share IPC Mortal • can die, cannot be reborn Example: data puller & web server Consumers Content Manager File Puller Web Server Volume Pod
44. Google confidential │ Do not distribute Volumes Very similar to Docker’s concept Pod scoped storage Share the pod’s lifetime & fate Support many types of volume plugins • Empty dir (and tmpfs) • Host path • Git repository • GCE Persistent Disk • AWS Elastic Block Store • iSCSI • NFS • GlusterFS • Ceph File and RBD • Cinder • Secret • ...
45. Google confidential │ Do not distribute Arbitrary metadata Attached to any API object Generally represent identity Queryable by selectors • think SQL ‘select ... where ...’ The only grouping mechanism • pods under a ReplicationController • pods in a Service • capabilities of a node (constraints) Labels
46. Google confidential │ Do not distribute App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE Selectors
47. Google confidential │ Do not distribute App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp Selectors
48. Google confidential │ Do not distribute App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = FE Selectors
49. Google confidential │ Do not distribute App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = BE Selectors
50. Google confidential │ Do not distribute Selectors App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = prod
51. Google confidential │ Do not distribute App: MyApp Phase: prod Role: FE App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = test Selectors
52. FE FE FE FE replicas: 4 template: ... labels: role: frontend stage: production Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Kubernetes - Master/Scheduler ReplicaSets
53. FE replicas: 1 template: ... labels: role: frontend stage: production Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Kubernetes - Master/Scheduler ReplicaSets
54. FE FE FE replicas: 3 template: ... labels: role: frontend stage: production Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Kubernetes - Master/Scheduler ReplicaSets
55. FE FE FE replicas: 3 template: ... labels: role: frontend stage: production Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Kubernetes - Master/Scheduler ReplicaSets
56. FE FE FE replicas: 3 template: ... labels: role: frontend stage: production Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Machine Host Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Container Agent Kubernetes - Master/Scheduler ReplicaSets FE
57. Google Cloud Platform observe diff act ReplicaSets
58. Google confidential │ Do not distribute ReplicaSets An example of control loops Runs out-of-process wrt API server Have 1 job: ensure N copies of a pod • if too few, start new ones • if too many, kill some • grouped by a selector Cleanly layered on top of the core • all access is by public APIs Replicated pods are fungible • No implied order or identity ReplicaSet - name = “my-rs” - selector = {“App”: “MyApp”} - podTemplate = { ... } - replicas = 4 API Server How many? 3 Start 1 more OK How many? 4
59. Google Cloud Platform Rolling Updates ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl apply ...
60. Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 Rolling Updates ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v2 Deployment - name: MyApp
61. Google Cloud Platform ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
62. Google Cloud Platform ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
63. Google Cloud Platform ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
64. Google Cloud Platform ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
65. Google Cloud Platform ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
66. Google Cloud Platform ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
67. Google confidential │ Do not distribute Services A group of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Gets a stable virtual IP and port • sometimes called the service portal • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables when backends change Hides complexity - ideal for non-native apps Virtual IP Client
68. Google Cloud Platform Efficiency
69. Google Cloud Platform Efficiency
70. Confidential & Proprietary Google Cloud Platform 70 Wrap Up
71. Google Container Engine - Inspired by a decade within Google - Reimagines Cluster computing - Designed for a multi-cloud world Photo by Connie Zhou
72. Google Container Engine New service for cluster-based compute • Provisioned cluster in seconds. Fully configured. • Fine-grained control over cluster. • Designed for multi-cloud. Runs Kubernetes. Releases • Now GA!! • No additional cost for up to 5 nodes Resources • Google Container Engine: http://cloud.google.com/container-engine • Kubernetes: http://kubernetes.io
73. 謝謝 Thank You