Understanding Kubernetes Through Design Principles

UIUC's "Reflections | Projections" conference 2018
https://reflectionsprojections.org/

Transcript

1. Understanding Kubernetes Through Design Principles September 20, 2018 Saad Ali Senior Software Engineer, Google github.com/saad-ali twitter.com/the_saad_ali
2. Introductions Saad Ali Google Fun fact:10 office moves in 4.5 years Andrew McHugh Google Fun fact: Has never had antibiotics Mihir Pandya UIUC Fun fact: Puts milk first when eating cereal
3. Agenda • Google: We’re hiring! • Very quickly: What is Kubernetes? • Illustrate principles of Kubernetes design, by showing how Kubernetes works. • Kubernetes Workshop
4. Sign-in: bit.ly/2MAvGgs Administrivia
5. Google: We’re hiring!
6. 2018 | Confidential and Proprietary We’re hiring! Here’s what we’re looking for: Software Engineer (BS/MS) 8/6/18 - late spring g.co/jobs/swegrad Software Engineer, Intern (BS/MS) 9/17/18 - 12/14/18 7/9/18 - 9/28/18 (winter internship) g.co/jobs/sweintern (BS) g.co/jobs/msintern (MS) Software Engineer, PhD 8/6/18 - late spring g.co/jobs/swephd Software Engineer, PhD, Intern 9/17/18 - 2/8/18 g.co/jobs/phdintern Site Reliability Engineer (+Intern) Deadline: same as SWE (+Intern) TBD if separate application link Engineering Practicum Program 9/17/18 - 10/31/18 g.co/jobs/engpracticum Engineering Residency mid fall - early spring g.co/jobs/engresident Associate Product Manager early aug - end of oct. g.co/jobs/apmgrad SETI (+Intern) Deadline: same as SWE (+Intern) TBD if separate application link APM, Intern 9/17/18 - 10/31/18 g.co/jobs/apmintern UX Eng, Design, Writer, Research, (+Intern) early jan. - late jan. google.com/design/jobs Technical Program Manager Internship: dec. - jan. Data Scientist jan. 2019 g.co/jobs/datascientist Cloud Tech Residency 8/23/18 - early dec. g.co/jobs/ctr Information Technology Residency Program Internship: 9/24/18 - 9/29/18, 1/7/19 - 1/18/19 Full-Time: 9/5/18 - until filled Hardware/Mechanical Engineer Internship: jan. 2019 Full-Time: year round Technical Solutions Consultant Internship: jan. 2019 Applications Engineer Internship and Full-Time: jan. 2019 Technical Writer Internship: late jan. AI Residency (BS-PhD) Full-Time: oct. - jan. SWE in Research (BS, MS, PhD) Internship: year round Check out more roles google.com/students
7. Cloud Technical Residency g.co/jobs/ctr INSERT PHOTO ❖ We’re on a mission to completely change the world of enterprise computing - you’ll have a front-row seat for the action! ❖ Full Time - rotational program within Cloud, various start dates ❖ Training + three project rotations ❖ Currently based in Austin, TX
8. 2018 | Confidential and Proprietary Resources Build your future with Google Learn more about our programs, events, scholarships and more at g.co/buildyourfuture g.co/techdevguide Google’s Guide to Technical Development YouTube Live (GoogleStudents page) Check out recorded workshops and info sessions Interview Prep Texts Programming Interviews Exposed Landing Your Next Job, 3rd Edition Elements of Programming Interviews Cracking the Coding Interview, 6th Edition The Algorithm Design Manual, Steven S Skiena Practice questions Check out TopCoder for tutorials & questions. Sign up for an account on leetcode.com or pramp.com and have your practice answers reviewed by others. Awesome interview prep videos Visit our Life at Google YouTube page where we've shared some fantastic videos to help you Prepare for a Google Engineering Interview. Worth watching in full!
9. What’s in it for me? • Goal: A deeper understanding of Kubernetes • Why you should care? ◦ Kubernetes quickly becoming standard ◦ Important in industry and research • Important tool for learning ◦ Understand the problem ◦ The “why” not just the “what”
10. What is Kubernetes? Problem: • Given a set of machines (VMs, physical machines, etc.) how do you deploy workloads (web applications, databases, batch machine learning jobs, etc,) to them? Node A Node B Node C Node D
11. Challenges: • Common libraries • No isolation • Highly coupled apps and OS What is Kubernetes? Old approach: • Multiple apps per machine Node A Node C Node D app kernel libs app app app Node A app kernel libs app app app kernel libs kernel libs
12. Better approach: • Virtual Machines Challenges: • Some isolation • Expensive, inefficient, slow • Still highly coupled to the OS • Hard to manage What is Kubernetes? Node A Node C Node D libs app app kernel libs app app kernel Node B libs app app kernel libs app app kernel libs kernel libs kernel
13. Even better approach: • Containers! What is Kubernetes? Benefits: • Performance • Repeatability • Isolation • Quality of service • Accounting • Visibility • Portability A fundamentally different way of managing applications Node A Node B libs app kernel libs app libs app libs app libs app kernel libs app libs app libs app
14. Challenges • How will you deploy? ◦ Deploy my containers ◦ Monitor my containerized app ◦ Make my containerized app robust ◦ Scale my containerized app • But who will manage your containers? ◦ You? Scripts? A system you write? Containers are great, but... What is Kubernetes? Node A Node B libs app kernel libs app libs app libs app libs app kernel libs app libs app libs app
15. What is Kubernetes? • Kubernetes is a system for monitoring & deploying containerized workloads to nodes in a cluster (a container orchestrator). • Greek for “Helmsman”; also the root of the word “Governor” • Supports multiple container runtimes (including Docker) • Supports multiple cloud and bare-metal environments • Inspired and informed by Google’s experiences • Open source, written in Go • Manage applications, not machines
16. What is Kubernetes? Application (Dev) Cluster Kernel/OS (System) Hardware
17. Problem #1: How to deploy a workload? Obvious solution! Master: API Server Node A Node B Start containers x
18. Problem #1: How to deploy a workload? Obvious solution! Problems with this approach? Master: API Server Node A Node B Container A Start containers x
19. Problem #1: How to deploy a workload? Obvious solution! Problems with this approach? What if: • Container crashes and dies? • What if node crashes and dies? • What if node B had a momentary blip? Master: API Server Node A Node B Container A
20. Obvious solution! Problems with this approach? What if: • Container crashes and dies? • What if node crashes and dies? • What if node B had a momentary blip? Problem #1: How to deploy a workload? Master: API Server Node A Node B Start containers x
21. Principle #1 Kubernetes APIs are declarative rather then imperative.
22. • Principle: Kubernetes APIs are declarative rather then imperative. • You: define desired state • System: works to drive towards that state How to deploy a workload? Kubernetes way!
23. • Principle: Kubernetes APIs are declarative rather then imperative. • You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B kubectl create -f podA.yaml
24. How to deploy a workload? Kubernetes way! Sample pod definition file apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: internal.mycorp.com:5000/mycontainer:1.7.9
25. • Principle: Kubernetes APIs are declarative rather then imperative. • You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition
26. • Principle: Kubernetes APIs are declarative rather then imperative. • You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
27. Automatic recovery! Why declarative over imperative?
28. • Principle: Kubernetes APIs are declarative rather then imperative. • You: define create API objects that is persisted on kube API server until deletion • System: all components work in parallel to drive do their part to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
29. • Principle: Kubernetes APIs are declarative rather then imperative. • You: define create API objects that is persisted on kube API server until deletion • System: all components work in parallel to drive do their part to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
30. • Principle: Kubernetes APIs are declarative rather then imperative. • You: define create API objects that is persisted on kube API server until deletion • System: all components work in parallel to drive do their part to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
31. Problem #2: how do nodes figure out what to do? Master: API Server Node A Node B kubectl create pod A Pod A definition Obvious solution!
32. Obvious solution! Problems with this approach? Master has to • Keep track of state of every other component! • “Catch up” any failed components that missed calls. Problem #2: how do nodes figure out what to do? Master: API Server Node A Node B Pod A Start pod A Pod A definition
33. Obvious solution! Problems with this approach? Master becomes: • Complex • Difficult to extend Problem #2: how do nodes figure out what to do? Master: API Server Node A Node B Pod A Start pod A Pod A definition
34. Principle #2 The Kubernetes control plane is transparent -- there are no hidden internal APIs.
35. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server
36. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch
37. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler kubectl create pod A watch watch watch
38. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition watch watch watch
39. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition node: Node B Update pod A watch watch
40. Why declarative over imperative? Simpler, more robust system that can easily recover from failure of components. • Components level triggered instead of edge triggered -- no “missing events” issues. • No single point of failure. • Simple master components.
41. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition node: Node B watch watch watch Pod A
42. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition node: Node B watch watch watch Pod A kubectl delete pod A
43. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch Pod A
44. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch
45. Why no hidden internal APIs? Makes Kubernetes composable and extensible. • Default component not working for you? ◦ Turn it off and replace it with your own. • Additional functionality not yet available? ◦ Write your own and to add it.
46. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch
47. • Principle: Kubernetes control plane transparent -- no hidden internal APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B My custom scheduler watch watch watch
48. Kubernetes Volume Plugins Kubernetes has many volume plugins Remote Storage • GCE Persistent Disk • AWS Elastic Block Store • Azure File Storage • Azure Data Disk • Dell EMC ScaleIO • iSCSI • Flocker • NFS • vSphere • GlusterFS • Ceph File and RBD • Cinder • Quobyte Volume • FibreChannel • VMware Photon PD Ephemeral Storage • EmptyDir • Expose Kubernetes API • Secret • ConfigMap • DownwardAPI Local Persistent Volume (Beta) Out-of-Tree • Flex (exec a binary) • CSI (Beta) Other • Host path
49. Ephemeral storage • Volume whose lifecycle is tied to the lifecycle of pod. ◦ Temp empty scratch file space from host machine, when pod starts. ◦ Deleted when pod is terminated. • Enables sharing state between containers in a pod. • Plugin: EmptyDir
50. Kube API Data • Kubernetes API has lots of data that is interesting to workloads ◦ Secrets - Sensitive info stored in KubeAPI ▪ e.g. passwords, certificates, etc. ◦ ConfigMap - Configuration info stored in KubeAPI ▪ e.g. application startup parameters, etc. ◦ DownwardAPI - Pod information in KubeAPI ▪ e.g. name/namespace/uid of my current pod.
51. Fetching Kube API Data • Problem: ◦ How does application fetch secrets, config map, etc. information? • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Obvious solution: ◦ Could modify application to communicate directly with API Server. Master Node 1 API Server Scheduler Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Object Watch for new Pods
52. Fetching Kube API Data • Principle: Meet the user where they are. • Do not require an app to be re-rewritten to work in Kubernetes. • Many apps accept secrets and config info as files or env variables, expose Kube API in the way that. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Objects
53. Fetching Kube API Data • Principle: Meet the user where they are. • App can consume Secrets, ConfigMaps, and DownwardAPI in the way that it knows how to already (files and env variables). Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: secretVolume node: Node1 Container for pod1 Secret volume Fetch Secret file
54. Why meet the user where they are? Minimize hurdles for deploying workloads on Kubernetes.
55. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 Scheduler Watch for new Pods
56. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Scheduler Watch for new Pods Schedule PodA to Node1
57. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Principle: The control plane should be transparent -- there are no hidden internal APIs. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1
58. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1 gcePD1
59. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend gcePD1 Container for pod1 Create container Mount volume
60. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend gcePD1 Container for pod1 Mount volume Watch state of container
61. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend gcePD1 Container for pod1 Mount volume Watch state of container
62. PV/PVC PersistentVolume and PersistentVolumeClaim Abstraction Decouple storage implementation from storage consumption
63. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: gcePD1 StorageClass1 storage: gcePD Cluster admin facing API object User facing API object
64. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: awsEBS1 StorageClass1 storage: awsEBS Cluster admin facing API object User facing API object
65. Why workload portability? Decouple distributed system application development from cluster implementation. Make Kubernetes a true abstraction layer, like an OS.
66. Kubernetes Principles Introduced 1. Kube API declarative over imperative. 2. No hidden internal APIs 3. Meet the user where they are 4. Workload portability
67. Workshop by Mihir Pandya • Demo time! • Follow along, and do it yourself: https://tinyurl.com/k8s-experience
68. Event feedback survey • tinyurl.com/sj37d Get Involved! • https://kubernetes.io/community/ • Special Interest Groups (SIGs) Thank you! Recruiting questions? • Email uiucstudents@google.com Contact Saad • github.com/saad-ali • twitter.com/the_saad_ali