Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding Kubernetes Through Design Principles

515740717a9ba5d00fb79158c9071097?s=47 Saad Ali
September 20, 2018

Understanding Kubernetes Through Design Principles

UIUC's "Reflections | Projections" conference 2018
https://reflectionsprojections.org/

515740717a9ba5d00fb79158c9071097?s=128

Saad Ali

September 20, 2018
Tweet

Transcript

  1. Understanding Kubernetes Through Design Principles September 20, 2018 Saad Ali

    Senior Software Engineer, Google github.com/saad-ali twitter.com/the_saad_ali
  2. Introductions Saad Ali Google Fun fact:10 office moves in 4.5

    years Andrew McHugh Google Fun fact: Has never had antibiotics Mihir Pandya UIUC Fun fact: Puts milk first when eating cereal
  3. Agenda • Google: We’re hiring! • Very quickly: What is

    Kubernetes? • Illustrate principles of Kubernetes design, by showing how Kubernetes works. • Kubernetes Workshop
  4. Sign-in: bit.ly/2MAvGgs Administrivia

  5. Google: We’re hiring!

  6. 2018 | Confidential and Proprietary We’re hiring! Here’s what we’re

    looking for: Software Engineer (BS/MS) 8/6/18 - late spring g.co/jobs/swegrad Software Engineer, Intern (BS/MS) 9/17/18 - 12/14/18 7/9/18 - 9/28/18 (winter internship) g.co/jobs/sweintern (BS) g.co/jobs/msintern (MS) Software Engineer, PhD 8/6/18 - late spring g.co/jobs/swephd Software Engineer, PhD, Intern 9/17/18 - 2/8/18 g.co/jobs/phdintern Site Reliability Engineer (+Intern) Deadline: same as SWE (+Intern) TBD if separate application link Engineering Practicum Program 9/17/18 - 10/31/18 g.co/jobs/engpracticum Engineering Residency mid fall - early spring g.co/jobs/engresident Associate Product Manager early aug - end of oct. g.co/jobs/apmgrad SETI (+Intern) Deadline: same as SWE (+Intern) TBD if separate application link APM, Intern 9/17/18 - 10/31/18 g.co/jobs/apmintern UX Eng, Design, Writer, Research, (+Intern) early jan. - late jan. google.com/design/jobs Technical Program Manager Internship: dec. - jan. Data Scientist jan. 2019 g.co/jobs/datascientist Cloud Tech Residency 8/23/18 - early dec. g.co/jobs/ctr Information Technology Residency Program Internship: 9/24/18 - 9/29/18, 1/7/19 - 1/18/19 Full-Time: 9/5/18 - until filled Hardware/Mechanical Engineer Internship: jan. 2019 Full-Time: year round Technical Solutions Consultant Internship: jan. 2019 Applications Engineer Internship and Full-Time: jan. 2019 Technical Writer Internship: late jan. AI Residency (BS-PhD) Full-Time: oct. - jan. SWE in Research (BS, MS, PhD) Internship: year round Check out more roles google.com/students
  7. Cloud Technical Residency g.co/jobs/ctr INSERT PHOTO ❖ We’re on a

    mission to completely change the world of enterprise computing - you’ll have a front-row seat for the action! ❖ Full Time - rotational program within Cloud, various start dates ❖ Training + three project rotations ❖ Currently based in Austin, TX
  8. 2018 | Confidential and Proprietary Resources Build your future with

    Google Learn more about our programs, events, scholarships and more at g.co/buildyourfuture g.co/techdevguide Google’s Guide to Technical Development YouTube Live (GoogleStudents page) Check out recorded workshops and info sessions Interview Prep Texts Programming Interviews Exposed Landing Your Next Job, 3rd Edition Elements of Programming Interviews Cracking the Coding Interview, 6th Edition The Algorithm Design Manual, Steven S Skiena Practice questions Check out TopCoder for tutorials & questions. Sign up for an account on leetcode.com or pramp.com and have your practice answers reviewed by others. Awesome interview prep videos Visit our Life at Google YouTube page where we've shared some fantastic videos to help you Prepare for a Google Engineering Interview. Worth watching in full!
  9. What’s in it for me? • Goal: A deeper understanding

    of Kubernetes • Why you should care? ◦ Kubernetes quickly becoming standard ◦ Important in industry and research • Important tool for learning ◦ Understand the problem ◦ The “why” not just the “what”
  10. What is Kubernetes? Problem: • Given a set of machines

    (VMs, physical machines, etc.) how do you deploy workloads (web applications, databases, batch machine learning jobs, etc,) to them? Node A Node B Node C Node D
  11. Challenges: • Common libraries • No isolation • Highly coupled

    apps and OS What is Kubernetes? Old approach: • Multiple apps per machine Node A Node C Node D app kernel libs app app app Node A app kernel libs app app app kernel libs kernel libs
  12. Better approach: • Virtual Machines Challenges: • Some isolation •

    Expensive, inefficient, slow • Still highly coupled to the OS • Hard to manage What is Kubernetes? Node A Node C Node D libs app app kernel libs app app kernel Node B libs app app kernel libs app app kernel libs kernel libs kernel
  13. Even better approach: • Containers! What is Kubernetes? Benefits: •

    Performance • Repeatability • Isolation • Quality of service • Accounting • Visibility • Portability A fundamentally different way of managing applications Node A Node B libs app kernel libs app libs app libs app libs app kernel libs app libs app libs app
  14. Challenges • How will you deploy? ◦ Deploy my containers

    ◦ Monitor my containerized app ◦ Make my containerized app robust ◦ Scale my containerized app • But who will manage your containers? ◦ You? Scripts? A system you write? Containers are great, but... What is Kubernetes? Node A Node B libs app kernel libs app libs app libs app libs app kernel libs app libs app libs app
  15. What is Kubernetes? • Kubernetes is a system for monitoring

    & deploying containerized workloads to nodes in a cluster (a container orchestrator). • Greek for “Helmsman”; also the root of the word “Governor” • Supports multiple container runtimes (including Docker) • Supports multiple cloud and bare-metal environments • Inspired and informed by Google’s experiences • Open source, written in Go • Manage applications, not machines
  16. What is Kubernetes? Application (Dev) Cluster Kernel/OS (System) Hardware

  17. Problem #1: How to deploy a workload? Obvious solution! Master:

    API Server Node A Node B Start containers x
  18. Problem #1: How to deploy a workload? Obvious solution! Problems

    with this approach? Master: API Server Node A Node B Container A Start containers x
  19. Problem #1: How to deploy a workload? Obvious solution! Problems

    with this approach? What if: • Container crashes and dies? • What if node crashes and dies? • What if node B had a momentary blip? Master: API Server Node A Node B Container A
  20. Obvious solution! Problems with this approach? What if: • Container

    crashes and dies? • What if node crashes and dies? • What if node B had a momentary blip? Problem #1: How to deploy a workload? Master: API Server Node A Node B Start containers x
  21. Principle #1 Kubernetes APIs are declarative rather then imperative.

  22. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: define desired state • System: works to drive towards that state How to deploy a workload? Kubernetes way!
  23. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B kubectl create -f podA.yaml
  24. How to deploy a workload? Kubernetes way! Sample pod definition

    file apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: internal.mycorp.com:5000/mycontainer:1.7.9
  25. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition
  26. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: create API object that is persisted on kube API server until deletion • System: all components work in parallel to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
  27. Automatic recovery! Why declarative over imperative?

  28. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: define create API objects that is persisted on kube API server until deletion • System: all components work in parallel to drive do their part to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
  29. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: define create API objects that is persisted on kube API server until deletion • System: all components work in parallel to drive do their part to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
  30. • Principle: Kubernetes APIs are declarative rather then imperative. •

    You: define create API objects that is persisted on kube API server until deletion • System: all components work in parallel to drive do their part to drive to that state How to deploy a workload? Kubernetes way! Master: API Server Node A Node B Pod A definition Pod A
  31. Problem #2: how do nodes figure out what to do?

    Master: API Server Node A Node B kubectl create pod A Pod A definition Obvious solution!
  32. Obvious solution! Problems with this approach? Master has to •

    Keep track of state of every other component! • “Catch up” any failed components that missed calls. Problem #2: how do nodes figure out what to do? Master: API Server Node A Node B Pod A Start pod A Pod A definition
  33. Obvious solution! Problems with this approach? Master becomes: • Complex

    • Difficult to extend Problem #2: how do nodes figure out what to do? Master: API Server Node A Node B Pod A Start pod A Pod A definition
  34. Principle #2 The Kubernetes control plane is transparent -- there

    are no hidden internal APIs.
  35. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server
  36. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch
  37. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler kubectl create pod A watch watch watch
  38. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition watch watch watch
  39. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition node: Node B Update pod A watch watch
  40. Why declarative over imperative? Simpler, more robust system that can

    easily recover from failure of components. • Components level triggered instead of edge triggered -- no “missing events” issues. • No single point of failure. • Simple master components.
  41. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition node: Node B watch watch watch Pod A
  42. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler Pod A definition node: Node B watch watch watch Pod A kubectl delete pod A
  43. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch Pod A
  44. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch
  45. Why no hidden internal APIs? Makes Kubernetes composable and extensible.

    • Default component not working for you? ◦ Turn it off and replace it with your own. • Additional functionality not yet available? ◦ Write your own and to add it.
  46. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B Master: Scheduler watch watch watch
  47. • Principle: Kubernetes control plane transparent -- no hidden internal

    APIs • All components watch the Kubernetes API, and figure out what they need to do. Everything watches and reacts to the API server Master: API Server Node A Node B My custom scheduler watch watch watch
  48. Kubernetes Volume Plugins Kubernetes has many volume plugins Remote Storage

    • GCE Persistent Disk • AWS Elastic Block Store • Azure File Storage • Azure Data Disk • Dell EMC ScaleIO • iSCSI • Flocker • NFS • vSphere • GlusterFS • Ceph File and RBD • Cinder • Quobyte Volume • FibreChannel • VMware Photon PD Ephemeral Storage • EmptyDir • Expose Kubernetes API • Secret • ConfigMap • DownwardAPI Local Persistent Volume (Beta) Out-of-Tree • Flex (exec a binary) • CSI (Beta) Other • Host path
  49. Ephemeral storage • Volume whose lifecycle is tied to the

    lifecycle of pod. ◦ Temp empty scratch file space from host machine, when pod starts. ◦ Deleted when pod is terminated. • Enables sharing state between containers in a pod. • Plugin: EmptyDir
  50. Kube API Data • Kubernetes API has lots of data

    that is interesting to workloads ◦ Secrets - Sensitive info stored in KubeAPI ▪ e.g. passwords, certificates, etc. ◦ ConfigMap - Configuration info stored in KubeAPI ▪ e.g. application startup parameters, etc. ◦ DownwardAPI - Pod information in KubeAPI ▪ e.g. name/namespace/uid of my current pod.
  51. Fetching Kube API Data • Problem: ◦ How does application

    fetch secrets, config map, etc. information? • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Obvious solution: ◦ Could modify application to communicate directly with API Server. Master Node 1 API Server Scheduler Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Object Watch for new Pods
  52. Fetching Kube API Data • Principle: Meet the user where

    they are. • Do not require an app to be re-rewritten to work in Kubernetes. • Many apps accept secrets and config info as files or env variables, expose Kube API in the way that. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Objects
  53. Fetching Kube API Data • Principle: Meet the user where

    they are. • App can consume Secrets, ConfigMaps, and DownwardAPI in the way that it knows how to already (files and env variables). Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: secretVolume node: Node1 Container for pod1 Secret volume Fetch Secret file
  54. Why meet the user where they are? Minimize hurdles for

    deploying workloads on Kubernetes.
  55. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 Scheduler Watch for new Pods
  56. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Scheduler Watch for new Pods Schedule PodA to Node1
  57. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Principle: The control plane should be transparent -- there are no hidden internal APIs. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1
  58. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1 gcePD1
  59. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend gcePD1 Container for pod1 Create container Mount volume
  60. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend gcePD1 Container for pod1 Mount volume Watch state of container
  61. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend gcePD1 Container for pod1 Mount volume Watch state of container
  62. PV/PVC PersistentVolume and PersistentVolumeClaim Abstraction Decouple storage implementation from storage

    consumption
  63. Master Node 1 API Server Kubelet Watch for new Pods,

    scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: gcePD1 StorageClass1 storage: gcePD Cluster admin facing API object User facing API object
  64. Master Node 1 API Server Kubelet Watch for new Pods,

    scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: awsEBS1 StorageClass1 storage: awsEBS Cluster admin facing API object User facing API object
  65. Why workload portability? Decouple distributed system application development from cluster

    implementation. Make Kubernetes a true abstraction layer, like an OS.
  66. Kubernetes Principles Introduced 1. Kube API declarative over imperative. 2.

    No hidden internal APIs 3. Meet the user where they are 4. Workload portability
  67. Workshop by Mihir Pandya • Demo time! • Follow along,

    and do it yourself: https://tinyurl.com/k8s-experience
  68. Event feedback survey • tinyurl.com/sj37d Get Involved! • https://kubernetes.io/community/ •

    Special Interest Groups (SIGs) Thank you! Recruiting questions? • Email uiucstudents@google.com Contact Saad • github.com/saad-ali • twitter.com/the_saad_ali