Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes Design Principles

Kubernetes Design Principles

Illustrating the principles of Kubernetes design, by showing how Kubernetes works.

515740717a9ba5d00fb79158c9071097?s=128

Saad Ali

July 16, 2018
Tweet

Transcript

  1. Kubernetes Design Principles May 3, 2018 Saad Ali Senior Software

    Engineer, Google Co-Lead SIG-Storage github.com/saad-ali twitter.com/the_saad_ali
  2. Agenda • Illustrate principles of Kubernetes design, by showing how

    Kubernetes works. • What’s in it for me? ◦ A deeper understanding of Kubernetes
  3. What is Kubernetes? • Containerization was the key ◦ Consistent,

    repeatable, reliable deployments on a wide variety of systems. • Who will manage it? ◦ You? Scripts? A system you write? • Kubernetes is a system for monitoring & deploying containerized workloads to nodes in a cluster.
  4. Creating a Pod • Principle: Kubernetes APIs are declarative rather

    then imperative. • Create an API object (using CLI or REST API) to represent what you want to do. • All the components in the system will work to drive towards that state, until the object is deleted. • Example: Declare container “mycontainer” should be running. apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: internal.mycorp.com:5000/mycontainer:1.7.9
  5. Why declarative over imperative? More robust system that can easily

    recover from failure of components. • No single point of failure. • Components level triggered instead of edge triggered -- no “missing events” issues.
  6. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node Create a new Pod that looks like...
  7. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node Create a new Pod that looks like...
  8. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node
  9. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Schedule PodA to Node1
  10. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Schedule PodA to Node1
  11. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Docker Daemon Create container Container for pod1
  12. Creating a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Docker Daemon Watch state of container Container for pod1
  13. Stopping a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. • To terminate a pod, you delete the pod object. • Principle: Kubernetes APIs are declarative rather then imperative. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Docker Daemon Container for pod1
  14. Stopping a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. • To terminate a pod, you delete the pod object. • Principle: Kubernetes APIs are declarative rather then imperative. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Stop container Container for pod1
  15. Stopping a Pod • Principle: The control plane should be

    transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. • To terminate a pod, you delete the pod object. • Principle: Kubernetes APIs are declarative rather then imperative. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon
  16. Why no hidden internal APIs? Makes Kubernetes composable and extensible.

    • Default component not working for you? ◦ Turn it off and replace it with your own. • Additional functionality not yet available? ◦ Write your own and to add it.
  17. Kubernetes Volume Plugins Kubernetes has many volume plugins Remote Storage

    • GCE Persistent Disk • AWS Elastic Block Store • Azure File Storage • Azure Data Disk • Dell EMC ScaleIO • iSCSI • Flocker • NFS • vSphere • GlusterFS • Ceph File and RBD • Cinder • Quobyte Volume • FibreChannel • VMware Photon PD Ephemeral Storage • EmptyDir • Expose Kubernetes API • Secret • ConfigMap • DownwardAPI Local Persistent Volume (Beta) Out-of-Tree • Flex (exec a binary) • CSI (Beta) Other • Host path
  18. • Volume whose lifecycle is tied to the lifecycle of

    pod. ◦ Temp empty scratch file space from host machine, when pod starts. ◦ Deleted when pod is terminated. • Enables sharing state between containers in a pod. • Plugin: EmptyDir Ephemeral storage
  19. • Kubernetes API has lots of data that is interesting

    to workloads ◦ Secrets - Sensitive info stored in KubeAPI ▪ e.g. passwords, certificates, etc. ◦ ConfigMap - Configuration info stored in KubeAPI ▪ e.g. application startup parameters, etc. ◦ DownwardAPI - Pod information in KubeAPI ▪ e.g. name/namespace/uid of my current pod. Kube API Data
  20. Fetching Kube API Data • Principle: The control plane should

    be transparent -- there are no hidden internal APIs. • Could modify application to communicate directly with API Server. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Object
  21. Fetching Kube API Data • Principle: The control plane should

    be transparent -- there are no hidden internal APIs. • Modify application to communicate directly with API Server • Principle: Meet the user where they are. • Do not require an app to be re-rewritten to work in Kubernetes. • Many apps accept secrets and config info as files or env variables, expose Kube API in the way that. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Objects
  22. • Principle: Meet the user where they are. • App

    can consume Secrets, ConfigMaps, and DownwardAPI in the way that it knows how to already (files and env variables). Fetching Kube API Data Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: secretVolume node: Node1 Container for pod1 Secret volume Fetch Secret file
  23. Why meet the user where they are? Minimize hurdles for

    deploying workloads on Kubernetes.
  24. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 Scheduler Watch for new Pods
  25. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Scheduler Watch for new Pods Schedule PodA to Node1
  26. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Principle: The control plane should be transparent -- there are no hidden internal APIs. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1
  27. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1
  28. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Create container Mount volume
  29. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Watch state of container
  30. Remote Storage • Could directly reference a remote volume (GCE

    PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Watch state of container
  31. PV/PVC PersistentVolume and PersistentVolumeClaim Abstraction Decouple storage implementation from storage

    consumption
  32. Master Node 1 API Server Kubelet Watch for new Pods,

    scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: gcePD1 StorageClass1 storage: gcePD Cluster admin facing API object User facing API object
  33. Master Node 1 API Server Kubelet Watch for new Pods,

    scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: awsEBS1 StorageClass1 storage: awsEBS Cluster admin facing API object User facing API object
  34. Why workload portability? Decouple distributed system application development from cluster

    implementation. Make Kubernetes a true abstraction layer, like an OS.
  35. Kubernetes Principles Introduced 1. Kube API declarative over imperative. 2.

    No hidden internal APIs 3. Meet the user where they are 4. Workload portability
  36. • ◦ ◦ • ◦ • ◦ ◦ ◦ Question?