Kubernetes Design Principles

Illustrating the principles of Kubernetes design, by showing how Kubernetes works.

1. Kubernetes Design Principles May 3, 2018 Saad Ali Senior Software Engineer, Google Co-Lead SIG-Storage github.com/saad-ali twitter.com/the_saad_ali
2. Agenda • Illustrate principles of Kubernetes design, by showing how Kubernetes works. • What’s in it for me? ◦ A deeper understanding of Kubernetes
3. What is Kubernetes? • Containerization was the key ◦ Consistent, repeatable, reliable deployments on a wide variety of systems. • Who will manage it? ◦ You? Scripts? A system you write? • Kubernetes is a system for monitoring & deploying containerized workloads to nodes in a cluster.
4. Creating a Pod • Principle: Kubernetes APIs are declarative rather then imperative. • Create an API object (using CLI or REST API) to represent what you want to do. • All the components in the system will work to drive towards that state, until the object is deleted. • Example: Declare container “mycontainer” should be running. apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: internal.mycorp.com:5000/mycontainer:1.7.9
5. Why declarative over imperative? More robust system that can easily recover from failure of components. • No single point of failure. • Components level triggered instead of edge triggered -- no “missing events” issues.
6. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node Create a new Pod that looks like...
7. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node Create a new Pod that looks like...
8. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node
9. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Schedule PodA to Node1
10. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Schedule PodA to Node1
11. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Docker Daemon Create container Container for pod1
12. Creating a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Docker Daemon Watch state of container Container for pod1
13. Stopping a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. • To terminate a pod, you delete the pod object. • Principle: Kubernetes APIs are declarative rather then imperative. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Pod A Node: node1 Docker Daemon Container for pod1
14. Stopping a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. • To terminate a pod, you delete the pod object. • Principle: Kubernetes APIs are declarative rather then imperative. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Stop container Container for pod1
15. Stopping a Pod • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Every component watches the Kubernetes API and works to drive towards desired state. • To terminate a pod, you delete the pod object. • Principle: Kubernetes APIs are declarative rather then imperative. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon
16. Why no hidden internal APIs? Makes Kubernetes composable and extensible. • Default component not working for you? ◦ Turn it off and replace it with your own. • Additional functionality not yet available? ◦ Write your own and to add it.
17. Kubernetes Volume Plugins Kubernetes has many volume plugins Remote Storage • GCE Persistent Disk • AWS Elastic Block Store • Azure File Storage • Azure Data Disk • Dell EMC ScaleIO • iSCSI • Flocker • NFS • vSphere • GlusterFS • Ceph File and RBD • Cinder • Quobyte Volume • FibreChannel • VMware Photon PD Ephemeral Storage • EmptyDir • Expose Kubernetes API • Secret • ConfigMap • DownwardAPI Local Persistent Volume (Beta) Out-of-Tree • Flex (exec a binary) • CSI (Beta) Other • Host path
18. • Volume whose lifecycle is tied to the lifecycle of pod. ◦ Temp empty scratch file space from host machine, when pod starts. ◦ Deleted when pod is terminated. • Enables sharing state between containers in a pod. • Plugin: EmptyDir Ephemeral storage
19. • Kubernetes API has lots of data that is interesting to workloads ◦ Secrets - Sensitive info stored in KubeAPI ▪ e.g. passwords, certificates, etc. ◦ ConfigMap - Configuration info stored in KubeAPI ▪ e.g. application startup parameters, etc. ◦ DownwardAPI - Pod information in KubeAPI ▪ e.g. name/namespace/uid of my current pod. Kube API Data
20. Fetching Kube API Data • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Could modify application to communicate directly with API Server. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Object
21. Fetching Kube API Data • Principle: The control plane should be transparent -- there are no hidden internal APIs. • Modify application to communicate directly with API Server • Principle: Meet the user where they are. • Do not require an app to be re-rewritten to work in Kubernetes. • Many apps accept secrets and config info as files or env variables, expose Kube API in the way that. Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A node: Node1 Container for pod1 Fetch Secret Objects
22. • Principle: Meet the user where they are. • App can consume Secrets, ConfigMaps, and DownwardAPI in the way that it knows how to already (files and env variables). Fetching Kube API Data Master Node 1 API Server Scheduler Kubelet Watch for new Pods Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: secretVolume node: Node1 Container for pod1 Secret volume Fetch Secret file
23. Why meet the user where they are? Minimize hurdles for deploying workloads on Kubernetes.
24. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 Scheduler Watch for new Pods
25. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Scheduler Watch for new Pods Schedule PodA to Node1
26. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Principle: The control plane should be transparent -- there are no hidden internal APIs. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1
27. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Attach gcePD1 to Node1
28. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Create container Mount volume
29. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Watch state of container
30. Remote Storage • Could directly reference a remote volume (GCE PD, AWS EBS, NFS, etc.) in pod definition just like ephemeral volumes (EmptyDir, SecretVolume, etc.). • Kubernetes will automatically make it available to workload • Problem: Pod definition is no longer portable across clusters. • Principle: Workload definitions should be portable across clusters. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: gcePD1 node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Storage Backend Watch state of container
31. PV/PVC PersistentVolume and PersistentVolumeClaim Abstraction Decouple storage implementation from storage consumption
32. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: gcePD1 StorageClass1 storage: gcePD Cluster admin facing API object User facing API object
33. Master Node 1 API Server Kubelet Watch for new Pods, scheduled to this node Docker Daemon Pod A storage: pvc-a node: Node1 Container for pod1 gcePD1 A/D Controller Watch for new Pods w/Volumes Watch state of container pvc-a storage: pv-1 storageClass: storageClass1 pv-1 storage: awsEBS1 StorageClass1 storage: awsEBS Cluster admin facing API object User facing API object
34. Why workload portability? Decouple distributed system application development from cluster implementation. Make Kubernetes a true abstraction layer, like an OS.
35. Kubernetes Principles Introduced 1. Kube API declarative over imperative. 2. No hidden internal APIs 3. Meet the user where they are 4. Workload portability
36. • ◦ ◦ • ◦ • ◦ ◦ ◦ Question?