Storage 101

In this introductory talk we’ll cover usage of persistent storage in Kubernetes so you can avoid its steep learning curve and common gotchas.

Why does it look so complicated? What is a PersistentVolume? What is a PersistentVolumeClaim? Why are there two separate objects instead of just one? How is storage presented into my containers? How do I write scalable persistent applications? Where do I look to debug when storage is not working? How do I connect Kubernetes to my storage backend? What is this Container Storage Interface (CSI) thing that everybody is talking about???

We will answer all of these questions and more. You will leave this talk with a solid foundation for thinking about storage in Kubernetes as well as a greater understanding of how you can put the various pieces together to fit your unique use case.

Transcript

1. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 1/67 Kubernetes Storage 101 David Zhu, Google Jan Šafránek, Red Hat 1 / 65
2. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 2/67 Kubernetes Container Pod orchestrator. Pod = one or more containers. Containers are stateless. Cleared on exit. Unless a persistent volume is used. 2 / 65
3. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 3/67 Pod Pod kind: Pod apiVersion: v1 metadata: name: mysql spec: containers: - image: mysql:5.6 name: mysql ports: - containerPort: 3306 name: mysql env: - name: MYSQL_ROOT_PASSWORD value: opensesame Database is lost when mysql container ends! 3 / 65
4. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 4/67 Pod Mounts PersistentVolumeClaim into container(s). PersistentVolumeClaim (PVC) Application request for storage. Created by user / devops. Binds to single PV. Usable in Pods. PersistentVolume (PV) Pointer to physical storage. Binds to single PVC. Created by admin ("pre-provisioning"). Created by Kubernetes on demand ("dynamic provisioning"). Kubernetes Persistent Storage Objects 4 / 65
5. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 5/67 Portable across Kubernetes clusters. Pod PersistentVolumeClaim (PVC) Not portable across Kubernetes clusters. PersistentVolume (PV) StorageClass Both contain details about the storage: Volume plugin. IP addresses of storage server(s). Paths. Usernames / passwords. ... Kubernetes Persistent Storage Objects Portability 5 / 65
6. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 6/67 StorageClass Collection of PersistentVolumes with the same characteristics. "Fast", "Cheap", "Replicated", ... Parameters for dynamic provisioning. Created by admin. Subject of quota per namespace. Kubernetes Persistent Storage Objects 6 / 65
7. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 7/67 Mounts PersistentVolumeClaim into container(s). kind: Pod apiVersion: v1 metadata: name: mysql spec: volumes: - name: data persistentVolumeClaim: claimName: my-mysql-claim containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD value: opensesame volumeMounts: - name: data mountPath: /var/lib/mysql Pod 7 / 65
8. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 8/67 Request for storage. kind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-mysql-claim spec: resources: requests: storage: 1Gi accessModes: - ReadWriteOnce PersistentVolumeClaim 8 / 65
9. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 9/67 Request for storage. kind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-mysql-claim spec: resources: requests: storage: 1Gi accessModes: - ReadWriteOnce "Give me 1 GiB of storage." PersistentVolumeClaim 9 / 65
10. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 10/67 Request for storage. kind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-mysql-claim spec: resources: requests: storage: 1Gi accessModes: - ReadWriteOnce "Give me 1 GiB of storage." "That is mountable to single pod as read/write." PersistentVolumeClaim 10 / 65
11. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 11/67 Request for storage. kind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-mysql-claim spec: resources: requests: storage: 1Gi accessModes: - ReadWriteOnce "Give me 1 GiB of storage." "That is mountable to single pod as read/write." "And I don't really care about the rest." PersistentVolumeClaim 11 / 65
12. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 12/67 $ kubectl create -f claim.yaml persistentvolumeclaim/my-mysql-claim created $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE my-mysql-claim Bound pvc-6428 1Gi RWO standard 26s PVC creation 12 / 65
13. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 13/67 $ kubectl create -f pod.yaml pod/mysql created $ kubectl get pod NAME READY STATUS RESTARTS AGE mysql 1/1 Running 0 19s Pod creation 13 / 65
14. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 14/67 PVC debugging $ kubectl get pvc NAME STATUS my-broken-claim Pending 14 / 65
15. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 15/67 PVC debugging $ kubectl get pvc NAME STATUS my-broken-claim Pending $ kubectl describe pvc ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 8s (x4 over 53s) persistentvolume-controller storageclass.storage.k8s .io "foo" not found 15 / 65
16. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 16/67 apiVersion: v1 kind: PersistentVolume metadata: name: pv1 spec: capacity: storage: 2Gi accessModes: - ReadWriteMany - ReadWriteOnce - ReadOnlyMany storageClassName: cheap persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.121.1 path: "/vol/share-1" Some metadata. PersistentVolume 16 / 65
17. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 17/67 apiVersion: v1 kind: PersistentVolume metadata: name: pv1 spec: capacity: storage: 2Gi accessModes: - ReadWriteMany - ReadWriteOnce - ReadOnlyMany storageClassName: cheap persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.121.1 path: "/vol/share-1" Pointer to storage. AWS EBS, Azure DD, Ceph FS & RBD, CSI, FC, Flex, GCE PD, Gluster, iSCSI, NFS, OpenStack Cinder, Photon, Quobyte, StorageOS, vSphere PersistentVolume 17 / 65
18. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 18/67 apiVersion: v1 kind: PersistentVolume metadata: name: pv1 spec: capacity: storage: 2Gi accessModes: - ReadWriteMany - ReadWriteOnce - ReadOnlyMany storageClassName: cheap persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.121.1 path: "/vol/share-1" Size of the volume. PersistentVolume 18 / 65
19. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 19/67 apiVersion: v1 kind: PersistentVolume metadata: name: pv1 spec: capacity: storage: 2Gi accessModes: - ReadWriteMany - ReadWriteOnce - ReadOnlyMany storageClassName: cheap persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.121.1 path: "/vol/share-1" Access modes that the volume supports. PersistentVolume 19 / 65
20. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 20/67 apiVersion: v1 kind: PersistentVolume metadata: name: pv1 spec: capacity: storage: 2Gi accessModes: - ReadWriteMany - ReadWriteOnce - ReadOnlyMany storageClassName: cheap persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.121.1 path: "/vol/share-1" StorageClass where this volume belongs. PersistentVolume 20 / 65
21. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 21/67 apiVersion: v1 kind: PersistentVolume metadata: name: pv1 spec: capacity: storage: 2Gi accessModes: - ReadWriteMany - ReadWriteOnce - ReadOnlyMany storageClassName: cheap persistentVolumeReclaimPolicy: Retain nfs: server: 192.168.121.1 path: "/vol/share-1" What to do when the volume is not needed any longer. Recycle (deprecated), Retain, Delete PersistentVolume 21 / 65
22. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 22/67 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "50" Collection of PersistentVolumes with the same characteristics. Usually admin territory. Global, not namespaced. StorageClass 22 / 65
23. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 23/67 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "50" Who dynamically provisions volumes. Name of hardcoded volume plugin. Name of external provisioner. Name of CSI driver. StorageClass 23 / 65
24. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 24/67 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "50" Parameters for dynamic provisioning. Depend on the provisioner. StorageClass 24 / 65
25. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 25/67 apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "50" One StorageClass in the cluster can be default. PVC without any StorageClass gets the default one. StorageClass 25 / 65
26. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 26/67 PersistentVolume Life Cycle PersistentVolume Life Cycle 26 / 65 26 / 65
27. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 27/67 PersistentVolume: Dynamic Provisioning 27 / 65
28. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 28/67 PersistentVolume: Dynamic Provisioning 28 / 65
29. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 29/67 PersistentVolume: Dynamic Provisioning 29 / 65
30. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 30/67 PersistentVolume: Dynamic Provisioning 30 / 65
31. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 31/67 PersistentVolume: Dynamic Provisioning 31 / 65
32. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 32/67 PersistentVolume: Dynamic Provisioning 32 / 65
33. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 33/67 PersistentVolume: Dynamic Provisioning 33 / 65
34. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 34/67 PersistentVolume: Dynamic Provisioning 34 / 65
35. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 35/67 PersistentVolume: Manual Provisioning "Brownfield" use case. Using data of old apps. 35 / 65
36. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 36/67 PersistentVolume: Manual Provisioning "Brownfield" use case. Using data of old apps. 36 / 65
37. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 37/67 PersistentVolume: Release 37 / 65
38. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 38/67 PersistentVolume: Release 38 / 65
39. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 39/67 PersistentVolume: Release 39 / 65
40. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 40/67 PVC is deleted: persistentVolumeReclaimPolicy is executed: Recycle (deprecated): All data from the volume are removed ("rm -rf *"). PV is Available for new PVCs. Delete: Volume is deleted in the storage backend. PV is deleted. Usually for dynamically-provisioned volumes Retain: PV is kept Released. No PVC can bind to it. Admin should manually prune Released volumes. In all cases, user can't access the data! PersistentVolume: Release 40 / 65
41. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 41/67 Automatic: persistentVolumeReclaimPolicy = Delete. Manual: PV is not Bound. Does not delete volume on storage backend! PersistentVolume Life Cycle: Deletion 41 / 65
42. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 42/67 Stateful applications Stateful applications 42 / 65 42 / 65
43. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 43/67 Pods are not for users Pod can be deleted. Preemption. Node is drained (for update, ...) Node goes down. -> Users should not create Pod objects. 43 / 65
44. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 44/67 Kubernetes high-level objects Deployment Runs X replicas of a single Pod template. When a pod is deleted, Deployment automatically creates a new one. Scalable up & down. All pods share the same PVC! 44 / 65
45. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 45/67 Deployment 45 / 65
46. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 46/67 Deployment All three pods can overwrite data of each other! Most applications crash / refuse to work. 46 / 65
47. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 47/67 Kubernetes high-level objects StatefulSet Runs X replicas of a single Pod template. Each pod gets its own PVC(s) from a PVC template. When a pod is deleted, StatefulSet automatically creates a new one. Each pod has a stable identity. Scalable up & down. 47 / 65
48. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 48/67 StatefulSet The pods must be aware of the other StatefulSet members! Usually very complex setup. 48 / 65
49. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 49/67 Storage features Storage features 49 / 65 49 / 65
50. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 50/67 Topology aware scheduling PV can be usable only by subset of nodes. Cloud regions / availability zones. Bare metal datacenters. ... Pod must be scheduled: Where the PV is reachable. Where is enough resources to run the pod (CPU, memory, GPU, ...) PV provisioning is delayed until Pod is created for scheduler to pick a node that matches both PV & Pod. 50 / 65
51. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 51/67 Topology aware scheduling: Delayed binding PV provisioning is delayed until Pod is created for scheduler to pick a node that matches both PV & Pod. $ kubectl get pvc NAME STATUS my-delayed-claim Pending $ kubectl describe pvc ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 9s persistentvolume-controller waiting for first consumer to be created before binding 51 / 65
52. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 52/67 Topology aware scheduling: Delayed binding PV provisioning is delayed until Pod is created for scheduler to pick a node that matches both PV & Pod. $ kubectl get pvc NAME STATUS my-delayed-claim Pending $ kubectl describe pvc ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 9s persistentvolume-controller waiting for first consumer to be created before binding $ kubectl create -f pod.yaml pod/mysql created 51 / 65
53. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 53/67 Topology aware scheduling: Delayed binding PV provisioning is delayed until Pod is created for scheduler to pick a node that matches both PV & Pod. $ kubectl get pvc NAME STATUS my-delayed-claim Pending $ kubectl describe pvc ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 9s persistentvolume-controller waiting for first consumer to be created before binding $ kubectl create -f pod.yaml pod/mysql created $ kubectl get pvc NAME STATUS my-delayed-claim Bound Wednesday, Hall 8.0 D2, 15:55: Improving Availability for Stateful Applications in Kubernetes - Michelle Au 51 / 65
54. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 54/67 Local volumes Unused local disks can be used as PVs. Extra speed. Lower reliability. No pod scheduling flexibility. 52 / 65
55. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 55/67 Raw block Pods can get a block device of a PV. For extra speed. For software defined storage. 53 / 65
56. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 56/67 Resize Only expansion is supported. Offline. Online (alpha). 54 / 65
57. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 57/67 In-line volumes Persistent Pod directly references volume Not portable! apiVersion: v1 kind: Pod metadata: name: test-ebs spec: containers: - image: k8s.gcr.io/test-webserver name: test-container volumeMounts: - mountPath: /test-ebs name: test-volume volumes: - name: test-volume awsElasticBlockStore: volumeID: <volume-id> fsType: ext4 55 / 65
58. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 58/67 In-line volumes Ephemeral ConfigMap Secret Downward API 56 / 65
59. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 59/67 Container Storage Interface (CSI) Industry standard that will enable storage vendors (SP) to develop a plugin once and have it work across a number of container orchestration (CO) systems. No change from user perspective, Pods & PVCs as usual. Extra work for cluster admin. New Kubernetes external components: external-attacher external-provisioner node-driver-registrar cluster-driver-registrar external-resizer external-snapshotter ... 57 / 65
60. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 60/67 Snapshots Alpha. Part of CSI. Can take a snapshot of PVC. PVC can be provisioned from a snapshot. 58 / 65
61. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 61/67 CSI migration Alpha. Remove in-tree volume plugins. Translate to CSI seamlessly. 59 / 65
62. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 62/67 Summary Summary 60 / 65 60 / 65
63. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 63/67 Pod Mounts PersistentVolumeClaim into container(s). PersistentVolumeClaim (PVC) Application request for storage. Created by user / devops. PersistentVolume (PV) Pointer to physical storage. Created by Kubernetes on demand ("dynamic provisioning"). StorageClass Collection of PersistentVolumes with the same characteristics. Parameters for dynamic provisioning. Persistent Storage objects 61 / 65
64. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 64/67 It's not all! It's not all! 62 / 65 62 / 65
65. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 65/67 Kubecon EU 2019 David Zhu, Google & Jan Šafránek: Tutorial: Back to Basics: Hands-On Deployment of Stateful Workloads on Kubernetes, Tue 11:05 Josh Berkus: Benchmarking Cloud Native Storage, Tue 11:55 Saad Ali: Debunking the Myth: Kubernetes Storage is Hard (keynote), Wed 9:58 Jared Watts: Data Without Borders - Using Rook Storage Orchestration at a Global Scale, Wed 11:05 Jared Watts & Bassam Tabbara: Deep Dive: Rook, Wed 11:55 Iqbal Farabi & Tara Baskara: Benchmarking Cloud Native Databases Performance on Kubernetes, Wed 11:55 Sheng Yang: Build a Kubernetes Based Cloud Native Storage Solution From Scratch, Wed 12:30 Federico Lucifredi & Sébastien Han: Rook, Ceph, and ARM: A Caffeinated Tutorial, Wed 16:45 Michelle Au: Improving Availability for Stateful Applications in Kubernetes, Wed 15:55 Saad Ali: Intro + Deep Dive: Kubernetes Storage SIG, Thu 11:05 63 / 65
66. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 66/67 Reach out Kubernetes SIG Storage Bi-weekly meetings Slack Mailing list 64 / 65
67. 5/28/2019 Kubernetes Storage 101 https://jsafrane.github.io/kubecon2019/#1 67/67 Questions? Questions? 65 / 65 65 / 65