Your ReadWriteMany (RWX) Storage in k8s with Manila CSI - OpenInfraSummit 2022

Kubernetes end users have trouble deploying multiple containers that write to the same persistent volume.

Kubernetes administrators lack native solutions for hard separation of shared storage across multiple tenants.

The Manila CSI plugin for the OpenStack Cloud Provider enables solutions for both problems.

Kubernetes end-users can follow a simple design pattern to write safely to dynamically provisioned general purpose RWX volumes

Kubernetes administrators can deploy Manila CSI using a straight-forward set of manifests

Manila administrators can set up service for Manila CSI with a minimum of fuss

Cloud Provider OpenStack Keystone can be used for hard tenant separation without sacrificing the economies of shared scale out infrastructure and elastic self-service provisioning

Demos, YAML, and manila configuration included.

https://github.com/vkmc/rxw-storage-k8s-manila-csi

Transcript

1. Your ReadWriteMany (RWX) Storage in k8s with Manila CSI VICTORIA MARTINEZ DE LA CRUZ GOUTHAM PACHA RAVI CHRISTIAN SCHWEDE
2. • Introduce k8s storage & CSI • Cinder CSI, Manila CSI • OpenStack Manila • How to deploy Manila CSI • How to use (w/ link to demo) • Future work Agenda
3. Storage in k8s
4. - Even stateless apps still need persistent storage - Physical Volumes (PV) became generally available in k8s 1.14 - In-tree drivers - requires vendors to contribute to k8s - align with release cadence - Reviews, maintenance etc. from upstream contributors required - Need a more flexible way to add storage drivers for container orchestrators Storage in k8s
5. - Developed as a standard for exposing arbitrary block and file storage systems to containerized workloads - Enable storage vendors to develop a plugin once and have it work across a number of container orchestration (CO) systems - More info in Container Storage Interface (CSI) for Kubernetes GA - Migrating environments using in-tree volume plugins to CSI versions is an ongoing effort. There is a CSI migration tool that can be used. Container Storage Interface (CSI)
6. Cinder CSI Block Storage RWO and RWX modes Multi backend support Hard multitenancy (via Keystone) Cloud Provider OpenStack CSI drivers Manila CSI File Storage RWO and RWX modes Multi backend support Hard multitenancy (via Keystone)
7. Access modes - ReadWriteOnce (RWO) - The volume can be mounted as read-write by a single node - e.g. stateful apps - ReadOnlyMany (ROX) - The volume can be mounted as read-only by many nodes - e.g. backups - ReadWriteMany (RWX) - the volume can be mounted as read-write by many nodes - e.g. shared file systems - ReadWriteOncePod (RWOP) (new in k8s 1.22) - the volume can be mounted as read-write by a single pod - e.g. sensitive data handling - More info here
8. OpenStack Manila and Manila CSI
9. - Shared file systems as a service for OpenStack, - Multi-backend - In OpenStack, supports 35+ storage backends (proprietary and open source), including NetApp, Ceph, Dell EMC, Gluster and more - Multi-protocol - In OpenStack supports NFS, CIFS, GlusterFS, HDFS, CephFS or MAPRFS - In k8s, supports NFS and CephFS - Manila CSI can create, expand, snapshot, restore and mount Manila shares OpenStack Manila and Manila CSI
10. Shares usage workflow
11. Shares usage workflow extended
12. Manila CSI sample deployment - Manila CSI Controller Plugin - Manila CSI Node Plugin is a proxy - CSI NFS - CSI Ceph
13. Manila CSI sample deployment
14. How to deploy Manila CSI
15. - Manila CSI driver - Sidecar containers - Roles - CRD - StateFulSet - DaemonSet - NFS CSI Driver - StorageClass What do we need?
16. - Deploy Manila CSI driver using helm charts helm repo add cpo https://kubernetes.github.io/cloud-provider-openstack helm repo update helm install manila-csi cpo/openstack-manila-csi - Deploy NFS CSI driver helm repo add csi-driver-nfs …/kubernetes-csi/csi-driver-nfs/master/charts helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs - Create a StorageClass https://github.com/kubernetes/cloud-provider-openstack/tree/master/examples/manila-csi-plugin How to deploy Manila CSI on k8s
17. - Manila CSI driver operator …checks if Manila is provided, if yes… 1. Installs Manila CSI driver 2. Installs NFS CSI driver 3. Create StorageClass for each share type - also resyncs - OpenShift 4.6+: installed by default How to deploy Manila CSI on OpenShift
18. How to use Manila CSI
19. - Define a StorageClass referencing to the manila.csi.openstack.org provisioner - Manila CSI operator queries manila share types and create these automatically How to use Manila CSI - Storage Classes apiVersion: v1 items: - apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-manila-default parameters: … provisioner: manila.csi.openstack.org reclaimPolicy: Delete volumeBindingMode: Immediate kind: List metadata: …
20. - Define a PVC referencing to the csi-manila-default StorageClass that has been created How to use Manila CSI - PVCs apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mypvc spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi storageClassName: csi-manila-default
21. Manila CSI RWX demo - https://asciinema.org/a/499321 How to use Manila CSI - Demo
22. - Performance testing - I/O tests w/FIO - API Scale tests w/OpenStack Browbeat - Latency - Backups - Using Velero/Restic and Kanister.io to perform backups (check out “Your Manila CephFS Share Backups Belong to S3” by Robert Vasek from KubeCon 2022) Future work
23. Questions? victoria@redhat.com gouthamr@redhat.com cschwede@redhat.com
24. • Lorem ipsum dolor sit amet, consectetur adipiscing elit. • Donec scelerisque vehicula posuere. Aenean eu tristique orci, vel dignissim turpis. • Lorem ipsum dolor sit amet, consectetur adipiscing elit. • Donec scelerisque vehicula posuere. Aenean eu tristique orci, vel dignissim turpis. Text right, smaller image left
25. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer at elit massa. Pellentesque in mi libero. In libero metus, semper et congue luctus, tincidunt nec ipsum. Nunc id nulla justo. Suspendisse potenti. Ut congue ex eu vulputate facilisis. Praesent tincidunt porttitor faucibus. Sed quam velit, dignissim ac tristique eu, scelerisque eu neque. Maecenas ut maximus sem. Text right, larger image left
26. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer at elit massa. Pellentesque in mi libero. In libero metus, semper et congue luctus, tincidunt nec ipsum. Nunc id nulla justo. Suspendisse potenti. Ut congue ex eu vulputate facilisis. Praesent tincidunt porttitor faucibus. Sed quam velit, dignissim ac tristique eu, scelerisque eu neque. Maecenas ut maximus sem. Text left, larger image right
27. Section header or impact slide
28. Thank you!
29. Container Storage Interface (CSI) Worker Node Node CSI Plugin node-driver-registrar liveness probe Controller Node Controller CSI Plugin external-provisioner external-attacher liveness probe Worker Node Node CSI Plugin node-driver-registrar liveness probe Worker Node Node CSI Plugin node-driver-registrar liveness probe Worker Node Node CSI Plugin node-driver-registrar liveness probe Worker Node Node CSI Plugin node-driver-registrar liveness probe Controller Node Controller CSI Plugin external-provisioner external-attacher liveness probe Controller Node Controller CSI Plugin external-provisioner external-attacher liveness probe
30. Cloud Provider OpenStack OpenStack Manila Cinder Keystone Nova Neutron Octavia Glance Swift K8s / OpenShift VM Controller VM Controller VM Controller VM Controller VM Controller VM Node VM Node VM Node K8s / OpenShift VM Controller VM Controller VM Controller VM Controller VM Controller VM Node VM Node VM Node