Confidential + Proprietary Confidential + Proprietary Kubernetes, Storage, and More! Saad Ali Staff Software Engineer, Google October 25, 2021 github.com/saad-ali twitter.com/the_saad_ali

Confidential + Proprietary Agenda ● Kubernetes ● Kubernetes Storage ● CSI ● SIG Storage ● On-going projects ● How do I get involved? ● Q&A

Confidential + Proprietary What is Kubernetes? Problem: ● Given a set of machines (VMs, physical machines, etc.) how do you deploy workloads (web applications, databases, batch machine learning jobs, etc,) to them? Node A Node B Node C Node D

Confidential + Proprietary What is Kubernetes? In the “before-Kubernetes” times: ● Dependency hell ● DIY bash/powershell scripts with deployment verification test. ● Bespoke workload monitoring system ● Human intervention to recover

Confidential + Proprietary What is Kubernetes? ● System for deploying and monitoring containerized workloads to nodes in a cluster. ● Greek for “Helmsman”. ● Inspired by Google’s experience with “Borg”. ● Open source, written in Go. ● Manage applications, not machines.

Confidential + Proprietary What is Kubernetes? Application (Dev) Cluster Kernel/OS (System) Hardware

Confidential + Proprietary ● To learn more, check out: https://youtu.be/ZuIQurh_kDk What is Kubernetes?

Confidential + Proprietary Introduction to Kubernetes Storage Problem ● Containers are ephemeral: no way to persist state Results in misinformation, like: ● “Only run stateless workloads on Kubernetes”

Confidential + Proprietary What is storage? Data Service Block/File Storage Physical Storage Object Store, SQL/NoSQL DB, Message Queue, etc. NFS, iSCSI Fibre Channel, etc. SSD/Flash Disk Stateful App Your stateful app

Confidential + Proprietary Introduction to Kubernetes Storage In the “before-Kubernetes” times: ● Bunch of local disks ○ Stateful app writes to local disk (e.g. node local SDD or HDD). ○ Pros: ■ Fast, easy. ○ Cons: ■ Unreliable ● Reliable External storage ○ Stateful app writes to reliable, external storage (e.g. storage appliance, cloud disk, etc.) ○ Pros: ■ Reliable, ○ Cons: ■ Difficult

Confidential + Proprietary Introduction to Kubernetes Storage ● With Kubernetes ○ Bunch of local disks ■ Can still use local disks (aka Kubernetes PersistentLocalVolumes) ○ Reliable External storage ■ Much better experience for “Reliable External storage” ● Automatic (intelligent) provisioning. ● Intelligent scheduling based on storage. ● Storage automatically available to correct node and pod. ● Storage moved along with workload. ● Portable Kubernetes Storage API -- write once run anywhere

Confidential + Proprietary Introduction to CSI Problem ● How does Kubernetes know how to talk to an arbitrary, external block/file storage system?

Confidential + Proprietary Introduction to CSI CSI makes Kubernetes volume layer truly extensible.

Confidential + Proprietary Introduction to CSI ● Plugin system for external block/file storage ● Easy to use CSI Drivers ○ Ensure storage system up and accessible. ○ Deploy CSI Driver YAML ● Easy to develop CSI Drivers ○ Write a GRPC service implementing CSI interface ○ Containerize it to make it easy to deploy. ○ Generate YAML to give to customers ■ Pair your CSI driver container with provided CSI Sidecar Containers

Confidential + Proprietary Introduction to Kubernetes Storage and CSI ● To learn more, check out: https://youtu.be/uSxlgK1bCuA ● For CSI see: https://youtu.be/ktwY1anKN58

Confidential + Proprietary What are Kubernetes SIGs? ● Kubernetes Special Interest Groups ○ Groups of contributors focused on development/improvements Kubernetes sub-area. ○ 25 Kubernetes SIGs today ■ See https://github.com/kubernetes/community/blob/master/sig-list.md ○ Examples: ■ SIG Network, SIG Storage, SIG Node, SIG Security, SIG UI, etc. ● Workgroups ○ Time bounded groups of contributors focused on development/improvements Kubernetes sub-area. ○ Owned by one or more SIG ○ Examples ■ Data Protection Workgroup (co-owned by SIG Storage and SIG Apps).

Confidential + Proprietary ● Group of Kubernetes Contributors responsible for: ○ Ensuring file and block storage (whether ephemeral or persistent, local or remote) are available wherever a container is scheduled. ○ Provisioning, attaching, mounting, unmounting, detaching, and deleting volumes ○ Influencing scheduling of containers based on storage (data gravity, availability, etc.). ○ Storage capacity management (container ephemeral storage usage, volume resizing, etc.) What does SIG Storage do?

Confidential + Proprietary SIG Storage Leadership SIG-Storage Co-Chair Saad Ali Xing Yang Michelle Au Jan Šafránek SIG-Storage Tech Lead

Confidential + Proprietary ● Some notable examples of features owned by SIG Storage: ○ Persistent Volume Claims and Persistent Volumes ○ Storage Classes and Dynamic Provisioning ○ Kubernetes volume plugins ○ Container Storage Interface (CSI) ○ Secret, ConfigMap, DownwardAPI Volumes ○ And lots more! ● SIG Storage Website: ○ https://github.com/kubernetes/community/tree/master/sig-storage What does SIG Storage do?

Confidential + Proprietary What does SIG Storage do? ● Code features, write tests, fix bugs for volume related features. ● Meet virtually every two weeks to plan and discuss. ● Meet face-to-face every now and then to close on bigger issues. ● Help each other and the community via slack and google groups.

Confidential + Proprietary What we did in 1.22 ● GA ○ CSI Windows ○ Pass pod service account token to CSI ■ Enables CSI drivers to authenticate as pod ● Alpha ○ Volume populator (re-design) ■ After provisioning, populate pod with data before giving to pod ○ Read Write Once Pod PV Access Mode ■ Enforces at most a single pod can mount a volume at a time ○ Delegate FSGroup to CSI Driver instead of Kubelet ■ More efficient fsgroup handling for certain drivers

Confidential + Proprietary What we are working on in 1.23 ● Targeting GA ○ Skip volume ownership (FSGroup) ○ CSI FSGroup Policy ○ Generic ephemeral volumes ● Targeting Beta ○ Delegate FSGroup to CSI Driver instead of Kubelet (alpha in 1.22) ○ CSI volume health (metrics) ○ Volume populator ○ On-going effort: CSI migration ○ On-going effort: Volume expansion ● Targeting Alpha ○ Object Storage API (COSI) ○ Recovering from resize failures ○ Prevent PV leaks when deleting out of order ○ Secret Deletion Protection (“Liens”)

Confidential + Proprietary How to get involved w/SIG Storage? ● Start at the SIG Storage page: ○ https://github.com/kubernetes/community/tree/master/sig-storage ● Attend the bi-weekly meetings: 9 AM PT every second Thursday. ○ Zoom meeting! Attend from anywhere. ○ Agenda doc -- feel free to add items for discussion to this doc. ○ Next one Nov 4, 2021 ● Familiarize yourself with the code. Start from main method walk through it. ○ Help fix a bug! ○ 233 open SIG storage Issues (as of 10/25/21) ○ Filter by “Help wanted” label. ● Help write tests!

Confidential + Proprietary How to get involved w/SIG Storage? ● Help write features! ○ There is a new Kubernetes version released every quarter (e.g. v1.9, v1.10, v1.11…) ○ Release schedules: ■ https://github.com/kubernetes/sig-release/tree/master/releases/ ● SIG Storage Planning Spreadsheet ○ Beginning of every quarter: planning and assignments ○ During quarter: help needed on assigned items & sometimes new items pop up. ● Every feature must have: ○ Enhancement issue in github.com/kubernetes/enhancements/ ○ KEP in github.com/kubernetes/enhancements/tree/master/keps/sig-storage

Confidential + Proprietary Confidential + Proprietary Thank you! Questions? github.com/saad-ali twitter.com/the_saad_ali