Rook: Intro and Deep Dive With Ceph

The slides of the following session at KubeCon+CloudNativeCon Japan 2025

https://kccncjpn2025.sched.com/event/1x6zk/rook-intro-and-deep-dive-with-ceph-satoru-takeuchi-cybozu-inc

Transcript

2. Rook: Intro and Deep Dive With Ceph Satoru Takeuchi, Cybozu, Inc. June 17th, 2025 2
3. Title Agenda • Introduction to Rook and Ceph • Block and Filesystem Storage • Object Storage • Other Features • Project Health 3
4. Introduction to Rook and Ceph 4
5. Title What is Rook? • An open source K8s operator to manage Ceph storage • For Admins ◦ Deploy, manage, upgrade Ceph cluster by CR • For Users ◦ Consume Ceph by PVC and OBC CR 5
6. Title What is Ceph? • All-in-one open source distributed storage platform 6 Name Type RBD Block Storage CephFS Large-scale share filesystem storage RGW S3-compatible object storage CephNFS Export CephFS and S3 object as NFS
7. Remote replications 7 Storage Type Feature RBD RBD mirroring CephFS CephFS mirroring RGW RGW multisite Ceph cluster Region A Region B Ceph cluster Replicate
8. Ceph’s Architecture • OSD daemons ◦ Manage data • MON daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 8 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … …
9. Ceph’s Architecture • OSD daemons ◦ Manage data • MON daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 9 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … …
10. Ceph’s Architecture • OSD daemons ◦ Manage data • MON daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 10 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … …
11. Ceph’s Architecture • OSD daemons ◦ Manage data • MON daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 11 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … …
12. Title • High scalability ◦ Real example: ~1800 OSDs, over 5 PiB • High durability ◦ Replication or Erasure Coding ◦ Configurable failure domains (e.g. rack) • High availability ◦ e.g. Add/remove/replace OSDs online Ceph’s Characteristics 12
13. Title Rook’s Architecture • Rook operator ◦ Manage Rook/Ceph clusters ◦ Provision Pod for each Ceph daemons • Ceph CSI ◦ A CSI Driver for Ceph ◦ Provisions storage from Ceph 13 Rook/Ceph Cluster Mons Mons MON Pod Mons MGR Pods Storage (e.g.RBD) OSD Pod OSD Pod OSD Pod disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … … MGR Pod Ceph CSI manage Rook operator manage
14. Example: Provisioning and expanding cluster 1. Deploy a minimum Ceph cluster 2. Expand the cluster disk1 14 Rook operator disk0
15. kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 1 Step1: Deploy a Minimum Cluster 15 Admin disk0 disk1 1. Create Rook operator
16. Step1: Deploy a Minimum Cluster kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 1 Rook/Ceph cluster (0 OSD) disk1 16 Rook Admin disk0 3. Create 2. Detect
17. Step1: Deploy a Minimum Cluster kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 1 Rook/Ceph cluster (1 OSD) disk1 17 Admin disk0 4. Create OSD0 Pod Rook operator
18. kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 2 Step2: Expand the cluster Rook/Ceph cluster (1 OSD) disk1 18 Admin disk0 1. Update OSD0 Pod Rook operator
19. Step2: Expand the cluster kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count: 2 Rook/Ceph cluster (2 OSDs) disk1 19 Admin disk0 3. Create OSD0 Pod OSD1 Pod 2. Detect Rook operator
20. Block and Filesystem Storage 20
21. Supported Configurations 21 Storage Volume Mode Access Mode RBD Block, Filesystem RWO, RWOP, ROX CephFS Filesystem RWX, RWO, ROX, RWOP CephNFS The same as above The same as above
22. Title Additional Features 22 Storage Volume Expansion, snapshot, and cloning Static provisioning QoS RBD ✅ ✅ ✅ CephFS ✅ ✅ CephNFS ✅
23. Example: Consuming a Block Volume 1. Create an RBD pool 2. Consume a block volume 23 Rook operator Rook/Ceph cluster Ceph CSI
24. Rook/Ceph cluster Step1: Create an RBD Pool kind: CephBlockPool metadata: name: my-pool spec: replicated: size: 3 — kind: StorageClass metadata: name: ceph-block parameters: pool: my-pool 24 Ceph CSI Admin 1. Create Rook operator
25. Rook/Ceph cluster Step1: Create an RBD Pool kind: CephBlockPool metadata: name: my-pool spec: replicated: size: 3 — kind: StorageClass metadata: name: ceph-block parameters: pool: my-pool 25 RBD pool (3 replicas) Ceph CSI Admin 3. Create 2. Detect Rook operator
26. Rook/Ceph cluster Step2: Consume a Block Volume 26 RBD pool (3 replicas) Ceph CSI User 1. Create kind: PersistentVolumeClaim metadata: name: my-pvc spec: resources: requests: storage: 1Gi storageClassName: ceph-block Rook operator
27. Rook/Ceph cluster Step2: Consume a Block Volume 27 RBD pool (3 replicas) Ceph CSI User kind: PersistentVolumeClaim metadata: name: my-pvc spec: resources: requests: storage: 1Gi storageClassName: ceph-block block volume 2. Detect 3. Create PV 4. Create Rook operator
28. Object Storage 28
29. How to provision and consume RGW? • Use ObjectBucket(OB) and ObjectBucketClaim(OBC) CR ◦ Similar to PV and PVC for block and filesystem storage 29 PV PVC Block or filesystem storage RGW bucket OB OBC Create User User Create
30. Example: Consuming a Bucket 1. Create an RGW pool 2. Create a bucket 3. Consume the created bucket 30 Rook/Ceph cluster Rook operator
31. Rook/Ceph cluster Step1: Create an RGW Pool kind: CephObjectStore metadata: name: my-store spec: dataPool: replicated: size: 3 … — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rgw-bucket parameters: objectStoreName: ceph-store 31 Admin 1. Create Rook operator
32. Rook/Ceph cluster Step1: Create an RGW Pool RGW pool (3 replicas) kind: CephObjectStore metadata: name: my-store spec: dataPool: replicated: size: 3 … — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-sc parameters: objectStoreName: ceph-store 32 Admin 3. Create 2. Detect Rook operator
33. kind: ObjectBucketClaim metadata: name: my-bucket spec: bucketName: my-bucket storageClassName: ceph-store Rook/Ceph cluster Step2: Create a Bucket RGW pool (3 replicas) 33 User 1. Create Rook operator
34. Rook/Ceph cluster Step2: Create a Bucket RGW pool (3 replicas) 34 User 3. Create Bucket Secret (Access Key Secret Key) ConfigMap (URL) 4. Create 2. Detect kind: ObjectBucketClaim metadata: name: my-bucket spec: bucketName: my-bucket storageClassName: ceph-store Rook operator
35. Rook/Ceph cluster Step3: Consume the Created Bucket RGW pool (3 replicas) 35 User 1. Create Bucket apiVersion: v1 kind: Pod metadata: name: my-app spec: containers: - envFrom: - configMapRef: name: my-bucket - secretRef: name: my-bucket Secret (Access Key Secret Key) ConfigMap (URL) Rook operator
36. Rook/Ceph cluster Step3: Consume the Created Bucket RGW pool (3 replicas) 36 User Bucket apiVersion: v1 kind: Pod metadata: name: my-app spec: containers: - envFrom: - configMapRef: name: my-bucket - secretRef: name: my-bucket Secret (Access Key Secret Key) ConfigMap (URL) My-app pod 3. Access 2. Use as envvars Rook operator
37. Another Interface to Access RGW • OB and OBC are not the K8s official way • Rook supports Container Object Storage Interface (COSI) ◦ The K8s official way ◦ Similar to CSI for block and filesystem storage • COSI will replace OB and OBC in the future 37
38. Other Features 38
39. Non-K8s environment External Cluster 39 • Consume external Ceph clusters from a Kubernetes cluster Ceph cluster K8s cluster PVC, OBCs Application Pods Rook&Ceph CSI Other K8s clusters Rook/Ceph cluster
40. Remote Replications 40 Ceph Feature Custom Resource RBD mirroring CephRBDMirror CephFS mirroring CephFilesystemMirror RGW multisite CephObjectRealm Rook/Ceph cluster K8s cluster in region A K8s cluster in region B Rook/Ceph cluster kind: CephRBDMirror … kind: CephRBDMirror … RBD pool RBD pool Replicate
41. • Create PDBs for each failure domains • Only one failure domain is allowed to be down at once • e.g: when the failure domani is “node” Managed PDB Configuration 41 node0 node1 Rook Admin apiVersion: policy/v1 kind: PodDisruptionBudget spec: maxUnavailable: 1 selector: matchLabels: app: rook-ceph-osd … 1. Create OSD0 Pod OSD1 Pod 2. Drain 3. Drain (blocked)
42. Title Administration Tools • Toolbox Pod ◦ A Pod for running arbitrary Ceph commands • Kubectl rook-ceph krew plugin ◦ Running handy Ceph operations 42 Admin All Ceph features Features covered by Rook By Rook’s CRs By toolbox pod & kubect rook-ceph
43. Project Health 43
44. Title Philosophy • Support latest Ceph and K8s • Make Ceph the best storage platform for K8s! 44
45. Title Stability • Marked as stable 6 years ago • Many upstream users running in production • Many downstream deployments running in production 45
46. Title Release Cycle • Major version: Always “1” for now • Minor version: Once per 4 months • Patch version: Biweekly or on demand 46
47. Title Active Community • GitHub and slack channel • 500+ contributors to the GitHub project ◦ e.g. Clyso, Cybozu, IBM/Red Hat, and Upbound • Monthly community meeting • CNCF Graduated project 47 GitHub https://github.com/rook/rook Slack https://slack.rook.io
48. Containers and Helm charts Docker Hub, Quay.io, GitHub Container Registry (GHCR) Website and Docs https://rook.io 48 Try Rook!
49. 49 Questions?