Share
Embed
Rook: Intro and Deep Dive With Ceph
The slides of the following session at KubeCon+CloudNativeCon Japan 2025
Satoru Takeuchi PRO
June 17, 2025
Transcript
49 slides
-
Rook: Intro and Deep Dive With Ceph Satoru Takeuchi, Cybozu,
Inc. June 17th, 2025 2 -
Title Agenda • Introduction to Rook and Ceph • Block
and Filesystem Storage • Object Storage • Other Features • Project Health 3 -
Title What is Rook? • An open source K8s operator
to manage Ceph storage • For Admins ◦ Deploy, manage, upgrade Ceph cluster by CR • For Users ◦ Consume Ceph by PVC and OBC CR 5 -
Title What is Ceph? • All-in-one open source distributed storage
platform 6 Name Type RBD Block Storage CephFS Large-scale share filesystem storage RGW S3-compatible object storage CephNFS Export CephFS and S3 object as NFS -
Remote replications 7 Storage Type Feature RBD RBD mirroring CephFS
CephFS mirroring RGW RGW multisite Ceph cluster Region A Region B Ceph cluster Replicate -
Ceph’s Architecture • OSD daemons ◦ Manage data • MON
daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 8 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … … -
Ceph’s Architecture • OSD daemons ◦ Manage data • MON
daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 9 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … … -
Ceph’s Architecture • OSD daemons ◦ Manage data • MON
daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 10 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … … -
Ceph’s Architecture • OSD daemons ◦ Manage data • MON
daemons ◦ Manage cluster’s state • MGR daemons ◦ Provide additional features 11 Ceph Cluster Mons Mons MON Mons MGR Storage (e.g.RBD) OSD OSD OSD disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … … -
Title • High scalability ◦ Real example: ~1800 OSDs, over
5 PiB • High durability ◦ Replication or Erasure Coding ◦ Configurable failure domains (e.g. rack) • High availability ◦ e.g. Add/remove/replace OSDs online Ceph’s Characteristics 12 -
Title Rook’s Architecture • Rook operator ◦ Manage Rook/Ceph clusters
◦ Provision Pod for each Ceph daemons • Ceph CSI ◦ A CSI Driver for Ceph ◦ Provisions storage from Ceph 13 Rook/Ceph Cluster Mons Mons MON Pod Mons MGR Pods Storage (e.g.RBD) OSD Pod OSD Pod OSD Pod disk disk disk Storage (e.g.RBD) Network storages (e.g. RBD) Storage pool … … MGR Pod Ceph CSI manage Rook operator manage -
Example: Provisioning and expanding cluster 1. Deploy a minimum Ceph
cluster 2. Expand the cluster disk1 14 Rook operator disk0 -
kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count:
1 Step1: Deploy a Minimum Cluster 15 Admin disk0 disk1 1. Create Rook operator -
Step1: Deploy a Minimum Cluster kind: CephCluster metadata: name: my-cluster
spec: storage: storageClassdeviceSets: - count: 1 Rook/Ceph cluster (0 OSD) disk1 16 Rook Admin disk0 3. Create 2. Detect -
Step1: Deploy a Minimum Cluster kind: CephCluster metadata: name: my-cluster
spec: storage: storageClassdeviceSets: - count: 1 Rook/Ceph cluster (1 OSD) disk1 17 Admin disk0 4. Create OSD0 Pod Rook operator -
kind: CephCluster metadata: name: my-cluster spec: storage: storageClassdeviceSets: - count:
2 Step2: Expand the cluster Rook/Ceph cluster (1 OSD) disk1 18 Admin disk0 1. Update OSD0 Pod Rook operator -
Step2: Expand the cluster kind: CephCluster metadata: name: my-cluster spec:
storage: storageClassdeviceSets: - count: 2 Rook/Ceph cluster (2 OSDs) disk1 19 Admin disk0 3. Create OSD0 Pod OSD1 Pod 2. Detect Rook operator -
Supported Configurations 21 Storage Volume Mode Access Mode RBD Block,
Filesystem RWO, RWOP, ROX CephFS Filesystem RWX, RWO, ROX, RWOP CephNFS The same as above The same as above -
Title Additional Features 22 Storage Volume Expansion, snapshot, and cloning
Static provisioning QoS RBD ✅ ✅ ✅ CephFS ✅ ✅ CephNFS ✅ -
Example: Consuming a Block Volume 1. Create an RBD pool
2. Consume a block volume 23 Rook operator Rook/Ceph cluster Ceph CSI -
Rook/Ceph cluster Step1: Create an RBD Pool kind: CephBlockPool metadata:
name: my-pool spec: replicated: size: 3 — kind: StorageClass metadata: name: ceph-block parameters: pool: my-pool 24 Ceph CSI Admin 1. Create Rook operator -
Rook/Ceph cluster Step1: Create an RBD Pool kind: CephBlockPool metadata:
name: my-pool spec: replicated: size: 3 — kind: StorageClass metadata: name: ceph-block parameters: pool: my-pool 25 RBD pool (3 replicas) Ceph CSI Admin 3. Create 2. Detect Rook operator -
Rook/Ceph cluster Step2: Consume a Block Volume 26 RBD pool
(3 replicas) Ceph CSI User 1. Create kind: PersistentVolumeClaim metadata: name: my-pvc spec: resources: requests: storage: 1Gi storageClassName: ceph-block Rook operator -
Rook/Ceph cluster Step2: Consume a Block Volume 27 RBD pool
(3 replicas) Ceph CSI User kind: PersistentVolumeClaim metadata: name: my-pvc spec: resources: requests: storage: 1Gi storageClassName: ceph-block block volume 2. Detect 3. Create PV 4. Create Rook operator -
How to provision and consume RGW? • Use ObjectBucket(OB) and
ObjectBucketClaim(OBC) CR ◦ Similar to PV and PVC for block and filesystem storage 29 PV PVC Block or filesystem storage RGW bucket OB OBC Create User User Create -
Example: Consuming a Bucket 1. Create an RGW pool 2.
Create a bucket 3. Consume the created bucket 30 Rook/Ceph cluster Rook operator -
Rook/Ceph cluster Step1: Create an RGW Pool kind: CephObjectStore metadata:
name: my-store spec: dataPool: replicated: size: 3 … — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rgw-bucket parameters: objectStoreName: ceph-store 31 Admin 1. Create Rook operator -
Rook/Ceph cluster Step1: Create an RGW Pool RGW pool (3
replicas) kind: CephObjectStore metadata: name: my-store spec: dataPool: replicated: size: 3 … — apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-sc parameters: objectStoreName: ceph-store 32 Admin 3. Create 2. Detect Rook operator -
kind: ObjectBucketClaim metadata: name: my-bucket spec: bucketName: my-bucket storageClassName: ceph-store
Rook/Ceph cluster Step2: Create a Bucket RGW pool (3 replicas) 33 User 1. Create Rook operator -
Rook/Ceph cluster Step2: Create a Bucket RGW pool (3 replicas)
34 User 3. Create Bucket Secret (Access Key Secret Key) ConfigMap (URL) 4. Create 2. Detect kind: ObjectBucketClaim metadata: name: my-bucket spec: bucketName: my-bucket storageClassName: ceph-store Rook operator -
Rook/Ceph cluster Step3: Consume the Created Bucket RGW pool (3
replicas) 35 User 1. Create Bucket apiVersion: v1 kind: Pod metadata: name: my-app spec: containers: - envFrom: - configMapRef: name: my-bucket - secretRef: name: my-bucket Secret (Access Key Secret Key) ConfigMap (URL) Rook operator -
Rook/Ceph cluster Step3: Consume the Created Bucket RGW pool (3
replicas) 36 User Bucket apiVersion: v1 kind: Pod metadata: name: my-app spec: containers: - envFrom: - configMapRef: name: my-bucket - secretRef: name: my-bucket Secret (Access Key Secret Key) ConfigMap (URL) My-app pod 3. Access 2. Use as envvars Rook operator -
Another Interface to Access RGW • OB and OBC are
not the K8s official way • Rook supports Container Object Storage Interface (COSI) ◦ The K8s official way ◦ Similar to CSI for block and filesystem storage • COSI will replace OB and OBC in the future 37 -
Non-K8s environment External Cluster 39 • Consume external Ceph clusters
from a Kubernetes cluster Ceph cluster K8s cluster PVC, OBCs Application Pods Rook&Ceph CSI Other K8s clusters Rook/Ceph cluster -
Remote Replications 40 Ceph Feature Custom Resource RBD mirroring CephRBDMirror
CephFS mirroring CephFilesystemMirror RGW multisite CephObjectRealm Rook/Ceph cluster K8s cluster in region A K8s cluster in region B Rook/Ceph cluster kind: CephRBDMirror … kind: CephRBDMirror … RBD pool RBD pool Replicate -
• Create PDBs for each failure domains • Only one
failure domain is allowed to be down at once • e.g: when the failure domani is “node” Managed PDB Configuration 41 node0 node1 Rook Admin apiVersion: policy/v1 kind: PodDisruptionBudget spec: maxUnavailable: 1 selector: matchLabels: app: rook-ceph-osd … 1. Create OSD0 Pod OSD1 Pod 2. Drain 3. Drain (blocked) -
Title Administration Tools • Toolbox Pod ◦ A Pod for
running arbitrary Ceph commands • Kubectl rook-ceph krew plugin ◦ Running handy Ceph operations 42 Admin All Ceph features Features covered by Rook By Rook’s CRs By toolbox pod & kubect rook-ceph -
Title Philosophy • Support latest Ceph and K8s • Make
Ceph the best storage platform for K8s! 44 -
Title Stability • Marked as stable 6 years ago •
Many upstream users running in production • Many downstream deployments running in production 45 -
Title Release Cycle • Major version: Always “1” for now
• Minor version: Once per 4 months • Patch version: Biweekly or on demand 46 -
Title Active Community • GitHub and slack channel • 500+
contributors to the GitHub project ◦ e.g. Clyso, Cybozu, IBM/Red Hat, and Upbound • Monthly community meeting • CNCF Graduated project 47 GitHub https://github.com/rook/rook Slack https://slack.rook.io -
Containers and Helm charts Docker Hub, Quay.io, GitHub Container Registry
(GHCR) Website and Docs https://rook.io 48 Try Rook!