Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Corporation Japan July 30th, 2020 Takuya Niita Oracle Cloud
Hangout Cafe – Season 3 #1 Cloud Native

Copyright © 2020, Oracle and/or its affiliates. All rights reserved
Oracle Java Oracle Corporation 2

3 Copyright © 2020, Oracle and/or its affiliates. All rights
reserved 3 2 1 Rook/Cephから見るCloud Native Storage Kubernetesの世界におけるストレージ管理 Agenda デモ(Rook/Ceph)

reserved • (@takuya_0301) • • SIer • • Oracle 1 3 • Cloud Native 1 • Kubernetes 1 • 4 CKA(Certified Kubernetes Administrator) •

Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Kubernetes
Oracle Corporation Japan July 30th, 2020 Takuya Niita

reserved Cloud Native Storage • • • • API Self-Managed • • RBAC • • Kubernetes – Cloud Native Storage – Kubernetes 参考: StorageOS Vision for Cloud Native Storage for Today’s Modern IT https://storageos.com/storageos-vision-cloud-native-storage-todays-modern

Software Defined Storage 7 Copyright © 2020, Oracle and/or its
affiliates. All rights reserved Software Defined Storage(SDS) • SNIA (2015 ) • / (Self-Managed) • (API) • - • - ( ) • - Software Defined Storage (SDS) API( / / ) API( ) #1 #2 #3 https://www.snia.org/sites/default/files/SNIA_Software_Defined_Storage_%20White_Paper_v1.pdf

reserved • • API • • / • Kubernetes – – Kubernetes Kubernetes

reserved “ ” • • • • “ ” • • Volumes Kubernetes – –

reserved Volumes • Volume Pod • NFS/iSCSI • Volume Pod Kubernetes – Volumes – Storage #A Pod Volume Kubernetes ”/data” ”/test-pd” hostPath manifest Storage #A Pod Volume Kubernetes

reserved Volumes Cloud Native Storage • • Kubernetes • StorageClass • PersistentVolume • PersistentVolumeClaim Kubernetes – –

reserved StorageClass(SC) • • Provisioner - Local Volume - - (Ceph ) - (iSCSI/NFS) • Provisioner - Rook-Ceph Rook-Ceph API ( ) - Provisioner Kubernetes – StorageClass – AWS EBS kubernetes.io/aws-ebs Ceph RBD kubernetes.io/rbd Local kubernetes.io/no-provisioner

reserved oracle.com/oci • Oracle Container Engine for Kubernetes(OKE) • OCI Block Volume oracle.com/oci-fss • OCI File Storage • manifest apply OCI Provisioner :https://kubernetes.io/docs/concepts/storage/storage-classes/

reserved PersistentVolume(PV) • • • - ReadWriteOnce - ReadWriteMany - ReadOnlyMany • StorageClass - StorageClassName - StorageClass ”oci” Kubernetes – PersistentVolume – StorageClass

reserved PersistentVolumeClaim(PVC) • • • - Persistent Volume(PV) • - Persistent Volume(PV) Kubernetes – PersistentVolumeClaim – StorageClass

reserved • SC PVC / • • PV/PVC • / Kubernetes – SC/PV/PVC – Storage 5Gi PVC 2Gi PV 2Gi SC Pod Volume Storage 5Gi PVC 2Gi SC Pod Volume PV 2Gi

reserved Kubernetes Kubernetes ( ) Kubernetes ( ) 3 • in-tree plugin • FlexVolume • CSI(Container Storage Interface) Kubernetes

reserved in-tree plugin( ) • Kubernetes • https://github.com/kubernetes/kubernete s/tree/master/pkg/volume • iSCSI Plugin • Kubernetes • Kubernetes • • Kubernetes v1.14 CSI ( ) • Kubernetes v1.17 Kubernetes – in-tree plugin –

reserved FlexVolume • out-of-tree plugin(Kubernetes ) • - root - /usr/libexec/kubernetes/kubelet- plugins/volume/exec/ - (API) • kubelet Volume / / • Flex Volume in-tree plugin • Kubernetes Kubernetes – FlexVolume – kubelet Storage Driver / Flex Volume Driver

reserved CSI(Container Storage Interface) • Kubernetes Mesos Docker Cloud Foundry API • • out-of-tree plugin • Kubernetes v1.9 v1.13 GA • CSI • in-tree plugin CSI( ) Kubernetes – CSI – https://github.com/kubernetes/community/blob/master/contributors/design- proposals/storage/container-storage-interface.md#recommended-mechanism-for- deploying-csi-drivers-on-kubernetes • ( ) • Kubernetes

Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Rook/Ceph
Cloud Native Storage Oracle Corporation Japan July 30th, 2020 Takuya Niita

reserved Rook • https://rook.io/ • Cloud Native Storage • Cloud Native Storage for Kubernetes • v1.3.8(2020/7 ) • CNCF • Incubating Stage(2020/7 ) • Ceph Cloud Native • v0.8 Ceph Cloud Native Storage – Rook –

reserved • Kubernetes • Kubernetes Operator • Kubernetes • Operator • Prometheus • • Stable - Ceph - EdgeFS( Closed Source) • alpha - Cassandra/CockroachDB/NFS/YugabyteDB Rook

reserved Ceph • https://ceph.io/ • 2003 Sage Weil UCSC • 2006 OSS • 2012 Inktank 2014 Red Hat • v15.2.4(2020/7 ) • C++ • CSI(Container Storage Interface) • cephalopod Ceph (cephalopod) • v15.x Octopus Cloud Native Storage – Ceph –

reserved • RADOS(Reliable Autonomic Distributed Object Store) • • RADOS - S3 SWIFT ( S3 SWIFT ( )) • Ceph Worker Node • CSI(Container Storage Interface) • OpenStack • • • OSD(Object Storage Device) • • Ceph Ceph Contributions

reserved (RADOSGW/LIBRADOS) • LIBRADOS • RADOS • RADOSGW(Ceph Object Gateway) • AmazonS3/Swift (CephFS) • NFS HDFS Shared Storage (RBD) • Ceph :https://docs.ceph.com/docs/master/architecture/

reserved OSD(Object Storage Device) • ( ) • OSD ( ) • Primary OSD Primary OSD Secondary OSD - OSD Placement Group(PG) - PG OSD Primary OSD Secondary OSD - • OSD Placement Group(PG) CRUSH ( ) Ceph – OSD – Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 OSD#1 OSD#2 OSD#3 OSD#4 Ceph Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD

reserved Pool • • Pool - (RADOSGW/CephFS/RBD) - Placement Group(PG) - Placement Group(PG) • OSD Pool • 1 OSD PG CRUSH • PG OSD Ceph – Pool/PG/CRUSH – 1( 2) 2( 3) PG1 OSD#1 OSD#3 Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 PG2 OSD#4 OSD#2 OSD#5 OSD#1 OSD#2 OSD#3 OSD#4 CRUSH CephFS Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD RBD

reserved • Ceph Ceph ( ) • • • (split brain ) 3 • • (Paxos) / Ceph – – Ceph Storage Cluster #1 Ceph Storage Cluster #2 Paxos Ceph OSD#1 OSD#2 OSD#3 OSD#4 Ceph

reserved • v12.x • (Prometheus/Zabbix ) • Ceph Dashboard Ceph – – Ceph Storage Cluster #1 (Prometheus/Zabbix) Ceph OSD#1 OSD#2

reserved Ceph – – 1( 2) PG1 PG2 2( 3) Ceph OSD#1 OSD#2 Ceph Storage Cluster #1 Ceph Storage Cluster #2 CRUSH Ceph Ceph Storage Cluster #3 CRUSH PG/OSD Primary OSD OSD#3 OSD#4 OSD#5 Paxos Paxos Primary OSD Secondary OSD

reserved • OSD Ceph • OSD • OSD Ceph OSD • OSD OSD • OSD Ceph ( / )

reserved Rook • Operator/(Agent) • Rook - Agent FlexVolume Ceph • MON Ceph • OSD’s Ceph OSD • MGR Ceph • MDS/RGW:RADOS • MDS CephFS RGW RADOSGW Rook-Ceph − – ” Many of the Ceph concepts like placement groups and crush maps are hidden so you don’t have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. ” • Ceph (PG CRUSH map ) • UX https://rook.io/docs/rook/v1.3/ceph-storage.html

reserved Rook-Ceph CSI • csi-cephfsplugin • CephFS CSI • csi-rbdplugin • RBD CSI • csi-cephfsplugin-provisioner/csi-rbdplugin-provisioner • kube-apiserver CephFS RBD • Rook v1.1 CSI Driver • FlexVolume Driver deprecate Rook-Ceph – CSI(Container Storage Interface) – Rook-Ceph CSI RBD provisioner <rook operator namespace>.rbd.csi.ceph.com

reserved Rook • Rook (rook-operator) rook-discover Ceph • OSD( ) Rook-Ceph CSI • CephFS RBD CSI provisioner Rook-Ceph − − kubelet Rook-Ceph CSI rook-operator rook-discover rook-discover rook-discover ceph-osd ceph-osd ceph-osd ceph- mon ceph- mon ceph- mon ceph-mgr provisioner csi-rbdplugin csi- cephfsplugin kubelet kubelet provisioner csi-rbdplugin csi- cephfsplugin provisioner csi-rbdplugin csi- cephfsplugin deployment daemonset

reserved • Rook Operator • Rook-Ceph Ceph • api-server Rook Operator • CSI Application Attach/Detach Rook-Ceph – Kubernetes – kubelet Worker Node Ceph Rook- Ceph CSI apiserver Application Rook Operator Master Node etcd Attach/Detach

reserved OSD on PVC • PV/PVC OSD Pod • PVC(Persistent Volume Claim) OSD • Rook 1.1 OSD on device • OSD • manifest • Kubernetes Rook-Ceph – Rook-Ceph OSD – PVC

reserved Rook-Ceph • Kubernetes( ) Ceph • Ceph SDS(Software Defined Storage) • Ceph Rook Rook • Ceph stable Operator • Cassandra CockroachDB YugabyteDB Rook Rook/Ceph

Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle
Corporation Japan July 30th, 2020 Takuya Niita

reserved Rook-Ceph • https://github.com/rook/rook/tree/master/cl uster/examples/kubernetes/ceph • 3 manifest • common.yaml • operator.yaml • cluster.yaml 1 – Rook-Ceph ( ) – Block Volume 100GB Ceph Oracle Container Engine for Kubernetes Block Volume 100GB Block Volume 100GB /

reserved common.yaml • Namespace • ”rook-ceph” • Custom Resource Definition(CRD) • cephclusters.ceph.rook.io • cephrbdmirrors.ceph.rook.io • cephfilesystems.ceph.rook.io • 14 CRD • RBAC • ClusterRoleBinding/ClusterRole/Role/Role Binding/ ServiceAccount 1 – Rook-Ceph (common.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/common.yaml ※ 2000 Namespace CRD

reserved operator.yaml • ConfigMap • Rook Operator • Deployment • Rook Operator - Ceph Rook Operator - rook-discover (FlexVolume )rook-agent 1 – Rook-Ceph (operator.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/operator.yaml ConfigMap Deployment

reserved cluster.yaml • CephCluster • OSD on device Ceph • Ceph - 1 – Rook-Ceph (cluster.yaml) – https://raw.githubusercontent.com/rook/rook /master/cluster/examples/kubernetes/ceph/ cluster.yaml CephCluster ( )

reserved Ceph • • - • Grafana - Prometheus • • • • OSD 1 – Rook-Ceph (Ceph ) – Ceph Ceph

reserved • Rook-Ceph • manifest • 10 • Ceph • 1 – Rook-Ceph (Wrap up) –

reserved Oracle Container Engine for Kubernetes(OKE) • 1 IOPS • ( ) Standard VM + Block Volume • Standard VM with Rook-Ceph(Block Volume Ceph ) Worker Node(Compute Instance) • Standard VM • VM.Standard.2.8/OCPU :8 OCPU(16 vCPU)/ :120GB/ :8.2Gbps Rook-Ceph • Rook:v1.3.7( v1.3.8) • Ceph:v15.2.4( ) 2 – Rook-Ceph ( ) –

reserved • Kubestone(benchmarking Operator that can evaluate the performance of Kubernetes installations) • https://kubestone.io/en/latest/ • fio Job • - Read/ Write ◦ iodepth=4/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth=64/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth 1 • Volume Size(Block Volume) • Job Pod /data PV 300Gi • Rook-Ceph OSD • 3 Woker Node 1OSD • ( ) 3 → 1 2 – Rook-Ceph ( ) –

reserved 2 – Rook-Ceph ( 1) – Block Volume Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 300Gi fio / Kubestone Job 3

reserved 2 – Rook-Ceph ( 2) – Block Volume Block Volume 100Gi Ceph (Ceph RBD) Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 100Gi Block Volume 100Gi fio Kubestone Job 3 300Gi

reserved • Read Rook-Ceph IOPS • - OSD - ( ) • Write Rook-Ceph IOPS • - (Rook-Ceph ) 2 – Rook-Ceph ( ) – iodepth=4 iodepth=64 1.49 1.00 0 0.5 1 1.5 2 Read Rook-Ceph Non Rook-Ceph 0.16 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph 1.21 1.00 0 0.5 1 1.5 Read Rook-Ceph Non Rook-Ceph 0.23 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph

reserved • Rook-Ceph Read • Write Ceph 2 – Rook-Ceph (Wrap up) –

Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle
Corporation Japan July 30th, 2020 Takuya Niita

53 Cloud Native Storage • Kubernetes • SC/PV/PVC • CSI
Rook-Ceph • ( ) Rook • Ceph • Kubernetes Rook-Ceph Rook-Ceph • Ceph Kubernetes Ceph • Rook Operator Copyright © 2020, Oracle and/or its affiliates. All rights reserved

reserved Kuberentes (Storage/Volume) https://kubernetes.io/docs/concepts/storage/ Rook Advent Calendar 2019 https://qiita.com/advent-calendar/2019/rook Rook https://rook.io/docs/rook/v1.3/ Rook(GitHub) https://github.com/rook/rook Ceph https://docs.ceph.com/docs/master/ Ceph(GitHub) https://github.com/ceph & Special Thanks!!

Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

More Decks by oracle4engineer

Other Decks in Technology

Featured

Transcript