Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Cloud Hangout Cafe(おちゃかふぇ)のセッションスライドです。
1. Kubernetesの世界におけるストレージ管理
2. Rook/Cephから見るCloud Native Storage
3. デモ(Rook/Ceph)

(セッションの録画)
https://youtu.be/n95_om_bOiI

(イベントページ)
https://ochacafe.connpass.com/event/178405/

oracle4engineer

July 30, 2020
Tweet

More Decks by oracle4engineer

Other Decks in Technology

Transcript

  1. Oracle Corporation Japan July 30th, 2020 Takuya Niita Oracle Cloud

    Hangout Cafe – Season 3 #1 Cloud Native
  2. 3 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved 3 2 1 Rook/Cephから見るCloud Native Storage Kubernetesの世界におけるストレージ管理 Agenda デモ(Rook/Ceph)
  3. 4 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • (@takuya_0301) • • SIer • • Oracle 1 3 • Cloud Native 1 • Kubernetes 1 • 4 CKA(Certified Kubernetes Administrator) •
  4. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Kubernetes

    Oracle Corporation Japan July 30th, 2020 Takuya Niita
  5. 6 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Cloud Native Storage • • • • API Self-Managed • • RBAC • • Kubernetes – Cloud Native Storage – Kubernetes 参考: StorageOS Vision for Cloud Native Storage for Today’s Modern IT https://storageos.com/storageos-vision-cloud-native-storage-todays-modern
  6. Software Defined Storage 7 Copyright © 2020, Oracle and/or its

    affiliates. All rights reserved Software Defined Storage(SDS) • SNIA (2015 ) • / (Self-Managed) • (API) • - • - ( ) • - Software Defined Storage (SDS) API( / / ) API( ) #1 #2 #3 https://www.snia.org/sites/default/files/SNIA_Software_Defined_Storage_%20White_Paper_v1.pdf
  7. 8 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • • API • • / • Kubernetes – – Kubernetes Kubernetes
  8. 9 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved “ ” • • • • “ ” • • Volumes Kubernetes – –
  9. 10 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Volumes • Volume Pod • NFS/iSCSI • Volume Pod Kubernetes – Volumes – Storage #A Pod Volume Kubernetes ”/data” ”/test-pd” hostPath manifest Storage #A Pod Volume Kubernetes
  10. 11 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Volumes Cloud Native Storage • • Kubernetes • StorageClass • PersistentVolume • PersistentVolumeClaim Kubernetes – –
  11. 12 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved StorageClass(SC) • • Provisioner - Local Volume - - (Ceph ) - (iSCSI/NFS) • Provisioner - Rook-Ceph Rook-Ceph API ( ) - Provisioner Kubernetes – StorageClass – AWS EBS kubernetes.io/aws-ebs Ceph RBD kubernetes.io/rbd Local kubernetes.io/no-provisioner
  12. 13 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved oracle.com/oci • Oracle Container Engine for Kubernetes(OKE) • OCI Block Volume oracle.com/oci-fss • OCI File Storage • manifest apply OCI Provisioner :https://kubernetes.io/docs/concepts/storage/storage-classes/
  13. 14 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved PersistentVolume(PV) • • • - ReadWriteOnce - ReadWriteMany - ReadOnlyMany • StorageClass - StorageClassName - StorageClass ”oci” Kubernetes – PersistentVolume – StorageClass
  14. 15 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved PersistentVolumeClaim(PVC) • • • - Persistent Volume(PV) • - Persistent Volume(PV) Kubernetes – PersistentVolumeClaim – StorageClass
  15. 16 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • SC PVC / • • PV/PVC • / Kubernetes – SC/PV/PVC – Storage 5Gi PVC 2Gi PV 2Gi SC Pod Volume Storage 5Gi PVC 2Gi SC Pod Volume PV 2Gi
  16. 17 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Kubernetes Kubernetes ( ) Kubernetes ( ) 3 • in-tree plugin • FlexVolume • CSI(Container Storage Interface) Kubernetes
  17. 18 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved in-tree plugin( ) • Kubernetes • https://github.com/kubernetes/kubernete s/tree/master/pkg/volume • iSCSI Plugin • Kubernetes • Kubernetes • • Kubernetes v1.14 CSI ( ) • Kubernetes v1.17 Kubernetes – in-tree plugin –
  18. 19 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved FlexVolume • out-of-tree plugin(Kubernetes ) • - root - /usr/libexec/kubernetes/kubelet- plugins/volume/exec/ - (API) • kubelet Volume / / • Flex Volume in-tree plugin • Kubernetes Kubernetes – FlexVolume – kubelet Storage Driver / Flex Volume Driver
  19. 20 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved CSI(Container Storage Interface) • Kubernetes Mesos Docker Cloud Foundry API • • out-of-tree plugin • Kubernetes v1.9 v1.13 GA • CSI • in-tree plugin CSI( ) Kubernetes – CSI – https://github.com/kubernetes/community/blob/master/contributors/design- proposals/storage/container-storage-interface.md#recommended-mechanism-for- deploying-csi-drivers-on-kubernetes • ( ) • Kubernetes
  20. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Rook/Ceph

    Cloud Native Storage Oracle Corporation Japan July 30th, 2020 Takuya Niita
  21. 22 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook • https://rook.io/ • Cloud Native Storage • Cloud Native Storage for Kubernetes • v1.3.8(2020/7 ) • CNCF • Incubating Stage(2020/7 ) • Ceph Cloud Native • v0.8 Ceph Cloud Native Storage – Rook –
  22. 23 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Kubernetes • Kubernetes Operator • Kubernetes • Operator • Prometheus • • Stable - Ceph - EdgeFS( Closed Source) • alpha - Cassandra/CockroachDB/NFS/YugabyteDB Rook
  23. 24 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Ceph • https://ceph.io/ • 2003 Sage Weil UCSC • 2006 OSS • 2012 Inktank 2014 Red Hat • v15.2.4(2020/7 ) • C++ • CSI(Container Storage Interface) • cephalopod Ceph (cephalopod) • v15.x Octopus Cloud Native Storage – Ceph –
  24. 25 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • RADOS(Reliable Autonomic Distributed Object Store) • • RADOS - S3 SWIFT ( S3 SWIFT ( )) • Ceph Worker Node • CSI(Container Storage Interface) • OpenStack • • • OSD(Object Storage Device) • • Ceph Ceph Contributions
  25. 26 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved (RADOSGW/LIBRADOS) • LIBRADOS • RADOS • RADOSGW(Ceph Object Gateway) • AmazonS3/Swift (CephFS) • NFS HDFS Shared Storage (RBD) • Ceph :https://docs.ceph.com/docs/master/architecture/
  26. 27 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved OSD(Object Storage Device) • ( ) • OSD ( ) • Primary OSD Primary OSD Secondary OSD - OSD Placement Group(PG) - PG OSD Primary OSD Secondary OSD - • OSD Placement Group(PG) CRUSH ( ) Ceph – OSD – Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 OSD#1 OSD#2 OSD#3 OSD#4 Ceph Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD
  27. 28 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Pool • • Pool - (RADOSGW/CephFS/RBD) - Placement Group(PG) - Placement Group(PG) • OSD Pool • 1 OSD PG CRUSH • PG OSD Ceph – Pool/PG/CRUSH – 1( 2) 2( 3) PG1 OSD#1 OSD#3 Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 PG2 OSD#4 OSD#2 OSD#5 OSD#1 OSD#2 OSD#3 OSD#4 CRUSH CephFS Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD RBD
  28. 29 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Ceph Ceph ( ) • • • (split brain ) 3 • • (Paxos) / Ceph – – Ceph Storage Cluster #1 Ceph Storage Cluster #2 Paxos Ceph OSD#1 OSD#2 OSD#3 OSD#4 Ceph
  29. 30 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • v12.x • (Prometheus/Zabbix ) • Ceph Dashboard Ceph – – Ceph Storage Cluster #1 (Prometheus/Zabbix) Ceph OSD#1 OSD#2
  30. 31 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Ceph – – 1( 2) PG1 PG2 2( 3) Ceph OSD#1 OSD#2 Ceph Storage Cluster #1 Ceph Storage Cluster #2 CRUSH Ceph Ceph Storage Cluster #3 CRUSH PG/OSD Primary OSD OSD#3 OSD#4 OSD#5 Paxos Paxos Primary OSD Secondary OSD
  31. 32 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • OSD Ceph • OSD • OSD Ceph OSD • OSD OSD • OSD Ceph ( / )
  32. 33 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook • Operator/(Agent) • Rook - Agent FlexVolume Ceph • MON Ceph • OSD’s Ceph OSD • MGR Ceph • MDS/RGW:RADOS • MDS CephFS RGW RADOSGW Rook-Ceph − – ” Many of the Ceph concepts like placement groups and crush maps are hidden so you don’t have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. ” • Ceph (PG CRUSH map ) • UX https://rook.io/docs/rook/v1.3/ceph-storage.html
  33. 34 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook-Ceph CSI • csi-cephfsplugin • CephFS CSI • csi-rbdplugin • RBD CSI • csi-cephfsplugin-provisioner/csi-rbdplugin-provisioner • kube-apiserver CephFS RBD • Rook v1.1 CSI Driver • FlexVolume Driver deprecate Rook-Ceph – CSI(Container Storage Interface) – Rook-Ceph CSI RBD provisioner <rook operator namespace>.rbd.csi.ceph.com
  34. 35 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook • Rook (rook-operator) rook-discover Ceph • OSD( ) Rook-Ceph CSI • CephFS RBD CSI provisioner Rook-Ceph − − kubelet Rook-Ceph CSI rook-operator rook-discover rook-discover rook-discover ceph-osd ceph-osd ceph-osd ceph- mon ceph- mon ceph- mon ceph-mgr provisioner csi-rbdplugin csi- cephfsplugin kubelet kubelet provisioner csi-rbdplugin csi- cephfsplugin provisioner csi-rbdplugin csi- cephfsplugin deployment daemonset
  35. 36 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Rook Operator • Rook-Ceph Ceph • api-server Rook Operator • CSI Application Attach/Detach Rook-Ceph – Kubernetes – kubelet Worker Node Ceph Rook- Ceph CSI api- server Application Rook Operator Master Node etcd Attach/Detach
  36. 37 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved OSD on PVC • PV/PVC OSD Pod • PVC(Persistent Volume Claim) OSD • Rook 1.1 OSD on device • OSD • manifest • Kubernetes Rook-Ceph – Rook-Ceph OSD – PVC
  37. 38 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook-Ceph • Kubernetes( ) Ceph • Ceph SDS(Software Defined Storage) • Ceph Rook Rook • Ceph stable Operator • Cassandra CockroachDB YugabyteDB Rook Rook/Ceph
  38. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle

    Corporation Japan July 30th, 2020 Takuya Niita
  39. 40 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook-Ceph • https://github.com/rook/rook/tree/master/cl uster/examples/kubernetes/ceph • 3 manifest • common.yaml • operator.yaml • cluster.yaml 1 – Rook-Ceph ( ) – Block Volume 100GB Ceph Oracle Container Engine for Kubernetes Block Volume 100GB Block Volume 100GB /
  40. 41 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved common.yaml • Namespace • ”rook-ceph” • Custom Resource Definition(CRD) • cephclusters.ceph.rook.io • cephrbdmirrors.ceph.rook.io • cephfilesystems.ceph.rook.io • 14 CRD • RBAC • ClusterRoleBinding/ClusterRole/Role/Role Binding/ ServiceAccount 1 – Rook-Ceph (common.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/common.yaml ※ 2000 Namespace CRD
  41. 42 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved operator.yaml • ConfigMap • Rook Operator • Deployment • Rook Operator - Ceph Rook Operator - rook-discover (FlexVolume )rook-agent 1 – Rook-Ceph (operator.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/operator.yaml ConfigMap Deployment
  42. 43 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved cluster.yaml • CephCluster • OSD on device Ceph • Ceph - 1 – Rook-Ceph (cluster.yaml) – https://raw.githubusercontent.com/rook/rook /master/cluster/examples/kubernetes/ceph/ cluster.yaml CephCluster ( )
  43. 44 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Ceph • • - • Grafana - Prometheus • • • • OSD 1 – Rook-Ceph (Ceph ) – Ceph Ceph
  44. 45 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Rook-Ceph • manifest • 10 • Ceph • 1 – Rook-Ceph (Wrap up) –
  45. 46 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Oracle Container Engine for Kubernetes(OKE) • 1 IOPS • ( ) Standard VM + Block Volume • Standard VM with Rook-Ceph(Block Volume Ceph ) Worker Node(Compute Instance) • Standard VM • VM.Standard.2.8/OCPU :8 OCPU(16 vCPU)/ :120GB/ :8.2Gbps Rook-Ceph • Rook:v1.3.7( v1.3.8) • Ceph:v15.2.4( ) 2 – Rook-Ceph ( ) –
  46. 47 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Kubestone(benchmarking Operator that can evaluate the performance of Kubernetes installations) • https://kubestone.io/en/latest/ • fio Job • - Read/ Write ◦ iodepth=4/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth=64/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth 1 • Volume Size(Block Volume) • Job Pod /data PV 300Gi • Rook-Ceph OSD • 3 Woker Node 1OSD • ( ) 3 → 1 2 – Rook-Ceph ( ) –
  47. 48 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved 2 – Rook-Ceph ( 1) – Block Volume Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 300Gi fio / Kubestone Job 3
  48. 49 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved 2 – Rook-Ceph ( 2) – Block Volume Block Volume 100Gi Ceph (Ceph RBD) Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 100Gi Block Volume 100Gi fio Kubestone Job 3 300Gi
  49. 50 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Read Rook-Ceph IOPS • - OSD - ( ) • Write Rook-Ceph IOPS • - (Rook-Ceph ) 2 – Rook-Ceph ( ) – iodepth=4 iodepth=64 1.49 1.00 0 0.5 1 1.5 2 Read Rook-Ceph Non Rook-Ceph 0.16 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph 1.21 1.00 0 0.5 1 1.5 Read Rook-Ceph Non Rook-Ceph 0.23 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph
  50. 51 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Rook-Ceph Read • Write Ceph 2 – Rook-Ceph (Wrap up) –
  51. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle

    Corporation Japan July 30th, 2020 Takuya Niita
  52. 53 Cloud Native Storage • Kubernetes • SC/PV/PVC • CSI

    Rook-Ceph • ( ) Rook • Ceph • Kubernetes Rook-Ceph Rook-Ceph • Ceph Kubernetes Ceph • Rook Operator Copyright © 2020, Oracle and/or its affiliates. All rights reserved
  53. 54 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Kuberentes (Storage/Volume) https://kubernetes.io/docs/concepts/storage/ Rook Advent Calendar 2019 https://qiita.com/advent-calendar/2019/rook Rook https://rook.io/docs/rook/v1.3/ Rook(GitHub) https://github.com/rook/rook Ceph https://docs.ceph.com/docs/master/ Ceph(GitHub) https://github.com/ceph & Special Thanks!!