Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Cloud Hangout Cafe(おちゃかふぇ)のセッションスライドです。
1. Kubernetesの世界におけるストレージ管理
2. Rook/Cephから見るCloud Native Storage
3. デモ(Rook/Ceph)

(セッションの録画)
https://youtu.be/n95_om_bOiI

(イベントページ)
https://ochacafe.connpass.com/event/178405/

Avatar for oracle4engineer

oracle4engineer PRO

July 30, 2020
Tweet

More Decks by oracle4engineer

Other Decks in Technology

Transcript

  1. Oracle Corporation Japan July 30th, 2020 Takuya Niita Oracle Cloud

    Hangout Cafe – Season 3 #1 Cloud Native
  2. 3 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved 3 2 1 Rook/Cephから見るCloud Native Storage Kubernetesの世界におけるストレージ管理 Agenda デモ(Rook/Ceph)
  3. 4 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • (@takuya_0301) • • SIer • • Oracle 1 3 • Cloud Native 1 • Kubernetes 1 • 4 CKA(Certified Kubernetes Administrator) •
  4. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Kubernetes

    Oracle Corporation Japan July 30th, 2020 Takuya Niita
  5. 6 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Cloud Native Storage • • • • API Self-Managed • • RBAC • • Kubernetes – Cloud Native Storage – Kubernetes 参考: StorageOS Vision for Cloud Native Storage for Today’s Modern IT https://storageos.com/storageos-vision-cloud-native-storage-todays-modern
  6. Software Defined Storage 7 Copyright © 2020, Oracle and/or its

    affiliates. All rights reserved Software Defined Storage(SDS) • SNIA (2015 ) • / (Self-Managed) • (API) • - • - ( ) • - Software Defined Storage (SDS) API( / / ) API( ) #1 #2 #3 https://www.snia.org/sites/default/files/SNIA_Software_Defined_Storage_%20White_Paper_v1.pdf
  7. 8 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • • API • • / • Kubernetes – – Kubernetes Kubernetes
  8. 9 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved “ ” • • • • “ ” • • Volumes Kubernetes – –
  9. 10 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Volumes • Volume Pod • NFS/iSCSI • Volume Pod Kubernetes – Volumes – Storage #A Pod Volume Kubernetes ”/data” ”/test-pd” hostPath manifest Storage #A Pod Volume Kubernetes
  10. 11 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Volumes Cloud Native Storage • • Kubernetes • StorageClass • PersistentVolume • PersistentVolumeClaim Kubernetes – –
  11. 12 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved StorageClass(SC) • • Provisioner - Local Volume - - (Ceph ) - (iSCSI/NFS) • Provisioner - Rook-Ceph Rook-Ceph API ( ) - Provisioner Kubernetes – StorageClass – AWS EBS kubernetes.io/aws-ebs Ceph RBD kubernetes.io/rbd Local kubernetes.io/no-provisioner
  12. 13 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved oracle.com/oci • Oracle Container Engine for Kubernetes(OKE) • OCI Block Volume oracle.com/oci-fss • OCI File Storage • manifest apply OCI Provisioner :https://kubernetes.io/docs/concepts/storage/storage-classes/
  13. 14 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved PersistentVolume(PV) • • • - ReadWriteOnce - ReadWriteMany - ReadOnlyMany • StorageClass - StorageClassName - StorageClass ”oci” Kubernetes – PersistentVolume – StorageClass
  14. 15 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved PersistentVolumeClaim(PVC) • • • - Persistent Volume(PV) • - Persistent Volume(PV) Kubernetes – PersistentVolumeClaim – StorageClass
  15. 16 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • SC PVC / • • PV/PVC • / Kubernetes – SC/PV/PVC – Storage 5Gi PVC 2Gi PV 2Gi SC Pod Volume Storage 5Gi PVC 2Gi SC Pod Volume PV 2Gi
  16. 17 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Kubernetes Kubernetes ( ) Kubernetes ( ) 3 • in-tree plugin • FlexVolume • CSI(Container Storage Interface) Kubernetes
  17. 18 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved in-tree plugin( ) • Kubernetes • https://github.com/kubernetes/kubernete s/tree/master/pkg/volume • iSCSI Plugin • Kubernetes • Kubernetes • • Kubernetes v1.14 CSI ( ) • Kubernetes v1.17 Kubernetes – in-tree plugin –
  18. 19 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved FlexVolume • out-of-tree plugin(Kubernetes ) • - root - /usr/libexec/kubernetes/kubelet- plugins/volume/exec/ - (API) • kubelet Volume / / • Flex Volume in-tree plugin • Kubernetes Kubernetes – FlexVolume – kubelet Storage Driver / Flex Volume Driver
  19. 20 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved CSI(Container Storage Interface) • Kubernetes Mesos Docker Cloud Foundry API • • out-of-tree plugin • Kubernetes v1.9 v1.13 GA • CSI • in-tree plugin CSI( ) Kubernetes – CSI – https://github.com/kubernetes/community/blob/master/contributors/design- proposals/storage/container-storage-interface.md#recommended-mechanism-for- deploying-csi-drivers-on-kubernetes • ( ) • Kubernetes
  20. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Rook/Ceph

    Cloud Native Storage Oracle Corporation Japan July 30th, 2020 Takuya Niita
  21. 22 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook • https://rook.io/ • Cloud Native Storage • Cloud Native Storage for Kubernetes • v1.3.8(2020/7 ) • CNCF • Incubating Stage(2020/7 ) • Ceph Cloud Native • v0.8 Ceph Cloud Native Storage – Rook –
  22. 23 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Kubernetes • Kubernetes Operator • Kubernetes • Operator • Prometheus • • Stable - Ceph - EdgeFS( Closed Source) • alpha - Cassandra/CockroachDB/NFS/YugabyteDB Rook
  23. 24 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Ceph • https://ceph.io/ • 2003 Sage Weil UCSC • 2006 OSS • 2012 Inktank 2014 Red Hat • v15.2.4(2020/7 ) • C++ • CSI(Container Storage Interface) • cephalopod Ceph (cephalopod) • v15.x Octopus Cloud Native Storage – Ceph –
  24. 25 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • RADOS(Reliable Autonomic Distributed Object Store) • • RADOS - S3 SWIFT ( S3 SWIFT ( )) • Ceph Worker Node • CSI(Container Storage Interface) • OpenStack • • • OSD(Object Storage Device) • • Ceph Ceph Contributions
  25. 26 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved (RADOSGW/LIBRADOS) • LIBRADOS • RADOS • RADOSGW(Ceph Object Gateway) • AmazonS3/Swift (CephFS) • NFS HDFS Shared Storage (RBD) • Ceph :https://docs.ceph.com/docs/master/architecture/
  26. 27 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved OSD(Object Storage Device) • ( ) • OSD ( ) • Primary OSD Primary OSD Secondary OSD - OSD Placement Group(PG) - PG OSD Primary OSD Secondary OSD - • OSD Placement Group(PG) CRUSH ( ) Ceph – OSD – Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 OSD#1 OSD#2 OSD#3 OSD#4 Ceph Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD
  27. 28 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Pool • • Pool - (RADOSGW/CephFS/RBD) - Placement Group(PG) - Placement Group(PG) • OSD Pool • 1 OSD PG CRUSH • PG OSD Ceph – Pool/PG/CRUSH – 1( 2) 2( 3) PG1 OSD#1 OSD#3 Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 PG2 OSD#4 OSD#2 OSD#5 OSD#1 OSD#2 OSD#3 OSD#4 CRUSH CephFS Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD RBD
  28. 29 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Ceph Ceph ( ) • • • (split brain ) 3 • • (Paxos) / Ceph – – Ceph Storage Cluster #1 Ceph Storage Cluster #2 Paxos Ceph OSD#1 OSD#2 OSD#3 OSD#4 Ceph
  29. 30 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • v12.x • (Prometheus/Zabbix ) • Ceph Dashboard Ceph – – Ceph Storage Cluster #1 (Prometheus/Zabbix) Ceph OSD#1 OSD#2
  30. 31 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Ceph – – 1( 2) PG1 PG2 2( 3) Ceph OSD#1 OSD#2 Ceph Storage Cluster #1 Ceph Storage Cluster #2 CRUSH Ceph Ceph Storage Cluster #3 CRUSH PG/OSD Primary OSD OSD#3 OSD#4 OSD#5 Paxos Paxos Primary OSD Secondary OSD
  31. 32 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • OSD Ceph • OSD • OSD Ceph OSD • OSD OSD • OSD Ceph ( / )
  32. 33 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook • Operator/(Agent) • Rook - Agent FlexVolume Ceph • MON Ceph • OSD’s Ceph OSD • MGR Ceph • MDS/RGW:RADOS • MDS CephFS RGW RADOSGW Rook-Ceph − – ” Many of the Ceph concepts like placement groups and crush maps are hidden so you don’t have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. ” • Ceph (PG CRUSH map ) • UX https://rook.io/docs/rook/v1.3/ceph-storage.html
  33. 34 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook-Ceph CSI • csi-cephfsplugin • CephFS CSI • csi-rbdplugin • RBD CSI • csi-cephfsplugin-provisioner/csi-rbdplugin-provisioner • kube-apiserver CephFS RBD • Rook v1.1 CSI Driver • FlexVolume Driver deprecate Rook-Ceph – CSI(Container Storage Interface) – Rook-Ceph CSI RBD provisioner <rook operator namespace>.rbd.csi.ceph.com
  34. 35 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook • Rook (rook-operator) rook-discover Ceph • OSD( ) Rook-Ceph CSI • CephFS RBD CSI provisioner Rook-Ceph − − kubelet Rook-Ceph CSI rook-operator rook-discover rook-discover rook-discover ceph-osd ceph-osd ceph-osd ceph- mon ceph- mon ceph- mon ceph-mgr provisioner csi-rbdplugin csi- cephfsplugin kubelet kubelet provisioner csi-rbdplugin csi- cephfsplugin provisioner csi-rbdplugin csi- cephfsplugin deployment daemonset
  35. 36 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Rook Operator • Rook-Ceph Ceph • api-server Rook Operator • CSI Application Attach/Detach Rook-Ceph – Kubernetes – kubelet Worker Node Ceph Rook- Ceph CSI api- server Application Rook Operator Master Node etcd Attach/Detach
  36. 37 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved OSD on PVC • PV/PVC OSD Pod • PVC(Persistent Volume Claim) OSD • Rook 1.1 OSD on device • OSD • manifest • Kubernetes Rook-Ceph – Rook-Ceph OSD – PVC
  37. 38 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook-Ceph • Kubernetes( ) Ceph • Ceph SDS(Software Defined Storage) • Ceph Rook Rook • Ceph stable Operator • Cassandra CockroachDB YugabyteDB Rook Rook/Ceph
  38. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle

    Corporation Japan July 30th, 2020 Takuya Niita
  39. 40 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Rook-Ceph • https://github.com/rook/rook/tree/master/cl uster/examples/kubernetes/ceph • 3 manifest • common.yaml • operator.yaml • cluster.yaml 1 – Rook-Ceph ( ) – Block Volume 100GB Ceph Oracle Container Engine for Kubernetes Block Volume 100GB Block Volume 100GB /
  40. 41 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved common.yaml • Namespace • ”rook-ceph” • Custom Resource Definition(CRD) • cephclusters.ceph.rook.io • cephrbdmirrors.ceph.rook.io • cephfilesystems.ceph.rook.io • 14 CRD • RBAC • ClusterRoleBinding/ClusterRole/Role/Role Binding/ ServiceAccount 1 – Rook-Ceph (common.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/common.yaml ※ 2000 Namespace CRD
  41. 42 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved operator.yaml • ConfigMap • Rook Operator • Deployment • Rook Operator - Ceph Rook Operator - rook-discover (FlexVolume )rook-agent 1 – Rook-Ceph (operator.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/operator.yaml ConfigMap Deployment
  42. 43 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved cluster.yaml • CephCluster • OSD on device Ceph • Ceph - 1 – Rook-Ceph (cluster.yaml) – https://raw.githubusercontent.com/rook/rook /master/cluster/examples/kubernetes/ceph/ cluster.yaml CephCluster ( )
  43. 44 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Ceph • • - • Grafana - Prometheus • • • • OSD 1 – Rook-Ceph (Ceph ) – Ceph Ceph
  44. 45 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Rook-Ceph • manifest • 10 • Ceph • 1 – Rook-Ceph (Wrap up) –
  45. 46 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Oracle Container Engine for Kubernetes(OKE) • 1 IOPS • ( ) Standard VM + Block Volume • Standard VM with Rook-Ceph(Block Volume Ceph ) Worker Node(Compute Instance) • Standard VM • VM.Standard.2.8/OCPU :8 OCPU(16 vCPU)/ :120GB/ :8.2Gbps Rook-Ceph • Rook:v1.3.7( v1.3.8) • Ceph:v15.2.4( ) 2 – Rook-Ceph ( ) –
  46. 47 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Kubestone(benchmarking Operator that can evaluate the performance of Kubernetes installations) • https://kubestone.io/en/latest/ • fio Job • - Read/ Write ◦ iodepth=4/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth=64/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth 1 • Volume Size(Block Volume) • Job Pod /data PV 300Gi • Rook-Ceph OSD • 3 Woker Node 1OSD • ( ) 3 → 1 2 – Rook-Ceph ( ) –
  47. 48 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved 2 – Rook-Ceph ( 1) – Block Volume Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 300Gi fio / Kubestone Job 3
  48. 49 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved 2 – Rook-Ceph ( 2) – Block Volume Block Volume 100Gi Ceph (Ceph RBD) Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 100Gi Block Volume 100Gi fio Kubestone Job 3 300Gi
  49. 50 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Read Rook-Ceph IOPS • - OSD - ( ) • Write Rook-Ceph IOPS • - (Rook-Ceph ) 2 – Rook-Ceph ( ) – iodepth=4 iodepth=64 1.49 1.00 0 0.5 1 1.5 2 Read Rook-Ceph Non Rook-Ceph 0.16 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph 1.21 1.00 0 0.5 1 1.5 Read Rook-Ceph Non Rook-Ceph 0.23 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph
  50. 51 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved • Rook-Ceph Read • Write Ceph 2 – Rook-Ceph (Wrap up) –
  51. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle

    Corporation Japan July 30th, 2020 Takuya Niita
  52. 53 Cloud Native Storage • Kubernetes • SC/PV/PVC • CSI

    Rook-Ceph • ( ) Rook • Ceph • Kubernetes Rook-Ceph Rook-Ceph • Ceph Kubernetes Ceph • Rook Operator Copyright © 2020, Oracle and/or its affiliates. All rights reserved
  53. 54 Copyright © 2020, Oracle and/or its affiliates. All rights

    reserved Kuberentes (Storage/Volume) https://kubernetes.io/docs/concepts/storage/ Rook Advent Calendar 2019 https://qiita.com/advent-calendar/2019/rook Rook https://rook.io/docs/rook/v1.3/ Rook(GitHub) https://github.com/rook/rook Ceph https://docs.ceph.com/docs/master/ Ceph(GitHub) https://github.com/ceph & Special Thanks!!