Share
Embed
ML環境でのRook/Ceph
Japan Rook Meetup #3(https://rook.connpass.com/event/174294/)の第3セッションで使用した資料になります.
本資料では,前半にML環境で使用しているRook/Cephについてご紹介した後,Rook/Cephの使っているor今後使用予定の機能について実践を交えながら深く掘り下げていきます.
Yuki Iwai
July 03, 2020
Transcript
32 slides
-
SELF INTRODUCTION ▸ ॴଐ: ۙـେֶେֶӃ(ؔʹ͋Δࢲཱେֶ) ▸ ૯߹ཧֶݚڀՊΤϨΫτϩχΫεܥֶઐ߈ ▸ ίϯϐϡʔλϏδϣϯݚڀࣨɹम࢜1 ▸
Twitter: ˏAokiTenzen ▸ Blog: https://tenzen.hatenablog.com/ 2 -
AGENDA I. Rook/Cephͷ༻ঢ়گ II. OSDs on PVCs with Pod Topology
Spread Constraints III. Rook/CephFSͷোυϝΠϯ ݚڀࣨͰ࣮ݧతʹՔಇ͍ͤͯ͞ΔKubernetes ClusterͰͷ Rook/Cephʹ͍͓ͭͯ͠͠·͢ɽ 3 -
ӡ༻͍ͯ͠ΔKubernetes Cluster(Ұ෦) ▸ ֓ཁ ▸ JupyterLabΛϢʔβʹఏڙ ▸ IngressʹΑΔΞΫηείϯτ ϩʔϧ ▸
Rook/CephͰϝΠϯετϨʔ δΛಈతʹ͍ग़͢ɽ ▸ ֎෦NFSαʔό͔Βαϒετ ϨʔδΛಈతʹ͍ग़͢ɽ ▸ ArgoCDͰ GitOps 6 -
ετϨʔδγεςϜબఆج४ ▸ ཁ݅ ▸ ಉ͡σʔλΛෳਓͰղੳ͍ͨ͠ɽ ▸ ཉ͍࣌͠ʹཉ͍͚ͩ͠ετϨʔδΛ͍ͨ ͍ɽ ▸ JupyterLab্
ͰϑΝΠϧ୯ ҐͰσʔλΛ Π ϯ λ ϥ Ϋ ςΟϒʹૢ࡞ ͠ͳ͕Βղੳ ͍ͨ͠ɽ 7 -
ετϨʔδγεςϜબఆج४ ▸ ཁ݅ ▸ ಉ͡σʔλΛෳਓͰղੳ͍ͨ͠ɽ ▸ ཉ͍࣌͠ʹཉ͍͚ͩ͠ετϨʔδΛ͍ͨ ͍ɽ ▸ JupyterLab্
ͰϑΝΠϧ୯ ҐͰσʔλΛ Π ϯ λ ϥ Ϋ ςΟϒʹૢ࡞ ͠ͳ͕Βղੳ ͍ͨ͠ɽ Rook/CephFSΛબ 8 -
Ceph ͷछྨ ▸ Ceph ▸ Object Storage(RGW) ▸ Amazon S3
Restful APIͱ OpenStack Swift APIޓͷ ΠϯλϑΣʔεΛఏڙ͢Δɽ ▸ Block Devices(RBD) ▸ ϒϩοΫσόΠεΛఏڙ͢Δɽ ▸ FileSystem(CephFS) ▸ POSIX४ڌͷϑΝΠϧγεςϜΛఏڙ͢Δɽ 9 -
Rookʹ͓͚ΔCeph Clusterͷछྨ Host-based Cluster Clusterཧऀ͕Ϛχ ϑΣετͰσόΠ εͷࢦఆΛߦ͍ɼ OSD Pod͕࡞͞ΕΔ λΠϓ
nodes: - name: "node0" devices: - name: "/dev/disk/by-id/ata-hoge0" - name: "/dev/disk/by-id/ata-hoge1" - name: "/dev/disk/by-id/ata-hoge2" - name: "node1" devices: - name: "/dev/disk/by-id/ata-fuga0" - name: "/dev/disk/by-id/ata-fuga1" - name: “/dev/disk/by-id/ata-fuga2" - name: "node2" devices: - name: "/dev/disk/by-id/ata-foo0" - name: "/dev/disk/by-id/ata-foo1" - name: “/dev/disk/by-id/ata-foo2" Rook v1.2͔Β ʮPersistent Block Device Namingʯ ʹରԠ 11 -
Rookʹ͓͚ΔCeph Clusterͷछྨ PVC-based Cluster Cluster͕PVCΛ࡞͠ɼ PVCʹج͍ͮͯCSI͕࡞͠ ͨPVΛͬͯOSD PodΛ࡞ ͢Δɽ storage:
storageClassDeviceSets: - name: japan_rook_sets count: 6 …(লུ) volumeClaimTemplates: - metadata: name: data spec: resources: requests: storage: 10Gi storageClassName: gp2 volumeMode: Block accessModes: - ReadWriteOnce OSD͋ͨΓͷ ༰ྔΛࢦఆ OSDΛࢦఆ storage class Λࢦఆ 12 -
OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ ▸ Rook v1.2ҎલPVC-basedͰOSDͷۉஔ͕ࠔͩͬͨɽ rackA
rackB rackC Nodeα Nodeβ Nodeγ Nodeδ Nodeε Nodeζ OSD0 OSD1 OSD2 OSD3 OSD4 OSD5 15 -
▸ Rook v1.2ҎલPVC-basedͰOSDͷۉஔ͕ࠔͩͬͨɽ rackA rackB rackC Nodeα Nodeβ Nodeγ Nodeδ
Nodeε Nodeζ OSD0 OSD1 OSD2 OSD3 OSD4 OSD6 Rook͞ΜͳΜ͔Μݴ͏ͯ ্ख͍ࣄͯ͘͠ΕΔΜͪΌ͏ʁ ຊʹͰ͖ͳ͍ͷʁ OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 16 -
‣ ࣮ࡍͬͯΈΔͱ… rack໊ Node໊ rackA pvcbased-w0 pvcbased-w1 rackB pvcbased-w2 pvcbased-w3
rackC pvcbased-w4 pvcbased-w5 Node໊ σόΠε pvcbased-w0 3 pvcbased-w1 3 pvcbased-w2 3 pvcbased-w3 3 pvcbased-w4 3 pvcbased-w5 3 rackผNode໊ rackผNode໊ osd 12 osd OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 17 -
rackA: 4×OSD pvcbased-w0: 3×OSD pvcbased-w1: 1×OSD rackB: 5×OSD pvcbased-w2: 3×OSD
pvcbased-w3: 2×OSD rackC: 3×OSD pvcbased-w4: 1×OSD pvcbased-w5: 2×OSD OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 18 -
rackA: 4×OSD pvcbased-w0: 3×OSD pvcbased-w1: 1×OSD rackB: 5×OSD pvcbased-w2: 3×OSD
pvcbased-w3: 2×OSD rackC: 3×OSD pvcbased-w4: 1×OSD pvcbased-w5: 2×OSD rack໊ OSD rackA 4 rackB 5 rackC 3 ༏लͳRook͞ΜͰ ͍͠ͷ͋Δ OSDͷஔ͕ ภͬͯ͠·ͬ ͍ͯΔ… OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 19 -
▸ Pod Topology Spread ConstraintsͷޮՌΛ͔֬ΊΔɽ ▸ ҎԼͷΑ͏ͳ͓͏ͪKubernetes ClusterΛߏஙɽ OSDs on
PVCs with Pod Topology Spread Constraints CPU σΟεΫ DRAM Intel Xeon Platinum 8167M(26C/52T) • Intel OPTANE SSD 900P 280GB ×1 • Intel DC P3700 800GB×2 ECC Registered 192GB ཧߏ role CPU DRAM Mster 1 4vCPU 12GB Worker 6 4vCPU 12GB Clusterߏ 20 -
OSDs on PVCs with Pod Topology Spread Constraints rack໊ Node໊
σόΠε Rook όʔδϣϯ osd CSI rackA pvcbased-w0 pvcbased-w1 3 3 v1.3.5 12 TopoLVM rackB pvcbased-w2 pvcbased-w3 3 3 rackC pvcbased-w4 pvcbased-w5 3 3 Clusterߏ 21 -
…লུ topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.rook.io/rack whenUnsatisfiable: DoNotSchedule labelSelector:
matchExpressions: - key: app operator: In values: - rook-ceph-osd - rook-ceph-osd-prepare - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchExpressions: - key: app operator: In values: - rook-ceph-osd - rook-ceph-osd-prepare ద༻͢ΔϚχϑΣετ 22 topology.rook.io/rackͰࢄஔ rackͷnodeͰͷࢄஔ OSDs on PVCs with Pod Topology Spread Constraints -
OSDs on PVCs with Pod Topology Spread Constraints PVC-based Cluster
OSD༻ͷPV OSD Pod 24 -
OSDs on PVCs with Pod Topology Spread Constraints rackA: 4×OSD
pvcbased-w0: 2×OSD pvcbased-w1: 2×OSD rackB: 4×OSD pvcbased-w2: 2×OSD pvcbased-w3: 2×OSD rackC: 4×OSD pvcbased-w4: 2×OSD pvcbased-w5: 2×OSD 25 -
Rook/CephFSোυϝΠϯͷઃఆ •topology.kubernetes.io/region •topology.kubernetes.io/zone •topology.rook.io/datacenter •topology.rook.io/room •topology.rook.io/pod •topology.rook.io/pdu •topology.rook.io/row •topology.rook.io/rack •topology.rook.io/chassis
27 ▸ Rook/CephͰҎԼͷΑ͏ͳোυϝΠϯ͕༻ҙ͞Ε͓ͯΓɼ Nodeϥϕϧ͚͢Δ͜ͱͰ༻Ͱ͖Δɽ -
•topology.kubernetes.io/region •topology.kubernetes.io/zone •topology.rook.io/datacenter •topology.rook.io/room •topology.rook.io/pod •topology.rook.io/pdu •topology.rook.io/row •topology.rook.io/rack •topology.rook.io/chassis 28
▸ Rook/CephͰҎԼͷΑ͏ͳোυϝΠϯ͕༻ҙ͞Ε͓ͯΓɼ Nodeϥϕϧ͚͢Δ͜ͱͰ༻Ͱ͖Δɽ ࠓճ͜ΕΛ༻ Rook/CephFSোυϝΠϯͷઃఆ -
▸ ҙ ▸ Kubernetes v1.17ΑΓલͰʮfailure-domain.beta.kubernetes.ioʯϥϕϧΛ༻ 29 •topology.kubernetes.io/region •topology.kubernetes.io/zone •topology.rook.io/datacenter •topology.rook.io/room
•topology.rook.io/pod •topology.rook.io/pdu •topology.rook.io/row •topology.rook.io/rack •topology.rook.io/chassis •failure-domain.beta.kubernetes.io/zone •failure-domain.beta.kubernetes.io/region ઌ΄Ͳ࡞ͨ͠ClusterͰࢄஔग़དྷ͍ͯΔ͔֬ೝ͢Δɽ Rook/CephFSোυϝΠϯͷઃఆ -
ࢀߟจݙ ▸ Cephͷछྨ ▸ https://docs.ceph.com/docs/mimic/architecture/ ▸ Japan Rook Meetup #2
ʮRook/Ceph upstream࠷৽ঢ়گʯ ▸ https://speakerdeck.com/sat/ceph-upstreamzui-xin-zhuang-kuang ▸ Persistent Block Device Naming ▸ https://wiki.archlinux.org/index.php/Persistent_block_device_naming ▸ Rook/CephͰͷোυϝΠϯ ▸ https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#osd-topology 31