ML環境でのRook/Ceph

Japan Rook Meetup #3(https://rook.connpass.com/event/174294/)の第3セッションで使用した資料になります．
本資料では，前半にML環境で使用しているRook/Cephについてご紹介した後，Rook/Cephの使っているor今後使用予定の機能について実践を交えながら深く掘り下げていきます．

Transcript

1. ML؀ڥͰͷRook/Ceph
   ͯΜͥΜ
   2020/07/03
   Japan Rook Meetup #3
   1
   

   View Slide

2. SELF INTRODUCTION
   ▸ ॴଐ: ۙـେֶେֶӃ(ؔ੢ʹ͋Δࢲཱେֶ)
   ▸ ૯߹ཧ޻ֶݚڀՊΤϨΫτϩχΫεܥ޻ֶઐ߈
   ▸ ίϯϐϡʔλϏδϣϯݚڀࣨɹम࢜1೥
   ▸ Twitter: ˏAokiTenzen
   ▸ Blog: https://tenzen.hatenablog.com/
   2
   

   View Slide

3. AGENDA
   I. Rook/Cephͷ࢖༻ঢ়گ
   II. OSDs on PVCs
   with Pod Topology Spread Constraints
   III. Rook/CephFSͷো֐υϝΠϯ
   ݚڀࣨͰ࣮ݧతʹՔಇ͍ͤͯ͞ΔKubernetes ClusterͰͷ
   Rook/Cephʹ͍͓ͭͯ࿩͠͠·͢ɽ
   3
   

   View Slide

4. ิ଍
   ͝঺հ͢Δ؀ڥ͸ɼMLΛ࢖͏ΞϓϦέʔγϣϯ
   Λಈ͔͢؀ڥͰ͸ͳ͘ɼMLΛ࢖ͬͯݚڀɾ։ൃΛ
   ߦ͏؀ڥʹͳΓ·͢ɽ
   4
   

   View Slide

5. Rook/Cephͷ࢖༻ঢ়گ
   5
   

   View Slide

6. ӡ༻͍ͯ͠ΔKubernetes Cluster(Ұ෦෼)
   ▸ ֓ཁ
   ▸ JupyterLabΛϢʔβʹఏڙ
   ▸ IngressʹΑΔΞΫηείϯτ
   ϩʔϧ
   ▸ Rook/CephͰϝΠϯετϨʔ
   δΛಈతʹ෷͍ग़͢ɽ
   ▸ ֎෦NFSαʔό͔Βαϒετ
   ϨʔδΛಈతʹ෷͍ग़͢ɽ
   ▸ ArgoCDͰ GitOps
   6
   

   View Slide

7. ετϨʔδγεςϜબఆج४
   ▸ ཁ݅
   ▸ ಉ͡σʔλΛෳ਺ਓͰղੳ͍ͨ͠ɽ
   ▸ ཉ͍࣌͠ʹཉ͍͠෼͚ͩετϨʔδΛ࢖͍ͨ
   ͍ɽ
   ▸ JupyterLab্
   ͰϑΝΠϧ୯
   ҐͰσʔλΛ
   Π ϯ λ ϥ Ϋ
   ςΟϒʹૢ࡞
   ͠ͳ͕Βղੳ
   ͍ͨ͠ɽ
   7
   

   View Slide

8. ετϨʔδγεςϜબఆج४
   ▸ ཁ݅
   ▸ ಉ͡σʔλΛෳ਺ਓͰղੳ͍ͨ͠ɽ
   ▸ ཉ͍࣌͠ʹཉ͍͠෼͚ͩετϨʔδΛ࢖͍ͨ
   ͍ɽ
   ▸ JupyterLab্
   ͰϑΝΠϧ୯
   ҐͰσʔλΛ
   Π ϯ λ ϥ Ϋ
   ςΟϒʹૢ࡞
   ͠ͳ͕Βղੳ
   ͍ͨ͠ɽ
   Rook/CephFSΛબ୒
   8
   

   View Slide

9. Ceph ͷछྨ
   ▸ Ceph
   ▸ Object Storage(RGW)
   ▸ Amazon S3 Restful APIͱ
   OpenStack Swift APIޓ׵ͷ
   ΠϯλϑΣʔεΛఏڙ͢Δɽ
   ▸ Block Devices(RBD)
   ▸ ϒϩοΫσόΠεΛఏڙ͢Δɽ
   ▸ FileSystem(CephFS)
   ▸ POSIX४ڌͷϑΝΠϧγεςϜΛఏڙ͢Δɽ
   9
   

   View Slide

10. Rookʹ͓͚ΔCeph Clusterͷछྨ
    ▸ Rookʹ͓͚ΔCeph Cluster͸2λΠϓ͋Δ
    ▸ Host-based Cluster
    ▸ PVC-based Cluster
    10
    

    View Slide

11. Rookʹ͓͚ΔCeph Clusterͷछྨ
    Host-based Cluster
    Cluster؅ཧऀ͕Ϛχ
    ϑΣετͰ௚઀σόΠ
    εͷࢦఆΛߦ͍ɼ
    OSD Pod͕࡞੒͞ΕΔ
    λΠϓ
    nodes:
    - name: "node0"
    devices:
    - name: "/dev/disk/by-id/ata-hoge0"
    - name: "/dev/disk/by-id/ata-hoge1"
    - name: "/dev/disk/by-id/ata-hoge2"
    - name: "node1"
    devices:
    - name: "/dev/disk/by-id/ata-fuga0"
    - name: "/dev/disk/by-id/ata-fuga1"
    - name: “/dev/disk/by-id/ata-fuga2"
    - name: "node2"
    devices:
    - name: "/dev/disk/by-id/ata-foo0"
    - name: "/dev/disk/by-id/ata-foo1"
    - name: “/dev/disk/by-id/ata-foo2"
    Rook v1.2͔Β͸
    ʮPersistent Block Device Namingʯ
    ʹରԠ
    11
    

    View Slide

12. Rookʹ͓͚ΔCeph Clusterͷछྨ
    PVC-based Cluster
    Cluster͕PVCΛ࡞੒͠ɼ
    PVCʹج͍ͮͯCSI͕࡞੒͠
    ͨPVΛ࢖ͬͯOSD PodΛ࡞
    ੒͢Δɽ
    storage:
    
    storageClassDeviceSets:
    
    - name: japan_rook_sets
    
    count: 6
    
    …(লུ)
    
    volumeClaimTemplates:
    
    - metadata:
    
    name: data
    
    spec:
    
    resources:
    
    requests:
    
    storage: 10Gi
    
    storageClassName: gp2
    
    volumeMode: Block
    
    accessModes:
    
    - ReadWriteOnce
    OSD͋ͨΓͷ
    ༰ྔΛࢦఆ
    OSD਺Λࢦఆ
    storage class
    Λࢦఆ
    12
    

    View Slide

13. Rookʹ͓͚ΔCeph Clusterͷछྨ
    ▸ ݱࡏ͸Host-based ClusterΛ࢖༻த
    ▸ ࠓޙPVC-based ClusterʹҠߦ͍ͨ͠ɽ
    ֤NodeͷσόΠε഑ஔͷৄ
    ࡉΛҙࣝͤͣʹ࢖͑Δ͜ͱ
    ͕ັྗ
    13
    

    View Slide

14. OSDs on PVCs
    with Pod Topology Spread Constraints
    14
    

    View Slide

15. OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ
    ▸ Rook v1.2Ҏલ͸PVC-basedͰOSDͷۉ౳഑ஔ͕ࠔ೉ͩͬͨɽ
    rackA
    rackB rackC
    Nodeα
    Nodeβ
    Nodeγ
    Nodeδ
    Nodeε
    Nodeζ
    OSD0 OSD1
    OSD2
    OSD3 OSD4 OSD5
    15
    

    View Slide

16. ▸ Rook v1.2Ҏલ͸PVC-basedͰOSDͷۉ౳഑ஔ͕ࠔ೉ͩͬͨɽ
    rackA
    rackB rackC
    Nodeα
    Nodeβ
    Nodeγ
    Nodeδ
    Nodeε
    Nodeζ
    OSD0 OSD1
    OSD2
    OSD3 OSD4 OSD6
    Rook͞ΜͳΜ΍͔Μ΍ݴ͏ͯ
    ্ख͍ࣄͯ͘͠ΕΔΜͪΌ͏ʁ
    ຊ౰ʹͰ͖ͳ͍ͷʁ
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 16
    

    View Slide

17. ‣ ࣮ࡍ΍ͬͯΈΔͱ…
    rack໊ Node໊
    rackA
    pvcbased-w0
    pvcbased-w1
    rackB
    pvcbased-w2
    pvcbased-w3
    rackC
    pvcbased-w4
    pvcbased-w5
    Node໊ σόΠε਺
    pvcbased-w0 3
    pvcbased-w1 3
    pvcbased-w2 3
    pvcbased-w3 3
    pvcbased-w4 3
    pvcbased-w5 3
    rackผNode໊
    rackผNode໊
    osd਺
    12
    osd਺
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 17
    

    View Slide

18. rackA: 4×OSD
    pvcbased-w0: 3×OSD
    pvcbased-w1: 1×OSD
    rackB: 5×OSD
    pvcbased-w2: 3×OSD
    pvcbased-w3: 2×OSD
    rackC: 3×OSD
    pvcbased-w4: 1×OSD
    pvcbased-w5: 2×OSD
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 18
    

    View Slide

19. rackA: 4×OSD
    pvcbased-w0: 3×OSD
    pvcbased-w1: 1×OSD
    rackB: 5×OSD
    pvcbased-w2: 3×OSD
    pvcbased-w3: 2×OSD
    rackC: 3×OSD
    pvcbased-w4: 1×OSD
    pvcbased-w5: 2×OSD
    rack໊ OSD਺
    rackA 4
    rackB 5
    rackC 3
    ༏लͳRook͞ΜͰ΋
    ೉͍͠΋ͷ͸͋Δ
    OSDͷ഑ஔ͕
    ภͬͯ͠·ͬ
    ͍ͯΔ…
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 19
    

    View Slide

20. ▸ Pod Topology Spread ConstraintsͷޮՌΛ͔֬ΊΔɽ
    ▸ ҎԼͷΑ͏ͳ͓͏ͪKubernetes ClusterΛߏஙɽ
    OSDs on PVCs with Pod Topology Spread Constraints
    CPU σΟεΫ DRAM
    Intel Xeon Platinum
    8167M(26C/52T)
    • Intel OPTANE SSD 900P 280GB ×1
    • Intel DC P3700 800GB×2
    ECC Registered
    192GB
    ෺ཧߏ੒
    role ୆਺ CPU DRAM
    Mster 1 4vCPU 12GB
    Worker 6 4vCPU 12GB
    Clusterߏ੒
    20
    

    View Slide

21. OSDs on PVCs with Pod Topology Spread Constraints
    rack໊ Node໊
    σόΠε
    ୆਺
    Rook
    όʔδϣϯ
    osd
    ਺
    CSI
    rackA
    pvcbased-w0
    pvcbased-w1
    3
    3
    v1.3.5 12 TopoLVM
    rackB
    pvcbased-w2
    pvcbased-w3
    3
    3
    rackC
    pvcbased-w4
    pvcbased-w5
    3
    3
    Clusterߏ੒
    21
    

    View Slide

22. …লུ
    topologySpreadConstraints:
    - maxSkew: 1
    topologyKey: topology.rook.io/rack
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
    matchExpressions:
    - key: app
    operator: In
    values:
    - rook-ceph-osd
    - rook-ceph-osd-prepare
    - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
    matchExpressions:
    - key: app
    operator: In
    values:
    - rook-ceph-osd
    - rook-ceph-osd-prepare
    ద༻͢ΔϚχϑΣετ
    22
    topology.rook.io/rackͰ෼ࢄ഑ஔ
    rack಺ͷnodeͰͷ෼ࢄ഑ஔ
    OSDs on PVCs with Pod Topology Spread Constraints
    

    View Slide

23. Hands ON!!
    OSDs on PVCs with Pod Topology Spread Constraints 23
    

    View Slide

24. OSDs on PVCs with Pod Topology Spread Constraints
    PVC-based Cluster
    OSD༻ͷPV
    OSD Pod
    24
    

    View Slide

25. OSDs on PVCs with Pod Topology Spread Constraints
    rackA: 4×OSD
    pvcbased-w0: 2×OSD
    pvcbased-w1: 2×OSD
    rackB: 4×OSD
    pvcbased-w2: 2×OSD
    pvcbased-w3: 2×OSD
    rackC: 4×OSD
    pvcbased-w4: 2×OSD
    pvcbased-w5: 2×OSD
    25
    

    View Slide

26. Rook/CephFS
    ͷো֐υϝΠϯ
    26
    

    View Slide

27. Rook/CephFSো֐υϝΠϯͷઃఆ
    •topology.kubernetes.io/region
    •topology.kubernetes.io/zone
    •topology.rook.io/datacenter
    •topology.rook.io/room
    •topology.rook.io/pod
    •topology.rook.io/pdu
    •topology.rook.io/row
    •topology.rook.io/rack
    •topology.rook.io/chassis
    27
    ▸ Rook/CephͰ͸ҎԼͷΑ͏ͳো֐υϝΠϯ͕༻ҙ͞Ε͓ͯΓɼ
    Node΁ϥϕϧ෇͚͢Δ͜ͱͰ࢖༻Ͱ͖Δɽ
    

    View Slide

28. •topology.kubernetes.io/region
    •topology.kubernetes.io/zone
    •topology.rook.io/datacenter
    •topology.rook.io/room
    •topology.rook.io/pod
    •topology.rook.io/pdu
    •topology.rook.io/row
    •topology.rook.io/rack
    •topology.rook.io/chassis
    28
    ▸ Rook/CephͰ͸ҎԼͷΑ͏ͳো֐υϝΠϯ͕༻ҙ͞Ε͓ͯΓɼ
    Node΁ϥϕϧ෇͚͢Δ͜ͱͰ࢖༻Ͱ͖Δɽ
    ࠓճ͸͜ΕΛ࢖༻
    Rook/CephFSো֐υϝΠϯͷઃఆ
    

    View Slide

29. ▸ ஫ҙ఺
    ▸ Kubernetes v1.17ΑΓલͰ͸ʮfailure-domain.beta.kubernetes.ioʯϥϕϧΛ࢖༻
    29
    •topology.kubernetes.io/region
    •topology.kubernetes.io/zone
    •topology.rook.io/datacenter
    •topology.rook.io/room
    •topology.rook.io/pod
    •topology.rook.io/pdu
    •topology.rook.io/row
    •topology.rook.io/rack
    •topology.rook.io/chassis
    •failure-domain.beta.kubernetes.io/zone
    •failure-domain.beta.kubernetes.io/region
    ઌ΄Ͳ࡞੒ͨ͠ClusterͰ෼ࢄ഑ஔग़དྷ͍ͯΔ͔֬ೝ͢Δɽ
    Rook/CephFSো֐υϝΠϯͷઃఆ
    

    View Slide

30. 30
    Rook/CephFSো֐υϝΠϯͷઃఆ
    ʦ3, 11, 5ʧ
    ͪΌΜͱ෼ࢄ͞Ε͍ͯΔʂʂ
    

    View Slide

31. ࢀߟจݙ
    ▸ Cephͷछྨ
    ▸ https://docs.ceph.com/docs/mimic/architecture/
    ▸ Japan Rook Meetup #2 ʮRook/Ceph upstream࠷৽ঢ়گʯ
    ▸ https://speakerdeck.com/sat/ceph-upstreamzui-xin-zhuang-kuang
    ▸ Persistent Block Device Naming
    ▸ https://wiki.archlinux.org/index.php/Persistent_block_device_naming
    ▸ Rook/CephͰͷো֐υϝΠϯ
    ▸ https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#osd-topology
    31
    

    View Slide

32. THANK YOU
    32
    

    View Slide