$30 off During Our Annual Pro Sale. View Details »

ML環境でのRook/Ceph

 ML環境でのRook/Ceph

Japan Rook Meetup #3(https://rook.connpass.com/event/174294/)の第3セッションで使用した資料になります
本資料では,前半にML環境で使用しているRook/Cephについてご紹介した後,Rook/Cephの使っているor今後使用予定の機能について実践を交えながら深く掘り下げていきます.

Yuki Iwai

July 03, 2020
Tweet

More Decks by Yuki Iwai

Other Decks in Technology

Transcript

  1. ML؀ڥͰͷRook/Ceph
    ͯΜͥΜ
    2020/07/03
    Japan Rook Meetup #3
    1

    View Slide

  2. SELF INTRODUCTION
    ▸ ॴଐ: ۙـେֶେֶӃ(ؔ੢ʹ͋Δࢲཱେֶ)
    ▸ ૯߹ཧ޻ֶݚڀՊΤϨΫτϩχΫεܥ޻ֶઐ߈
    ▸ ίϯϐϡʔλϏδϣϯݚڀࣨɹम࢜1೥
    ▸ Twitter: ˏAokiTenzen
    ▸ Blog: https://tenzen.hatenablog.com/
    2

    View Slide

  3. AGENDA
    I. Rook/Cephͷ࢖༻ঢ়گ
    II. OSDs on PVCs
    with Pod Topology Spread Constraints
    III. Rook/CephFSͷো֐υϝΠϯ
    ݚڀࣨͰ࣮ݧతʹՔಇ͍ͤͯ͞ΔKubernetes ClusterͰͷ
    Rook/Cephʹ͍͓ͭͯ࿩͠͠·͢ɽ
    3

    View Slide

  4. ิ଍
    ͝঺հ͢Δ؀ڥ͸ɼMLΛ࢖͏ΞϓϦέʔγϣϯ
    Λಈ͔͢؀ڥͰ͸ͳ͘ɼMLΛ࢖ͬͯݚڀɾ։ൃΛ
    ߦ͏؀ڥʹͳΓ·͢ɽ
    4

    View Slide

  5. Rook/Cephͷ࢖༻ঢ়گ
    5

    View Slide

  6. ӡ༻͍ͯ͠ΔKubernetes Cluster(Ұ෦෼)
    ▸ ֓ཁ
    ▸ JupyterLabΛϢʔβʹఏڙ
    ▸ IngressʹΑΔΞΫηείϯτ
    ϩʔϧ
    ▸ Rook/CephͰϝΠϯετϨʔ
    δΛಈతʹ෷͍ग़͢ɽ
    ▸ ֎෦NFSαʔό͔Βαϒετ
    ϨʔδΛಈతʹ෷͍ग़͢ɽ
    ▸ ArgoCDͰ GitOps
    6

    View Slide

  7. ετϨʔδγεςϜબఆج४
    ▸ ཁ݅
    ▸ ಉ͡σʔλΛෳ਺ਓͰղੳ͍ͨ͠ɽ
    ▸ ཉ͍࣌͠ʹཉ͍͠෼͚ͩετϨʔδΛ࢖͍ͨ
    ͍ɽ
    ▸ JupyterLab্
    ͰϑΝΠϧ୯
    ҐͰσʔλΛ
    Π ϯ λ ϥ Ϋ
    ςΟϒʹૢ࡞
    ͠ͳ͕Βղੳ
    ͍ͨ͠ɽ
    7

    View Slide

  8. ετϨʔδγεςϜબఆج४
    ▸ ཁ݅
    ▸ ಉ͡σʔλΛෳ਺ਓͰղੳ͍ͨ͠ɽ
    ▸ ཉ͍࣌͠ʹཉ͍͠෼͚ͩετϨʔδΛ࢖͍ͨ
    ͍ɽ
    ▸ JupyterLab্
    ͰϑΝΠϧ୯
    ҐͰσʔλΛ
    Π ϯ λ ϥ Ϋ
    ςΟϒʹૢ࡞
    ͠ͳ͕Βղੳ
    ͍ͨ͠ɽ
    Rook/CephFSΛબ୒
    8

    View Slide

  9. Ceph ͷछྨ
    ▸ Ceph
    ▸ Object Storage(RGW)
    ▸ Amazon S3 Restful APIͱ
    OpenStack Swift APIޓ׵ͷ
    ΠϯλϑΣʔεΛఏڙ͢Δɽ
    ▸ Block Devices(RBD)
    ▸ ϒϩοΫσόΠεΛఏڙ͢Δɽ
    ▸ FileSystem(CephFS)
    ▸ POSIX४ڌͷϑΝΠϧγεςϜΛఏڙ͢Δɽ
    9

    View Slide

  10. Rookʹ͓͚ΔCeph Clusterͷछྨ
    ▸ Rookʹ͓͚ΔCeph Cluster͸2λΠϓ͋Δ
    ▸ Host-based Cluster
    ▸ PVC-based Cluster
    10

    View Slide

  11. Rookʹ͓͚ΔCeph Clusterͷछྨ
    Host-based Cluster
    Cluster؅ཧऀ͕Ϛχ
    ϑΣετͰ௚઀σόΠ
    εͷࢦఆΛߦ͍ɼ
    OSD Pod͕࡞੒͞ΕΔ
    λΠϓ
    nodes:
    - name: "node0"
    devices:
    - name: "/dev/disk/by-id/ata-hoge0"
    - name: "/dev/disk/by-id/ata-hoge1"
    - name: "/dev/disk/by-id/ata-hoge2"
    - name: "node1"
    devices:
    - name: "/dev/disk/by-id/ata-fuga0"
    - name: "/dev/disk/by-id/ata-fuga1"
    - name: “/dev/disk/by-id/ata-fuga2"
    - name: "node2"
    devices:
    - name: "/dev/disk/by-id/ata-foo0"
    - name: "/dev/disk/by-id/ata-foo1"
    - name: “/dev/disk/by-id/ata-foo2"
    Rook v1.2͔Β͸
    ʮPersistent Block Device Namingʯ
    ʹରԠ
    11

    View Slide

  12. Rookʹ͓͚ΔCeph Clusterͷछྨ
    PVC-based Cluster
    Cluster͕PVCΛ࡞੒͠ɼ
    PVCʹج͍ͮͯCSI͕࡞੒͠
    ͨPVΛ࢖ͬͯOSD PodΛ࡞
    ੒͢Δɽ
    storage:

    storageClassDeviceSets:

    - name: japan_rook_sets

    count: 6

    …(লུ)

    volumeClaimTemplates:

    - metadata:

    name: data

    spec:

    resources:

    requests:

    storage: 10Gi

    storageClassName: gp2

    volumeMode: Block

    accessModes:

    - ReadWriteOnce
    OSD͋ͨΓͷ
    ༰ྔΛࢦఆ
    OSD਺Λࢦఆ
    storage class
    Λࢦఆ
    12

    View Slide

  13. Rookʹ͓͚ΔCeph Clusterͷछྨ
    ▸ ݱࡏ͸Host-based ClusterΛ࢖༻த
    ▸ ࠓޙPVC-based ClusterʹҠߦ͍ͨ͠ɽ
    ֤NodeͷσόΠε഑ஔͷৄ
    ࡉΛҙࣝͤͣʹ࢖͑Δ͜ͱ
    ͕ັྗ
    13

    View Slide

  14. OSDs on PVCs
    with Pod Topology Spread Constraints
    14

    View Slide

  15. OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ
    ▸ Rook v1.2Ҏલ͸PVC-basedͰOSDͷۉ౳഑ஔ͕ࠔ೉ͩͬͨɽ
    rackA
    rackB rackC
    Nodeα
    Nodeβ
    Nodeγ
    Nodeδ
    Nodeε
    Nodeζ
    OSD0 OSD1
    OSD2
    OSD3 OSD4 OSD5
    15

    View Slide

  16. ▸ Rook v1.2Ҏલ͸PVC-basedͰOSDͷۉ౳഑ஔ͕ࠔ೉ͩͬͨɽ
    rackA
    rackB rackC
    Nodeα
    Nodeβ
    Nodeγ
    Nodeδ
    Nodeε
    Nodeζ
    OSD0 OSD1
    OSD2
    OSD3 OSD4 OSD6
    Rook͞ΜͳΜ΍͔Μ΍ݴ͏ͯ
    ্ख͍ࣄͯ͘͠ΕΔΜͪΌ͏ʁ
    ຊ౰ʹͰ͖ͳ͍ͷʁ
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 16

    View Slide

  17. ‣ ࣮ࡍ΍ͬͯΈΔͱ…
    rack໊ Node໊
    rackA
    pvcbased-w0
    pvcbased-w1
    rackB
    pvcbased-w2
    pvcbased-w3
    rackC
    pvcbased-w4
    pvcbased-w5
    Node໊ σόΠε਺
    pvcbased-w0 3
    pvcbased-w1 3
    pvcbased-w2 3
    pvcbased-w3 3
    pvcbased-w4 3
    pvcbased-w5 3
    rackผNode໊
    rackผNode໊
    osd਺
    12
    osd਺
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 17

    View Slide

  18. rackA: 4×OSD
    pvcbased-w0: 3×OSD
    pvcbased-w1: 1×OSD
    rackB: 5×OSD
    pvcbased-w2: 3×OSD
    pvcbased-w3: 2×OSD
    rackC: 3×OSD
    pvcbased-w4: 1×OSD
    pvcbased-w5: 2×OSD
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 18

    View Slide

  19. rackA: 4×OSD
    pvcbased-w0: 3×OSD
    pvcbased-w1: 1×OSD
    rackB: 5×OSD
    pvcbased-w2: 3×OSD
    pvcbased-w3: 2×OSD
    rackC: 3×OSD
    pvcbased-w4: 1×OSD
    pvcbased-w5: 2×OSD
    rack໊ OSD਺
    rackA 4
    rackB 5
    rackC 3
    ༏लͳRook͞ΜͰ΋
    ೉͍͠΋ͷ͸͋Δ
    OSDͷ഑ஔ͕
    ภͬͯ͠·ͬ
    ͍ͯΔ…
    OSDs on PVCsʹ͓͚ΔPod Topology Spread Constraintsͷ༗༻ੑ 19

    View Slide

  20. ▸ Pod Topology Spread ConstraintsͷޮՌΛ͔֬ΊΔɽ
    ▸ ҎԼͷΑ͏ͳ͓͏ͪKubernetes ClusterΛߏஙɽ
    OSDs on PVCs with Pod Topology Spread Constraints
    CPU σΟεΫ DRAM
    Intel Xeon Platinum
    8167M(26C/52T)
    • Intel OPTANE SSD 900P 280GB ×1
    • Intel DC P3700 800GB×2
    ECC Registered
    192GB
    ෺ཧߏ੒
    role ୆਺ CPU DRAM
    Mster 1 4vCPU 12GB
    Worker 6 4vCPU 12GB
    Clusterߏ੒
    20

    View Slide

  21. OSDs on PVCs with Pod Topology Spread Constraints
    rack໊ Node໊
    σόΠε
    ୆਺
    Rook
    όʔδϣϯ
    osd

    CSI
    rackA
    pvcbased-w0
    pvcbased-w1
    3
    3
    v1.3.5 12 TopoLVM
    rackB
    pvcbased-w2
    pvcbased-w3
    3
    3
    rackC
    pvcbased-w4
    pvcbased-w5
    3
    3
    Clusterߏ੒
    21

    View Slide

  22. …লུ
    topologySpreadConstraints:
    - maxSkew: 1
    topologyKey: topology.rook.io/rack
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
    matchExpressions:
    - key: app
    operator: In
    values:
    - rook-ceph-osd
    - rook-ceph-osd-prepare
    - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
    matchExpressions:
    - key: app
    operator: In
    values:
    - rook-ceph-osd
    - rook-ceph-osd-prepare
    ద༻͢ΔϚχϑΣετ
    22
    topology.rook.io/rackͰ෼ࢄ഑ஔ
    rack಺ͷnodeͰͷ෼ࢄ഑ஔ
    OSDs on PVCs with Pod Topology Spread Constraints

    View Slide

  23. Hands ON!!
    OSDs on PVCs with Pod Topology Spread Constraints 23

    View Slide

  24. OSDs on PVCs with Pod Topology Spread Constraints
    PVC-based Cluster
    OSD༻ͷPV
    OSD Pod
    24

    View Slide

  25. OSDs on PVCs with Pod Topology Spread Constraints
    rackA: 4×OSD
    pvcbased-w0: 2×OSD
    pvcbased-w1: 2×OSD
    rackB: 4×OSD
    pvcbased-w2: 2×OSD
    pvcbased-w3: 2×OSD
    rackC: 4×OSD
    pvcbased-w4: 2×OSD
    pvcbased-w5: 2×OSD
    25

    View Slide

  26. Rook/CephFS
    ͷো֐υϝΠϯ
    26

    View Slide

  27. Rook/CephFSো֐υϝΠϯͷઃఆ
    •topology.kubernetes.io/region
    •topology.kubernetes.io/zone
    •topology.rook.io/datacenter
    •topology.rook.io/room
    •topology.rook.io/pod
    •topology.rook.io/pdu
    •topology.rook.io/row
    •topology.rook.io/rack
    •topology.rook.io/chassis
    27
    ▸ Rook/CephͰ͸ҎԼͷΑ͏ͳো֐υϝΠϯ͕༻ҙ͞Ε͓ͯΓɼ
    Node΁ϥϕϧ෇͚͢Δ͜ͱͰ࢖༻Ͱ͖Δɽ

    View Slide

  28. •topology.kubernetes.io/region
    •topology.kubernetes.io/zone
    •topology.rook.io/datacenter
    •topology.rook.io/room
    •topology.rook.io/pod
    •topology.rook.io/pdu
    •topology.rook.io/row
    •topology.rook.io/rack
    •topology.rook.io/chassis
    28
    ▸ Rook/CephͰ͸ҎԼͷΑ͏ͳো֐υϝΠϯ͕༻ҙ͞Ε͓ͯΓɼ
    Node΁ϥϕϧ෇͚͢Δ͜ͱͰ࢖༻Ͱ͖Δɽ
    ࠓճ͸͜ΕΛ࢖༻
    Rook/CephFSো֐υϝΠϯͷઃఆ

    View Slide

  29. ▸ ஫ҙ఺
    ▸ Kubernetes v1.17ΑΓલͰ͸ʮfailure-domain.beta.kubernetes.ioʯϥϕϧΛ࢖༻
    29
    •topology.kubernetes.io/region
    •topology.kubernetes.io/zone
    •topology.rook.io/datacenter
    •topology.rook.io/room
    •topology.rook.io/pod
    •topology.rook.io/pdu
    •topology.rook.io/row
    •topology.rook.io/rack
    •topology.rook.io/chassis
    •failure-domain.beta.kubernetes.io/zone
    •failure-domain.beta.kubernetes.io/region
    ઌ΄Ͳ࡞੒ͨ͠ClusterͰ෼ࢄ഑ஔग़དྷ͍ͯΔ͔֬ೝ͢Δɽ
    Rook/CephFSো֐υϝΠϯͷઃఆ

    View Slide

  30. 30
    Rook/CephFSো֐υϝΠϯͷઃఆ
    ʦ3, 11, 5ʧ
    ͪΌΜͱ෼ࢄ͞Ε͍ͯΔʂʂ

    View Slide

  31. ࢀߟจݙ
    ▸ Cephͷछྨ
    ▸ https://docs.ceph.com/docs/mimic/architecture/
    ▸ Japan Rook Meetup #2 ʮRook/Ceph upstream࠷৽ঢ়گʯ
    ▸ https://speakerdeck.com/sat/ceph-upstreamzui-xin-zhuang-kuang
    ▸ Persistent Block Device Naming
    ▸ https://wiki.archlinux.org/index.php/Persistent_block_device_naming
    ▸ Rook/CephͰͷো֐υϝΠϯ
    ▸ https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#osd-topology
    31

    View Slide

  32. THANK YOU
    32

    View Slide