Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Native Storage Meetup #1 Longhorn

Wenhan Shi
September 17, 2021

Cloud Native Storage Meetup #1 Longhorn

Wenhan Shi

September 17, 2021
Tweet

More Decks by Wenhan Shi

Other Decks in Technology

Transcript

  1. © Copyright 2021 Rancher. All Rights Reserved. 1
    © Copyright 2021 Rancher. All Rights Reserved. 1
    Cloud Native Storage Meetup #1
    Longhorn
    17th Sep 2021
    Wenhan Shi
    Support Engineer

    View Slide

  2. © Copyright 2021 Rancher. All Rights Reserved. 2
    ⾃⼰紹介
    • 施 ⽂翰 (シ ブンカン) Wenhan Shi
    • 経歴
    • ⽇⽴製作所 - Linux カーネルモジュールの保守サポート
    • Red Hat K.K. - GlusterFS/OpenShift サポート
    • Canonical Japan K.K. - Ubuntu/OpenStack/Kubernetes サポート
    • Rancher Lab -> SUSE - Rancher及び周辺製品のサポート
    @shi_wenhan
    [email protected]

    View Slide

  3. © Copyright 2021 Rancher. All Rights Reserved. 3
    Open Source
    Distributed Block Storage Software for Kubernetes
    https://longhorn.io/

    View Slide

  4. © Copyright 2021 Rancher. All Rights Reserved. 4
    Longhorn Web UI

    View Slide

  5. © Copyright 2021 Rancher. All Rights Reserved. 5 5
    • volume rebuildの仕組み
    • v1.2.0の紹介

    View Slide

  6. © Copyright 2021 Rancher. All Rights Reserved. 6 6
    • volume rebuildの仕組み
    • v1.2.0の紹介

    View Slide

  7. © Copyright 2021 Rancher. All Rights Reserved. 7
    Longhorn Web UI
    • Volumeの各コピーのStatusが確認できる
    • データは各ノードの /var/lib/longhorn に保存

    View Slide

  8. © Copyright 2021 Rancher. All Rights Reserved. 8
    アーキテクチャー

    View Slide

  9. © Copyright 2021 Rancher. All Rights Reserved. 9
    アーキテクチャー
    Engine
    (Data Plane)
    Manager
    (Control Plane)

    View Slide

  10. © Copyright 2021 Rancher. All Rights Reserved. 10
    アーキテクチャー
    Worker
    node
    Longhorn
    node
    Longhorn
    node
    Pod
    Engine
    manager pod
    Pod
    Replica
    Manager Pod
    Replica
    Manager Pod
    Engine
    Process
    Replica
    Process
    Engine
    Process
    Replica
    Process
    Ext4/xfs Ext4/xfs
    Replica
    Process
    Ext4/xfs
    Replica
    Process
    Ext4/xfs
    PVC/PV
    Volume
    PVC/PV
    Volume
    Longhorn
    Manager
    (DS)
    Longhorn
    Manager
    (DS)
    Longhorn
    Manager
    (DS)

    View Slide

  11. © Copyright 2021 Rancher. All Rights Reserved. 11
    エラーが突然やってくる
    msg="pvc-c2a86740-04cb-4982-9d41-f86e07570d5e-e-254bed26:
    time="2021-08-22T22:13:30Z" level=error msg="Setting replica
    tcp://10.42.6.168:10075 to ERR due to: r/w timeout""

    View Slide

  12. © Copyright 2021 Rancher. All Rights Reserved. 12
    アーキテクチャー
    Worker
    node
    Longhorn
    node
    Longhorn
    node
    Pod
    Engine
    manager pod
    Pod
    Replica
    Manager Pod
    Replica
    Manager Pod
    Engine
    Process
    Replica
    Process
    Engine
    Process
    Replica
    Process
    Ext4/xfs Ext4/xfs
    Replica
    Process
    Ext4/xfs
    Replica
    Process
    Ext4/xfs
    PVC/PV
    Volume
    PVC/PV
    Volume
    Longhorn
    Manager
    (DS)
    Longhorn
    Manager
    (DS)
    Longhorn
    Manager
    (DS)
    msg="Prepare to delete
    instance pvc-c2a86740-
    04cb-4982-9d41-
    f86e07570d5e-r-cd767c21"

    View Slide

  13. © Copyright 2021 Rancher. All Rights Reserved. 13
    その後は
    • ディスク上のDataはまだ利用可能
    • 新しいプロセスを作成
    • 差分だけコピー
    • dead -> online
    Longhorn
    node
    Replica
    Manager Pod
    Replica
    Process
    Replica
    Process
    Ext4/xfs Ext4/xfs
    Longhorn
    Manager
    (DS)
    Worker
    node
    Pod
    Engine
    manager pod
    Pod
    Engine
    Process
    Engine
    Process
    PVC/PV
    Volume
    PVC/PV
    Volume
    Longhorn
    Manager
    (DS)
    Replica
    Process

    View Slide

  14. © Copyright 2021 Rancher. All Rights Reserved. 14
    その後は
    • replica-replenishment-wait-interval (600s)
    後でも回復不可
    • 新しいプロセスを作成
    • 健康状態のレプリカからフルコピー
    • dead -> online
    Longhorn
    node
    Replica
    Manager Pod
    Replica
    Process
    Replica
    Process
    Ext4/xfs Ext4/xfs
    Longhorn
    Manager
    (DS)
    Worker
    node
    Pod
    Engine
    manager pod
    Pod
    Engine
    Process
    Engine
    Process
    PVC/PV
    Volume
    PVC/PV
    Volume
    Longhorn
    Manager
    (DS)
    Replica
    Process
    Ext4/xfs

    View Slide

  15. © Copyright 2021 Rancher. All Rights Reserved. 15 15
    • volume rebuildの仕組み
    • v1.2.0の紹介

    View Slide

  16. © Copyright 2021 Rancher. All Rights Reserved. 16
    https://github.com/longhorn/longhorn/releases/tag/v1.2.0
    Latest release: Longhorn 1.2.0

    View Slide

  17. © Copyright 2021 Rancher. All Rights Reserved. 17
    v1.2.0 Hightlight
    • #1815 - [FEATURE] Support CSI volume cloning
    • Cloning support for the longhorn CSI driver
    • #1859 Volume Encryption / #902 backup encryption
    • #587 automatic rebalance the replicas
    • automatically check if a new replica can be scheduled on the new node to restore the redundancy.
    • #2206 Enhance Longhorn data plane on low performance enviroment

    View Slide

  18. © Copyright 2021 Rancher. All Rights Reserved. 18
    v1.2.0 knownissue
    • #2964 [BUG] Longhorn 1.2.0 - wrong volume permissions inside container / broken fsGroup
    • Issue: Will get a Permission denied when access the folder.
    • WA: Add a --default-fstype=ext4 to the csi-provisioner container
    • Will be fixed in v1.2.1, which is scheduled to be released at September 24, 2021
    ...
    containers:
    - args:
    - --v=2
    - --csi-address=$(ADDRESS)
    - --timeout=1m50s
    - --leader-election
    - --leader-election-namespace=$(POD_NAMESPACE)
    - --default-fstype=ext4
    env:
    ...

    View Slide

  19. © Copyright 2021 Rancher. All Rights Reserved. 19
    Information
    • Homepage
    • https://longhorn.io/
    • Documentation
    • https://longhorn.io/docs/
    • Roadmap
    • https://github.com/longhorn/longhorn/wiki/Roadmap
    • Development updates
    • https://github.com/longhorn/longhorn/milestones/

    View Slide

  20. © Copyright 2021 Rancher. All Rights Reserved. 20 20
    Thank you

    View Slide