Japan Rook Meetup #4 Cloud Native Storageのもう一つの選択肢 Longhorn

Cloud Native Storageのもう一つの選択肢
Longhorn を紹介します。

Transcript

1. © Copyright 2020 Rancher Labs. All Rights Reserved. 1 © Copyright 2020 Rancher Labs. All Rights Reserved. 1 Japan Rook Meetup #4 Cloud Native Storageのもう一つの選択肢 Longhorn 2020/10/02 Wenhan Shi
2. © Copyright 2020 Rancher Labs. All Rights Reserved. 2 ⾃⼰紹介 • 施 ⽂翰 （シ ブンカン） Wenhan Shi • @shi_wenhan • wenhan.shi@rancher.com • 経歴 • ⽇⽴製作所 - Linux カーネルモジュールの保守サポート • Red Hat K.K. - GlusterFS/OpenShift サポート • Canonical Japan K.K. - Ubuntu/OpenStack/Kubernetes サポート • Rancher Lab Inc, - Support Engineer
3. © Copyright 2020 Rancher Labs. All Rights Reserved. 3 Longhorn! • 横に伸びた⻑い⾓を持った家畜⽜。 • その⾁に脂肪が少なく、低コレステロール&低カロリで、ステーキとして⼤⼈気 https://en.wikipedia.org/wiki/Texas_Longhorn https://www.longhornsteakhouse.com/
4. © Copyright 2020 Rancher Labs. All Rights Reserved. 4 Distributed Block Storage Software for Kubernetes https://longhorn.io/ One click to add persistent storage support to any Kubernetes cluster
5. © Copyright 2020 Rancher Labs. All Rights Reserved. 5 A CNCF sandbox project Longhorn! • クラウドネイティブ向けの分散型ブロックストレージソフトウェア • 3rd June 2020から1.0 GA • 軽量だが⾼信頼＆使いやすい(単独Web UI) • Rancher カタログ・Kubectl・helmからインストール可能 • Certificated Kubernetesならご利⽤可能 • backendストレージに依存しない、Ext4／xfsサポート • Volume単位の容量拡張、Snapshot、backup&restore機能 • Volumeに影響なしのLive Upgrade
6. © Copyright 2020 Rancher Labs. All Rights Reserved. 6 アーキテクチャー
7. © Copyright 2020 Rancher Labs. All Rights Reserved. 7 https://longhorn.io/docs/1.0.2/deploy/install/install-with-rancher/ Install Longhorn from Rancher UI
8. © Copyright 2020 Rancher Labs. All Rights Reserved. 8 https://longhorn.io/docs/1.0.2/deploy/install/install-with-kubectl/ Install Longhorn from kubectl • LonghornのGitリポジトリのYAMLファイルを利用 • インストール後の確認 $ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml $ kubectl -n longhorn-system get pod NAME READY STATUS RESTARTS AGE csi-attacher-6fdc77c485-8wlpg 1/1 Running 0 9d csi-attacher-6fdc77c485-psqlr 1/1 Running 0 9d csi-attacher-6fdc77c485-wkn69 1/1 Running 0 9d csi-provisioner-78f7db7d6d-rj9pr 1/1 Running 0 9d csi-provisioner-78f7db7d6d-sgm6w 1/1 Running 0 9d csi-provisioner-78f7db7d6d-vnjww 1/1 Running 0 9d engine-image-ei-6e2b0e32-2p9nk 1/1 Running 0 9d engine-image-ei-6e2b0e32-s8ggt 1/1 Running 0 9d engine-image-ei-6e2b0e32-wgkj5 1/1 Running 0 9d longhorn-csi-plugin-g8r4b 2/2 Running 0 9d longhorn-csi-plugin-kbxrl 2/2 Running 0 9d longhorn-csi-plugin-wv6sb 2/2 Running 0 9d longhorn-driver-deployer-788984b49c-zzk7b 1/1 Running 0 9d longhorn-manager-nr5rs 1/1 Running 0 9d longhorn-manager-rd4k5 1/1 Running 0 9d longhorn-manager-snb9t 1/1 Running 0 9d longhorn-ui-67b9b6887f-n7x9q 1/1 Running 0 9d
9. © Copyright 2020 Rancher Labs. All Rights Reserved. 9 https://longhorn.io/docs/1.0.2/deploy/install/install-with-helm/ Install Longhorn from helm(2&3) • LonghornのGitリポジトリのYAMLファイルを利用 • インストール後の確認 $ helm repo add longhorn https://charts.longhorn.io $ helm repo update # helm 2 $ helm install longhorn/longhorn --name longhorn --namespace longhorn-system # helm 3 $ kubectl create namespace longhorn-system $ helm install longhorn longhorn/longhorn --namespace longhorn-system $ kubectl -n longhorn-system get pod NAME READY STATUS RESTARTS AGE csi-attacher-6fdc77c485-8wlpg 1/1 Running 0 9d csi-attacher-6fdc77c485-psqlr 1/1 Running 0 9d csi-attacher-6fdc77c485-wkn69 1/1 Running 0 9d …<snip>
10. © Copyright 2020 Rancher Labs. All Rights Reserved. 10 Longhorn UIのデプロイ • Rancher カタログからインストールした場合は不要 • Ingressを作って、外部からのアクセスをLonghorn UI podに誘導 • 設定方法はhttps://longhorn.io/docs/1.0.2/deploy/accessing-the-ui/longhorn-ingress/を参照 • 設定が終わったら、longhorn-frontend svcのアドレスからアクセス kubectl -n longhorn-system get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE longhorn-backend ClusterIP 10.20.248.250 <none> 9500/TCP 58m longhorn-frontend ClusterIP 10.20.245.110 <none> 80/TCP 58m
11. © Copyright 2020 Rancher Labs. All Rights Reserved. 11 Longhorn UI
12. © Copyright 2020 Rancher Labs. All Rights Reserved. 12 Pod with Longhorn Volume -1 • 4node環境を⽤意し、そのうちの三つをLonghorn⽤ • Workloadを全部cp1ノードに ❯ kubectl get node NAME STATUS ROLES AGE VERSION longhorn-demo-cp1 Ready controlplane,etcd,worker 2d19h v1.18.8 longhorn-demo-worker1 Ready worker 2d19h v1.18.8 longhorn-demo-worker2 Ready worker 2d19h v1.18.8 longhorn-demo-worker3 Ready worker 2d19h v1.18.8 cp1 &(worker) Pod worker1 worker2 worker3
13. © Copyright 2020 Rancher Labs. All Rights Reserved. 13 Pod with Longhorn Volume - 2 • StorageClassを指定し、PodからVolumeが利用可能 ❯ cat pod1.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: longhorn-volv-pvc-1 spec: accessModes: - ReadWriteOnce storageClassName: longhorn resources: requests: storage: 2Gi … ❯ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES longhorn-demo 1/1 Running 0 3m28s 10.42.0.24 longhorn-demo-cp1 <none> <none>
14. © Copyright 2020 Rancher Labs. All Rights Reserved. 14 Pod with Longhorn Volume - 2 • volume/pv/pvcが自動で作ってくれる ❯ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE longhorn-volv-pvc-1 Bound pvc-fa8fb473-01d5-4544-92a1-11b122115905 2Gi RWO longhorn 29m ❯ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-fa8fb473-01d5-4544-92a1-11b122115905 2Gi RWO Delete Bound default/longhorn-volv-pvc-1 longhorn 29m root@longhorn-demo-cp1:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 2G 0 disk /var/lib/kubelet/pods/a6db3565-c598-4fe0-8b75-d8c850be075f/volumes/……
15. © Copyright 2020 Rancher Labs. All Rights Reserved. 15 Pod with Longhorn Volume - 3 • VolumeのReplicaは各Longhornノードの/var/lib/longhornに作られる
16. © Copyright 2020 Rancher Labs. All Rights Reserved. 16 Pod with Longhorn Volume - 4 • ちょっと中身を覗いてみる root@longhorn-demo-worker1:/var/lib/longhorn# tree . ├── engine-binaries │ └── longhornio-longhorn-engine-v1.0.2 │ └── longhorn ├── longhorn-disk.cfg └── replicas ├── pvc-47892157-f218-4fed-ac2e-9b1505844bae-1998aacc │ ├── revision.counter │ ├── volume-head-000.img │ ├── volume-head-000.img.meta │ └── volume.meta └── pvc-fa8fb473-01d5-4544-92a1-11b122115905-9c450d92 ├── revision.counter ├── volume-head-000.img ├── volume-head-000.img.meta └── volume.meta
17. © Copyright 2020 Rancher Labs. All Rights Reserved. 17 アーキテクチャー Engine (Data Plane) Manager (Control Plane)
18. © Copyright 2020 Rancher Labs. All Rights Reserved. 18 Longhorn Manager Longhorn Manager (Orchestrates all the volumes) Longhorn CSI Plugin Longhorn API Longhorn UI Longhorn API Kubernetes API Server Volume (CRD) Kubernetes Cluster Container Storage Interface API Engine- Replica Engine - Engine
19. © Copyright 2020 Rancher Labs. All Rights Reserved. 19 アーキテクチャー Worker node Longhorn node Longhorn node Pod Engine manager pod Pod Replica Manager Pod Replica Manager Pod Engine Process Replica Process Engine Process Replica Process Ext4/xfs Ext4/xfs Replica Process Ext4/xfs Replica Process Ext4/xfs PVC/PV Volume PVC/PV Volume Longhorn Manager (DS) Longhorn Manager (DS) Longhorn Manager (DS)
20. © Copyright 2020 Rancher Labs. All Rights Reserved. 20 https://longhorn.io/docs/1.0.2/high-availability/recover-volume/ Demo - Kill Engine Manager pod Worker node Longhorn node Longhorn node Pod Engine manager pod Pod Replica Manager Pod Replica Manager Pod Engine Process Replica Process Engine Process Replica Process Ext4/xfs Ext4/xfs Replica Process Ext4/xfs Replica Process Ext4/xfs PVC Volume PVC Volume Longhorn Manager (DS) Longhorn Manager (DS) Longhorn Manager (DS) • エンジンPod自体が自動的に回復 • auto-remountも可能だが、liveness probeが必要 • 手動の場合、WorkloadのPodの Containerの再起動 CRASH
21. © Copyright 2020 Rancher Labs. All Rights Reserved. 21 Demo - Kill replica Manager pod Worker node Longhorn node Longhorn node Pod Engine manager pod Pod Replica Manager Pod Replica Manager Pod Engine Process Replica Process Engine Process Replica Process Ext4/xfs Ext4/xfs Replica Process Ext4/xfs Replica Process Ext4/xfs PVC Volume PVC Volume Longhorn Manager (DS) Longhorn Manager (DS) Longhorn Manager (DS) • 他のReplicaプロセスが生きているので、IO に影響なし • クラッシュされたReplicaプロセスは自動的 に復活 CRASH
22. © Copyright 2020 Rancher Labs. All Rights Reserved. 22 複数Replicaのどれが最新？ • revision.counterに注目 root@longhorn-demo-worker1:/var/lib/longhorn# tree . ├── engine-binaries │ └── longhornio-longhorn-engine-v1.0.2 │ └── longhorn ├── longhorn-disk.cfg └── replicas ├── pvc-47892157-f218-4fed-ac2e-9b1505844bae-1998aacc │ ├── revision.counter │ ├── volume-head-000.img │ ├── volume-head-000.img.meta │ └── volume.meta └── pvc-fa8fb473-01d5-4544-92a1-11b122115905-9c450d92 ├── revision.counter ├── volume-head-000.img ├── volume-head-000.img.meta └── volume.meta
23. © Copyright 2020 Rancher Labs. All Rights Reserved. 23 複数Replicaのどれが最新？ • このカウンターを参考し、どのReplicaのデータは最新なのかを判断 • 下記の例では、三つとも最新 worker1 revision.counter = 100 worker2 revision.counter = 100 worker3 revision.counter = 100 最 新 最 新 最 新
24. © Copyright 2020 Rancher Labs. All Rights Reserved. 24 複数Replicaのどれが最新？ • 最新ではないと判断された時、このReplicaへのIOは停止され、他のReplicaから最新のデー タをコピー worker1 revision.counter = 100 worker2 revision.counter = 98 worker3 revision.counter = 100 最 新 最 新 最新データ
25. © Copyright 2020 Rancher Labs. All Rights Reserved. 25 複数Replicaのどれが最新？ • 他のReplicaがDownした場合、生きているReplicaが一つだけでもIOは継続 worker1 revision.counter = 100 worker2 revision.counter = 98 worker3 revision.counter = 95 最 新 CRASH CRASH
26. © Copyright 2020 Rancher Labs. All Rights Reserved. 26 複数Replicaのどれが最新？ • Downしたノードが復活された時、最新Replicaから最新データのSyncを行う worker1 revision.counter = 100 worker2 revision.counter = 98 worker3 revision.counter = 95 最 新 CRASH 最新データ
27. © Copyright 2020 Rancher Labs. All Rights Reserved. 27 ユースケース1 VM Vmstore local disk Other SAN LUN LVM VG /data/longhorn Combine multiple storage backends into one Mounted on linux FS
28. © Copyright 2020 Rancher Labs. All Rights Reserved. 28 Kubernetes Cluster Region A ユースケース2 Pod A Standby DR Volume passively replicates from incremental snapshots being sent to a backup store S3 / NFS backup Store Multi-Cluster, Multi-site Disaster Recovery Kubernetes Cluster Region B Pod A Scheduled backups continually send deltas of volume to external cold storage In the event of failover, user syncs with the latest backup and becomes active for workloads in the cluster. Time to recovery is short because most of the delta is already replicated
29. © Copyright 2020 Rancher Labs. All Rights Reserved. 29 Information • Homepage • https://longhorn.io/ • Documentation • https://longhorn.io/docs/1.0.0/ • Roadmap • https://github.com/longhorn/longhorn/wiki/Roadmap • Development updates • https://github.com/longhorn/longhorn/milestones/
30. © Copyright 2020 Rancher Labs. All Rights Reserved. 30 30 Q & A