Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[EN]Trident Deep Dive

3499a1d71fa70b8ee44816ca9e7329fe?s=47 bells17
February 25, 2022

[EN]Trident Deep Dive



February 25, 2022

More Decks by bells17

Other Decks in Programming


  1. Trident Deep Dive 2022 Feb Translated by: Yoshiki Fujiwara IDC

    Frontier Inc. Software Engineer Daiki Hayakawa NetApp G.K. Japan Cloud Solutions Architect for AWS Yoshiki Fujiwara
  2. ▶ @bells17 ▶ Software Engineer@IDC Frontier inc. ▶ What I

    usually do : + Kubernetes Development of related components + Kubernetes as a Service Development ▶ Kubernetes SIG-Docs Japanese localization reviewer ▶ Kubernetes Internal Organizer ▶ #kubenews ▶ @bells17_
  3. #kubenews Steams on YouTube Almost Every Friday 22:00 JST- We

    have technical chats mainly Kubernetes/Cloud Native related news.
  4. What I talk today ▶ Trident Overview ▶ Kubernetes and

    CSI ▶ Trident Implementation
  5. Cautionary Note ▶ The presenter is not a storage expert

    (especially iSCSI or NFS ..) ▶ Trident version is expected to be v21.07.2 ▶ Trident is expected to be used with the following settings: + Use Trident from Kubernetes + Trident is basically installed using Trident Operator (described later) ▶ Since it is only an explanation of understanding as a result of following the implementation of Trident, there may be cases where it differs from the actual behavior
  6. Mr. Ohno of NetApp also explains the architecture of Trident

    in Cloud Native Storage Meetup # 1, so please refer to that as well(Presented in Japanese) https://youtu.be/2xEUyAzoNmY?t=3583
  7. What is Trident?

  8. What is Trident? ▶ Trident is an application for taking

    advantage of various NetApp storage products (e.g. ONTAP, E- Series, Cloud Volumes Service for AWS, etc...) ▶ With Trident, you will be able to operate NetApp storage products in a container environment such as Kubernetes ▶ The following two platforms are currently supported: + Kubernetes ← I will talk how Trident works on Kubernetes here + Docker ▶ Trident is an OSS (https://github.com/NetApp/trident) ▶ Trident is one of the applications included in Project "Astra" ▶ Astra consists of following applications in addition to Trident + Astra Control: Kubernetes cluster management and operations console + Astra Data Store: Kubernetes native shared file service
  9. https://docs.netapp.com/us-en/astra-family/intro-family.html

  10. Trident Components

  11. Trident Components ▶ Trident(Core): The main body of Trident consists

    of a CSI Driver for cooperating with Kubernetes, a Rest API server for tridentctl, and various controllers for controlling the state of Trident ▶ tridentctl: The Command line tool for operating Trident from cli ▶ Trident Operator: Kubernetes Operator for managing Trident installations and upgrades on Kubernetes clusters + It was added from Trident v20.04.0 + You can install trident with the tridentctl install command in addition to the Trident Operator
  12. Trident installation example

  13. Create an ONTAP Storage VM (when using ONTAP)





  18. Trident and Kubernetes

  19. Trident and Kubernetes ▶ Trident runs as a CSI Driver

    for manage various NetApp storage on Kubernetes ▶ It also implements multiple Kubernetes Controllers for Trident management ▶ Therefore, before we get into the implementation of Trident, It is better to know about the outline such as + Kubernetes Operator(Kubernetes Controller) + CSI(driver)
  20. What is Kubernetes? ▶ Kubernetes is one of the container

    orchestrators ▶ You can build a cluster composed of etcd / control plane / worker node, run various containers on node on Kubernetes, and enable to link the running container and network nicely ▶ By declaratively describing the container and other resources to be deployed in the manifest file, Kubernetes will perform adjustment processing nicely so that it will be in the declared state ▶ A container orchestrator that recreated from Borg, a container platform operated internally by Google, for OSS ▶ Kubernetes has also been donated to the Cloud Native Computing Foundation(CNCF) and is managed on a community basis as a CNCF Graduated project
  21. https://github.com/kubernetes/website/blob/fb6364da0afd19e8a9515aaae2de9bc74a0a6abd/static/images/docs/components-of-kubernetes.png

  22. How Trident extends Kubernetes ▶ Kubernetes Operator + CRD +

    Kubernetes Controller ▶ CSI Driver
  23. https://github.com/NetApp/trident/blob/v21.07.2/operator/controllers/orchestrator/apis/netapp/v1/types.go CRD

  24. Kubernetes Controller

  25. https://github.com/kubernetes/sample-controller/blob/master/docs/images/client-go-controller-interaction.jpeg How Controller works

  26. ▶ Kubernetesのコードリーディングをする上で知っておくと良さそうなこと ▶ Kubernetes Internal #1 Click following links for

    details around the Kubernetes Controller implementations(Presented in Japanese)
  27. Kubernetes and CSI

  28. CSI ▶ Definition of common specifications for using storage in

    Container Orchestrator (CO) such as Kubernetes, Mesos, Cloud Foundry, etc + So it's not a specification only for Kubernetes + For example, Hashicorp Nomad is using CSI under the hood ▶ Storage providers aim to be able to utilize Kubernetes and other COs by writing a driver that supports CSI once ▶ The CSI specification is defined in the the spec.md file in the container- storage- interface/spec repository on Github
  29. Specifications defined by CSI ▶ The communication method and provision

    method of CSI Driver ▶ Features provided by CSI Driver ▶ gRPC Protocol Buffers to use CSI Driver by CO
  30. How CSI Driver Communicates and being provided ▶ Need to

    be provided in container image format (Docker, OCI, etc.) ▶ Communication between CSI Driver and CO needs to ... + Use gRPC protocol + Via UNIX domain socket
  31. Features provided by CSI ▶ Create / Delete volume ▶

    Attach / Detach volume to node ▶ Volume Mount / Unmount ▶ Create / Delete volume Snapshot ▶ etc...
  32. gRPC interfaces defined in Protocol Buffers ▶ Controller Plugin ▶

    Node Plugin
  33. Controller Plugin ▶ A gRPC server that operates as a

    control plane for its CSI Driver ▶ Implementation of the following gRPC services: + Controller Service + Identity Service ▶ It provides the ability to control the volume and the snapshot ▶ Specifically, it provides the following features: + Create / Delete volume + Attach / Detach volume to node + Create / Delete volume snapshot
  34. Node Plugin ▶ A gRPC server that operates on each

    CO-participating Worker Node ▶ Implementation of the following gRPC services: + Node Service + Identity Service ▶ It provides the feature to operate the volume on each target Worker Node ▶ It mainly provides the following features: + Format volume attached to node + Mount / Unmount Volume
  35. None
  36. Volume Lifecycle

  37. CSI wrap-up ▶ CSI is a common specification defined by

    storage providers to provide volume plug-ins to CO ▶ Specifically, the following are defined + Definition of operating environment and communication method of CSI Driver + Container image format + UNIX domain socket / gRCP protocol + Definition of RPC interface + Controller Plugin + Node Plugin
  38. Kubernetes and Volume Plugin

  39. There are three Kubernetes volume plugins ▶ In-Tree Volume +

    Volume plugin implemented inside Kubernetes code + ConfigMap / Secret / EmptyDir etc. fall into here ▶ FlexVolume + Plugin created before CSI appeared (Kubernetes v1.8) + It seems that it is not used much because it requires knowledge about the internal implementation of Kubernetes + Deprecated in Recent Kubernetes Updated versions ▶ CSI Driver + Alpha is available from Kubernetes v1.9 + Kubernetes provides a sidecar application that works with the CSI Driver, so you can provide volume plugins without the need for knowledge of the internal implementation of Kubernetes
  40. Integration between Kubernetes and CSI Driver

  41. Integration by Sidecar application https://github.com/kubernetes/community/blob/d83cd53979d08ac0e0e33704c6aec6b1c3cb7c8d/contributors/design-proposals/storage/container-storage-interface_diagram1.png

  42. Processing flow when creating a volume

  43. Controller Sidecar external-provisioner Create / Delete volume external-attacher Attach /

    Detach volume external-resizer Resize the volume external-snapshotter Create / Delete snapshot livenessprobe HTTP proxy for Liveness Probe
  44. Node Sidecar node-driver-registrar It provides the feature to register the

    CSI driver in Kubelet by using the feature called “Plugin Watcher” in Kubelet
  45. ▶ CSI⼊⾨(スライド) ▶ CSI⼊⾨(セッション動画) Click following links for a more

    detailed explanation of Kubernetes and CSI(Presented in Japanese)
  46. Trident implementation

  47. Trident Components(Again)

  48. Trident Overview

  49. Trident Operator

  50. Trident Core(setup)

  51. Trident Core(PVC Create ~ Volume Mount)

  52. Trident Core: Other components ▶ TransactionMonitor: Execution processing task management

    Controller using TridentTransaction resource for managing volume creation etc ▶ PeriopdicallyReconcileNodeAccessOnBackends: A Controller that checks that the appropriate policy settings are set so that each node and each Trident Backend can connect ▶ k8shelper + Node Controller: A Controller that deletes the target Trident Node when deleting a node and deletes the target node information from the policy of each Trident Backend ▶ CRD Controllers + reconcileTMR: Set the SnapMirror according to the TridentMirrorRelationship resource and update the status of the TridentMirrorRelationship ▶ etc

  54. tridentctl/Rest API

  55. Conclusion

  56. Impressions of reading the implementation ▶ It was characteristic and

    interesting to be able to use TridentBackend for multiple NetApp storage from one Trident ▶ On the other hand, I got the impression that it would be difficult to start using it because the setting method of TridentBackend and StorageClass is complicated ▶ There were some features and resource definitions (eg Snapshot resources, etc.) that were developed from a relatively old age or seem to be unused now. I got the impression that the implementation could be kept simple by redefining the features, implementations, and support scope of the application
  57. Reference materials ▶ https://youtu.be/2xEUyAzoNmY?t=3583 ▶ https://github.com/NetApp/trident/tree/v21.07.2 ▶ https://netapp-trident.readthedocs.io/en/stable-v21.07/dag/kubernetes/index.html ▶ https://netapp-trident.readthedocs.io/en/stable-v21.07/kubernetes/deploying/operator-deploy.html

    ▶ https://netapp-trident.readthedocs.io/en/stable-v21.07/kubernetes/operations/tasks/managing-backends/tbc.html ▶ https://netapp-trident.readthedocs.io/en/stable-v21.07/kubernetes/operations/tasks/monitoring.html#trident-autosupport-telemetry ▶ https://hub.docker.com/r/netapp/trident-autosupport ▶ https://netapp-trident.readthedocs.io/en/stable-v21.07/kubernetes/concepts/objects.html ▶ https://library.netapp.com/ecmdocs/ECMLP2372138/html/GUID-3FC8A37A-FFCC-4070-A9F0-1B9B3FB79BF8.html ▶ https://milestone-of-se.nesuke.com/sv-basic/architecture/disk-term/ ▶ https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/7/html/dm_multipath/mpio_overview ▶ https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/7/html/dm_multipath/mpio_description ▶ https://qiita.com/ochiba/items/39dbcda84ec17aefed07 ▶ https://tech-mmmm.blogspot.com/2020/05/iscsi-dm-multipathrheliscsi.html ▶ https://milestone-of-se.nesuke.com/sv-basic/architecture/iscsi-summary/ ▶ https://library.netapp.com/ecmdocs/ECMLP2573234/html/GUID-EC3C367B-79E0-4DBA-8036-22094557357A.html ▶ https://qiita.com/OPySPGcLYpJE0Tc/items/be9daae23b80478b81ff ▶ https://qiita.com/hana_shin/items/cbd428faf92534e25f7b ▶ https://atmarkit.itmedia.co.jp/ait/articles/0807/02/news142.html ▶ https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/8/html/managing_storage_devices/getting-started-with-iscsi_managing-storage-devices ▶ https://docs.netapp.com/ja-jp/ontap/system-admin/command-line-interface-concept.html ▶ https://docs.netapp.com/ja-jp/ontap/volumes/commands-manage-flexvol-volumes-reference.html ▶ http://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-cmpr-9101/home.html ▶ https://docs.netapp.com/us-en/ontap/concepts/snapmirror-cloud-backups-object-store-concept.html ▶ https://netapp-trident.readthedocs.io/en/stable-v21.07/kubernetes/operations/tasks/volumes/topology.html?highlight=supportedtopology
  58. Thanks / Question? ▶ @bells17 ▶ Slide: https://speakerdeck.com/bells17 ▶ @bells17_

  59. ・Points that I was interested in after reading the code

    ・Points that the panelist was interested in after listening to this session Discussion/Q&A
  60. • セッションコード:1689 ⼊⾨ タイトル:クラウド知っ得シリーズ NetApp Astra スピーカ:ネットアップ合同会社 ソリューション技術本部 SE第1部 ソリューションズエンジニア

    Zhao Mandy • セッションコード:1687 中級 タイトル:クラウドネイティブアプリケーションの採 ⽤を加速する「Astra Data Store」とは︖ スピーカ:ネットアップ合同会社 ソリューションアーキテクト部 シニアソリューションアーキテクト ⼤削 緑 • セッションコード:1740 中級 タイトル: NetApp Astraを利⽤したKubernetes環境の データポータビリティの実現 スピーカ:ネットワンシステムズ株式会社 ビジネス開発本部 第1応⽤技術部 クラウドインフラチーム ⾦只 圭司 Related Sessions with Trident in NetApp INSIGHT Japan 2022 Digital (and more...)
  61. Meet the Specialists

  62. Appendix

  63. k8s helper ▶ PVC Controller: It resizes PV(C) resources when

    PVC is resized + The CSI driver's sidecar "csi-resizer" should resize the PV(C), so it seems unnecessary ▶ PV Controller: It deletes the volume associated with the deleted PV + Since the deletion process itself is performed on the CSI driver side, it seems that it is for retrying the deletion if the deletion of the volume was not successful due to some influence ▶ StorageClass Controller: It generates TridentStorageClass according to the creation of k8s Storage Class + Also, if a k8s StorageClass prior to v1 is created, a v1 k8s StorageClass will be generated ▶ Node Controller: It deletes the target TridentNode when deleting a node, and delete the target node information from the policy of each TridentBackend ▶ reconcileNodes: It compares TridentNode and k8s Node and remove that TridentNode if k8s Node does not exist ▶ handleFailedPVUpgrades: If there is a transaction for which the PV upgrade process has not been completed, stop the PV upgrade, delete the PV that was being created, and create a PV with the old settings + PV upgrade seems to be a feature operatated only with tridentctl, but the usage is unknown + PV upgrade seems to create a new PV that imports the PV from which it was upgraded and replacing it
  64. CRD Controllers ▶ reconcileBackendConfig: It converts TridentBackendConfig to TridentBackend and

    store + If the event of k8s Secret with the secret information of TridentBackendConfig occurs, it generates + TridentBackendConfig including the data of k8s secret and execute the event of reconcileBackendConfig + If the TridentBackend resource is deleted, it executes reconcileBackendConfig to regenerate the TridentBackend resource based on the associated TridentBackendConfig ▶ reconcileTMR: Set the SnapMirror according to the TridentMirrorRelationship resource and update the status of the TridentMirrorRelationship ▶ handleTridentSnapshotInfo: It gets the volumeSnapshot(Content) of k8s from the snapshotName stored in the TridentSnapshotInfo resource, and stores the SnapshotHandle(≒Snapshot ID) in the status of TridentSnapshotInfo + However, the TridentSnapshotInfo resource doesn't seem to be operated at all elsewhere, so it seems unlikely that this recocile loop will work in the first place
  65. Other Controllers ▶ TransactionMonitor: An execution processing task management Controller

    operating TridentTransaction resource for managing volume creation etc ▶ PeriopdicallyReconcileNodeAccessOnBackends: A Controller that checks that the appropriate policy settings are done so that each node and each TridentBackend can connect
  66. nodePrep ▶ A feature that automatically installs packages required for

    NFS/iSCSI -> Starts Service (looks like a beta feature) ▶ Supported Linux distributions are as follows: + Ubuntu + RHEL/CentOS ▶ Packages to be installed + NFS + Ubuntu: nfs-common + RHEL/CentOS: nfs-utils + iSCSI + Ubuntu: lsscsi, sg3-utils, scsitools, open-iscsi, multipath-tools + RHEL/CentOS: lsscsi, sg3_utils, iscsi-initiator-utils, device-mapper-multipath ▶ Service to be started + NFS + rpc-statd + iSCSI: + Ubuntu: iscsid, multipathd + RHEL/CentOS: iscsid, open-iscsi, multipathd
  67. CSI driver mount procedure (NFS) ▶ NodeStageVolume: + If nodePrep

    process is enabled, it installs package etc + It writes information such as mountOptions, NFS Server IP, NFS Path as a file named + volumePublishInfo.json + It creates this file in the path for the target volume provided by a Kubernetes CSI Sidecar ▶ NodePublishVolume: + It gets information from volumePublishInfo.json + It creates a mount destination directory + It mounts an NFS volume with the “mount -t nfs” command
  68. CSI driver mount procedure(iSCSI) ▶ NodeStageVolume: + If nodePrep process

    is enabled, it installs package etc. + It creates an iSCSI target with the iscsiadm command and log in + It scans the path to a particular LUN and waits for all SCSI disk-by-paths for that LUN to be created + It waits for the multipath device to be created + After that, if a file system other than Raw Block Volume is specified, it formats. + Available file systems: xfs / ext3 / ext4 ▶ NodePublishVolume: + It mount the device with the mount command
  69. Storage Pool ▶ Trident's Storage Pool is a pool of

    resources for which PV is assigned ▶ This Storage Pool is categorized as follows + Physical Storage Pool + Virtual Storage Pool ▶ Physical Storage Pool is an Aggregator for ONTAP + Aggregator = RAID bundled to improve storage performance and scalability ▶ Virtual Storage Pool + One Physical Storage Pool made into multiple pools + It set up multiple different Virtual Storage Pools such as IOPS and assign them according to the required requirements + It combines multiple Physical Storage Pools into one Pool + Physical Storage Pools with different physical arrangements and network topologies can be combined into one Virtual Storage Pool and scheduled to the appropriate Topology
  70. TridentMirrorRelationshipとSnap Mirror ▶ You can use the TridentMirrorRelationship resource to

    set the SnapMirror for a volume ▶ However, it seems that this function is still under development and there is no document etc ▶ You can mirror data with other volumes like SnapMirror = rsync ▶ Basically, the log of the written contents is added by Incremental Forever ▶ It's also possible to mirror the entire Storage VM (which seems impossible with Trident) ▶ It is also possible to transfer to S3 (object storage that can use the protocol) called SnapMirror Cloud