▶ @bells17 ▶ Software Engineer@IDC Frontier inc. ▶ What I usually do : + Kubernetes Development of related components + Kubernetes as a Service Development ▶ Kubernetes SIG-Docs Japanese localization reviewer ▶ Kubernetes Internal Organizer ▶ #kubenews ▶ @bells17_
Cautionary Note ▶ The presenter is not a storage expert (especially iSCSI or NFS ..) ▶ Trident version is expected to be v21.07.2 ▶ Trident is expected to be used with the following settings: + Use Trident from Kubernetes + Trident is basically installed using Trident Operator (described later) ▶ Since it is only an explanation of understanding as a result of following the implementation of Trident, there may be cases where it differs from the actual behavior
Mr. Ohno of NetApp also explains the architecture of Trident in Cloud Native Storage Meetup # 1, so please refer to that as well(Presented in Japanese) https://youtu.be/2xEUyAzoNmY?t=3583
What is Trident? ▶ Trident is an application for taking advantage of various NetApp storage products (e.g. ONTAP, E- Series, Cloud Volumes Service for AWS, etc...) ▶ With Trident, you will be able to operate NetApp storage products in a container environment such as Kubernetes ▶ The following two platforms are currently supported: + Kubernetes ← I will talk how Trident works on Kubernetes here + Docker ▶ Trident is an OSS (https://github.com/NetApp/trident) ▶ Trident is one of the applications included in Project "Astra" ▶ Astra consists of following applications in addition to Trident + Astra Control: Kubernetes cluster management and operations console + Astra Data Store: Kubernetes native shared file service
Trident Components ▶ Trident(Core): The main body of Trident consists of a CSI Driver for cooperating with Kubernetes, a Rest API server for tridentctl, and various controllers for controlling the state of Trident ▶ tridentctl: The Command line tool for operating Trident from cli ▶ Trident Operator: Kubernetes Operator for managing Trident installations and upgrades on Kubernetes clusters + It was added from Trident v20.04.0 + You can install trident with the tridentctl install command in addition to the Trident Operator
Trident and Kubernetes ▶ Trident runs as a CSI Driver for manage various NetApp storage on Kubernetes ▶ It also implements multiple Kubernetes Controllers for Trident management ▶ Therefore, before we get into the implementation of Trident, It is better to know about the outline such as + Kubernetes Operator(Kubernetes Controller) + CSI(driver)
What is Kubernetes? ▶ Kubernetes is one of the container orchestrators ▶ You can build a cluster composed of etcd / control plane / worker node, run various containers on node on Kubernetes, and enable to link the running container and network nicely ▶ By declaratively describing the container and other resources to be deployed in the manifest file, Kubernetes will perform adjustment processing nicely so that it will be in the declared state ▶ A container orchestrator that recreated from Borg, a container platform operated internally by Google, for OSS ▶ Kubernetes has also been donated to the Cloud Native Computing Foundation(CNCF) and is managed on a community basis as a CNCF Graduated project
▶ Kubernetesのコードリーディングをする上で知っておくと良さそうなこと ▶ Kubernetes Internal #1 Click following links for details around the Kubernetes Controller implementations(Presented in Japanese)
CSI ▶ Definition of common specifications for using storage in Container Orchestrator (CO) such as Kubernetes, Mesos, Cloud Foundry, etc + So it's not a specification only for Kubernetes + For example, Hashicorp Nomad is using CSI under the hood ▶ Storage providers aim to be able to utilize Kubernetes and other COs by writing a driver that supports CSI once ▶ The CSI specification is defined in the the spec.md file in the container- storage- interface/spec repository on Github
Specifications defined by CSI ▶ The communication method and provision method of CSI Driver ▶ Features provided by CSI Driver ▶ gRPC Protocol Buffers to use CSI Driver by CO
How CSI Driver Communicates and being provided ▶ Need to be provided in container image format (Docker, OCI, etc.) ▶ Communication between CSI Driver and CO needs to ... + Use gRPC protocol + Via UNIX domain socket
Controller Plugin ▶ A gRPC server that operates as a control plane for its CSI Driver ▶ Implementation of the following gRPC services: + Controller Service + Identity Service ▶ It provides the ability to control the volume and the snapshot ▶ Specifically, it provides the following features: + Create / Delete volume + Attach / Detach volume to node + Create / Delete volume snapshot
Node Plugin ▶ A gRPC server that operates on each CO-participating Worker Node ▶ Implementation of the following gRPC services: + Node Service + Identity Service ▶ It provides the feature to operate the volume on each target Worker Node ▶ It mainly provides the following features: + Format volume attached to node + Mount / Unmount Volume
CSI wrap-up ▶ CSI is a common specification defined by storage providers to provide volume plug-ins to CO ▶ Specifically, the following are defined + Definition of operating environment and communication method of CSI Driver + Container image format + UNIX domain socket / gRCP protocol + Definition of RPC interface + Controller Plugin + Node Plugin
There are three Kubernetes volume plugins ▶ In-Tree Volume + Volume plugin implemented inside Kubernetes code + ConfigMap / Secret / EmptyDir etc. fall into here ▶ FlexVolume + Plugin created before CSI appeared (Kubernetes v1.8) + It seems that it is not used much because it requires knowledge about the internal implementation of Kubernetes + Deprecated in Recent Kubernetes Updated versions ▶ CSI Driver + Alpha is available from Kubernetes v1.9 + Kubernetes provides a sidecar application that works with the CSI Driver, so you can provide volume plugins without the need for knowledge of the internal implementation of Kubernetes
Integration by Sidecar application https://github.com/kubernetes/community/blob/d83cd53979d08ac0e0e33704c6aec6b1c3cb7c8d/contributors/design-proposals/storage/container-storage-interface_diagram1.png
Node Sidecar node-driver-registrar It provides the feature to register the CSI driver in Kubelet by using the feature called “Plugin Watcher” in Kubelet
Trident Core: Other components ▶ TransactionMonitor: Execution processing task management Controller using TridentTransaction resource for managing volume creation etc ▶ PeriopdicallyReconcileNodeAccessOnBackends: A Controller that checks that the appropriate policy settings are set so that each node and each Trident Backend can connect ▶ k8shelper + Node Controller: A Controller that deletes the target Trident Node when deleting a node and deletes the target node information from the policy of each Trident Backend ▶ CRD Controllers + reconcileTMR: Set the SnapMirror according to the TridentMirrorRelationship resource and update the status of the TridentMirrorRelationship ▶ etc
CRD operated by Trident $3%໊ ༻్ 5SJEFOU0SDIBTUSBUPS 0QFSBUFECZ5SJEFOU0QFSBUPSUPJOTUBMM5SJEFOU 5SJEFOU#BDLFOE$POpH 3FTPVSDFTGPSHFOFSBUJOH5SJEFOU#BDLFOEUIBUNBOBHFTUPSBHFJOGPSNBUJPOTVDIBT 0/5"1UIBU5SJEFOUDPOOFDUTUP 5SJEFOU#BDLFOE 0QFSBUFEUPNBOBHFTUPSBHFJOGPSNBUJPOTVDIBT0/5"1UPXIJDI5SJEFOUDPOOFDUT :PVDBODSFBUFXJUIl5SJEFOU#BDLFOE$POpHzPSlUSJEFODUMDSFBUFCBDLFOEz 5SJEFOU4UPSBHF$MBTT 3FTPVSDFTGPS5SJEFOUUPQSPDFTTXJUI,T4UPSBHF$MBTT "TGBSBTUIFJNQMFNFOUBUJPOJTDPODFSOFE
JUQSPCBCMZEPFTOUIBWFUPCFQFSTJTUFOU 5SJEFOU/PEF 3FTPVSDFTGPSTUPSJOHOPEFJOGPSNBUJPOGPS5SJEFOU (FOFSBUFEGSPN$4*/PEFBOEEFMFUFEPOUIF$4*4FSWFSTJEFXIFOUIFOPEFJT EFMFUFE 0QFSBUFEUPTFUQPMJDJFTTPUIBUFBDITFSWFSDBODPOOFDUUPWBSJPVT#BDLFOET 5SJEFOU7PMVNF 3FTPVSDFTGPSNBOBHJOHWPMVNFTEJTQFOTFEWJB5SJEFOU "TGBSBTUIFJNQMFNFOUBUJPOJTDPODFSOFE
JUQSPCBCMZEPFTOUIBWFUPCFQFSTJTUFOU 5SJEFOU7FSTJPO 3FTPVSDFTGPSTUPSJOHUIF5SJEFOUWFSTJPOSVOOJOHPOUIF$POUSPMMFSTJEF 0QFSBUFEUPDIFDLJGJUNBUDIFTUIF5SJEFOUWFSTJPOSVOOJOHPOUIF/PEFTJEF 5SJEFOU.JSSPS3FMBUJPOTIJQ 3FTPVSDFTGPSTFUUJOH4OBQ.JSSPSTFUUJOHTPOWPMVNFT 5SJEFOU5SBOTBDUJPO 3FTPVSDFTGPSNBOBHJOHUIFQSPDFTTJOHTUBUVTTVDIBTWPMVNFDSFBUJPO 1SPDFTTJOHJTSPMMFECBDLXIFOUIFFYFDVUJPOUJNFJTQSPMPOHFEFYDFQUGPSTPNF SFTPVSDFT There are other CRDs, but they are unlikely to be used
Impressions of reading the implementation ▶ It was characteristic and interesting to be able to use TridentBackend for multiple NetApp storage from one Trident ▶ On the other hand, I got the impression that it would be difficult to start using it because the setting method of TridentBackend and StorageClass is complicated ▶ There were some features and resource definitions (eg Snapshot resources, etc.) that were developed from a relatively old age or seem to be unused now. I got the impression that the implementation could be kept simple by redefining the features, implementations, and support scope of the application
k8s helper ▶ PVC Controller: It resizes PV(C) resources when PVC is resized + The CSI driver's sidecar "csi-resizer" should resize the PV(C), so it seems unnecessary ▶ PV Controller: It deletes the volume associated with the deleted PV + Since the deletion process itself is performed on the CSI driver side, it seems that it is for retrying the deletion if the deletion of the volume was not successful due to some influence ▶ StorageClass Controller: It generates TridentStorageClass according to the creation of k8s Storage Class + Also, if a k8s StorageClass prior to v1 is created, a v1 k8s StorageClass will be generated ▶ Node Controller: It deletes the target TridentNode when deleting a node, and delete the target node information from the policy of each TridentBackend ▶ reconcileNodes: It compares TridentNode and k8s Node and remove that TridentNode if k8s Node does not exist ▶ handleFailedPVUpgrades: If there is a transaction for which the PV upgrade process has not been completed, stop the PV upgrade, delete the PV that was being created, and create a PV with the old settings + PV upgrade seems to be a feature operatated only with tridentctl, but the usage is unknown + PV upgrade seems to create a new PV that imports the PV from which it was upgraded and replacing it
CRD Controllers ▶ reconcileBackendConfig: It converts TridentBackendConfig to TridentBackend and store + If the event of k8s Secret with the secret information of TridentBackendConfig occurs, it generates + TridentBackendConfig including the data of k8s secret and execute the event of reconcileBackendConfig + If the TridentBackend resource is deleted, it executes reconcileBackendConfig to regenerate the TridentBackend resource based on the associated TridentBackendConfig ▶ reconcileTMR: Set the SnapMirror according to the TridentMirrorRelationship resource and update the status of the TridentMirrorRelationship ▶ handleTridentSnapshotInfo: It gets the volumeSnapshot(Content) of k8s from the snapshotName stored in the TridentSnapshotInfo resource, and stores the SnapshotHandle(≒Snapshot ID) in the status of TridentSnapshotInfo + However, the TridentSnapshotInfo resource doesn't seem to be operated at all elsewhere, so it seems unlikely that this recocile loop will work in the first place
Other Controllers ▶ TransactionMonitor: An execution processing task management Controller operating TridentTransaction resource for managing volume creation etc ▶ PeriopdicallyReconcileNodeAccessOnBackends: A Controller that checks that the appropriate policy settings are done so that each node and each TridentBackend can connect
CSI driver mount procedure (NFS) ▶ NodeStageVolume: + If nodePrep process is enabled, it installs package etc + It writes information such as mountOptions, NFS Server IP, NFS Path as a file named + volumePublishInfo.json + It creates this file in the path for the target volume provided by a Kubernetes CSI Sidecar ▶ NodePublishVolume: + It gets information from volumePublishInfo.json + It creates a mount destination directory + It mounts an NFS volume with the “mount -t nfs” command
CSI driver mount procedure(iSCSI) ▶ NodeStageVolume: + If nodePrep process is enabled, it installs package etc. + It creates an iSCSI target with the iscsiadm command and log in + It scans the path to a particular LUN and waits for all SCSI disk-by-paths for that LUN to be created + It waits for the multipath device to be created + After that, if a file system other than Raw Block Volume is specified, it formats. + Available file systems: xfs / ext3 / ext4 ▶ NodePublishVolume: + It mount the device with the mount command
Storage Pool ▶ Trident's Storage Pool is a pool of resources for which PV is assigned ▶ This Storage Pool is categorized as follows + Physical Storage Pool + Virtual Storage Pool ▶ Physical Storage Pool is an Aggregator for ONTAP + Aggregator = RAID bundled to improve storage performance and scalability ▶ Virtual Storage Pool + One Physical Storage Pool made into multiple pools + It set up multiple different Virtual Storage Pools such as IOPS and assign them according to the required requirements + It combines multiple Physical Storage Pools into one Pool + Physical Storage Pools with different physical arrangements and network topologies can be combined into one Virtual Storage Pool and scheduled to the appropriate Topology
TridentMirrorRelationshipとSnap Mirror ▶ You can use the TridentMirrorRelationship resource to set the SnapMirror for a volume ▶ However, it seems that this function is still under development and there is no document etc ▶ You can mirror data with other volumes like SnapMirror = rsync ▶ Basically, the log of the written contents is added by Incremental Forever ▶ It's also possible to mirror the entire Storage VM (which seems impossible with Trident) ▶ It is also possible to transfer to S3 (object storage that can use the protocol) called SnapMirror Cloud
bells17
chatwork_hr
subomi
ntsk
chobishiba
tsukushi
mot_techtalk
ivargrimstad
nozaki
joker1007
akarin
maimux2x
eileencodes
jacobian
colly
chriscoyier
vanstee
bcantrill
zakiyama
maggiecrowley
philhawksworth
pedronauck
csswizardry