Kubernetes Upgrade Operator + Cluster API: Managing upgrade version of multiple Kubernetes Clusters.

The methodologies for managing Kubernetes clusters using Cluster API and managing upgrades of provider-specific components and providing recommendations to users to minimize disruptions during the upgrade process.

Transcript

1. Viettel Cloud Version 1.0 Kubernetes Upgrade Operator + Cluster API: Managing upgrade version of multiple cluster. Hoang Dinh Nguyen
2. About me Hoang Dinh Nguyen works as a Cloud Solution Engineer at Viettel Group, where he is responsible for developing solutions for cloud platform services (Viettel Kubernetes Engine, Cloud Observability, and Internal Developer Platform) on Viettel Public Cloud. Also as approver of KusionStack Community - Kusion (Platform Orchestrator) a first-ever open-source project that included in both the CNCF Sandbox and the Platform Tooling Landscape. email: hoangndst@gmail.com linkedin: https://www.linkedin.com/in/hoangndst telegram: https://t.me/hoangndst
3. Viettel Cloud Version 1.0 TOC Introduction Best way to upgrade a Kubernetes Cluster Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator - Rolling Upgrade - Blue Green Upgrade
4. 1. Introducing Cluster API (CAPI) • Kubernetes sub-project • Started by the Kubernetes Special Interest Group (SIG) Cluster Lifecycle • Provisioning and operating multiple Kubernetes clusters. • Extend to support any infrastructure • Become core component of leading technology companies
5. 1. Introducing The importance of keeping the Kubernetes cluster up to date. • Security • Stability and Reliability • Feature Enhancements • Compatibility • Community Support
6. 2. Best way to upgrade a Kubernetes Cluster How to upgrade Kubernetes version? Upgrade the control plane. Upgrade the worker nodes. 1. Update any resources that aren't supported in the target version. 1. Cordon and drain a worker node. 2. Upgrade etcd (if needed). 2. Update kubelet configuration (if needed). 3. Upgrade kube-apiserver. 3. Upgrade the kubelet. 4. Upgrade kube-controller-manager. 4. Uncordon the node. 5. Upgrade kube-scheduler. 5. Repeat for each node in the cluster. 6. Upgrade your CNI plugin (if needed). 7. Upgrade provider-specific components (e.g., cloud-controller-manager , CSI driver). 1. Upgrade kubelet and kubectl.
7. 2. Best way to upgrade a Kubernetes Cluster Upgrade with node replacement Upgrade the control plane. Upgrade the worker nodes. 1. Update any resources that aren't supported in the target version. 1. Cordon and drain a worker node. 2. Provision a new control plane node. 2. Provision a new node. 3. Remove the old control plane node. 3. Remove the old node. 4. Repeat for each node in the cluster.
8. 2. Best way to upgrade a Kubernetes Cluster Advantages of node replacement 1. Fewer steps to manage - good for automation 2. Clean slate - no chance for configuration drift 3. Same process work for all release type => Node replacement upgrade keeping your Kubernetes cluster up-to-date and reliable with minimal disruption.
9. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator Cluster API Concepts
10. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator Cluster API Concepts
11. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator How Cluster API provision a cluster
12. 3. Integration of Automatic Upgrade Strategies with CAPI Integrate Rolling Upgrade with CAPI - Mishy Operator apiVersion: platform.a1y.io/v1alpha1 kind: RollingUpgrade metadata: name: rollingupgrade-gcp namespace: gcp spec: clusterName: gke-cluster-01 version: v1.25.10 machineTemplate: instanceType: n1-standard-1 image: ubuntu-2004-lts strategy: maxSurge: 3 maxUnavailable: 0 apiVersion: platform.a1y.io/v1alpha1 kind: RollingUpgrade metadata: name: rollingupgrade-aws namespace: aws spec: clusterName: eks-cluster-01 version: v1.25.10 machineTemplate: iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io instanceType: t3.medium sshKeyName: eks-cluster-01 strategy: maxSurge: 3 maxUnavailable: 0
13. 3. Integration of Automatic Upgrade Strategies with CAPI Integrate Rolling Upgrade with CAPI - Mishy Operator
14. 3. Integration of Automatic Upgrade Strategies with CAPI Integrate Rolling Upgrade with CAPI - Mishy Operator 1. 2. 3. Replace node one-by-one is slow Workload can get stuck draining Upgrade need to be expedient
15. 3. Integration of Automatic Upgrade Strategies with CAPI Integrate Rolling Upgrade with CAPI - Mishy Operator Rolling Upgrade config: maxSurge, maxUnavailable 1. maxSurge: a. Creates a new surge node before removing an existing one b. Choose the maximum number of additional nodes that can be added to the node pool during an upgrade 1. maxUnavailable: a. Makes an existing node unavailable to recreate it b. Choose the maximum number of nodes that can be simultaneously unavailable during an upgrade
16. 3. Integration of Automatic Upgrade Strategies with CAPI Integrate Rolling Upgrade with CAPI - Mishy Operator
17. 3. Integration of Automatic Upgrade Strategies with CAPI Integrate Rolling Upgrade with CAPI - Mishy Operator
18. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator Strategy Configuration Description Configuration Typical use case Balanced (Default), slower but least disruptive maxSurge=1 maxUnavailable=0 Most workloads Fast, no surge resources, most disruptive maxSurge=0 maxUnavailable=10 Large node pools after jobs have to run to completion Fast, most surge resources and less disruptive maxSurge=10 maxUnavailable=0 Large node pools Slowest, disruptive, no surge resources maxSurge=0 maxUnavailable=1 Resource-constrained node pool with reservation
19. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator The other side of the coin 1. Performed in a rolling fashion 2. Less granular control over workload disruption 3. Cannot rollback after upgrade 4. What if workloads are disruption sensitive? 5. Want to have more control to mitigate risks during an upgrade
20. 3. Integration of Automatic Upgrade Strategies with CAPI Intergate Blue Green Upgrade with CAPI - Mishy Operator apiVersion: platform.a1y.io/v1alpha1 kind: BlueGreenUpgrade metadata: name: bluegreenupgrade-gcp namespace: gcp spec: clusterName: gke-cluster-01 version: v1.25.10 machineTemplate: instanceType: n1-standard-1 image: ubuntu-2004-lts soakTime: 2h0m0s apiVersion: platform.a1y.io/v1alpha1 kind: BlueGreenUpgrade metadata: name: bluegreenupgrade-aws namespace: aws spec: clusterName: eks-cluster-01 version: v1.25.10 machineTemplate: iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io instanceType: t3.medium sshKeyName: eks-cluster-01 soakTime: 2h0m0s
21. 3. Integration of Automatic Upgrade Strategies with CAPI Intergate Blue Green Upgrade with CAPI - Mishy Operator
22. 3. Integration of Automatic Upgrade Strategies with CAPI Intergate Blue Green Upgrade with CAPI - Mishy Operator
23. 3. Integration of Automatic Upgrade Strategies with CAPI Intergate Blue Green Upgrade with CAPI - Mishy Operator
24. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator Rolling back to previous version
25. 3. Integration of Automatic Upgrade Strategies with CAPI - Mishy Operator Rolling back to previous version
26. Comments, Questions or Suggestions for Mishy Operator? hoangndst@gmail.com
27. Thank you. Hoang Dinh Nguyen (Steve Techs) email: hoangndst@gmail.com linkedin: https://www.linkedin.com/in/hoangndst telegram: https://t.me/hoangndst