Virtual_Kubernetes_Clusters__Tips_and_Tricks_-_Rejekts.pdf

A lot of interest in virtual Kubernetes clusters and the open source tool vcluster has developed over the last year. vcluster allows platform teams to provide virtual Kubernetes clusters to their users. A virtual cluster appears to be a full-blown Kubernetes cluster to the users, but it runs within a namespace of the host cluster. This allows users to have admin access to the cluster, use multiple namespaces in it, and manage global objects like CRDs.

During the last year, many new features have been added to vcluster, and we’ve seen it used for use cases that we hadn’t even imagined. This talk will provide tips and tricks to help teams get more from their virtual clusters and show off some fun things you can do with them.

We’ll cover: How to share resources like ingresses from the host cluster, using vcluster’s isolated mode to automatically add network policies and Pod Security Standards to your virtual clusters, pausing and resuming virtual clusters, and writing plugins with the vcluster SDK. We’ll also cover some weirder examples like using vcluster for shadow IT (users don’t need to have elevated privileges in the host cluster to start a virtual cluster) and running a virtual cluster inside a virtual cluster.

Transcript

1. Virtual Kubernetes Clusters: Tips and Tricks Rich Burroughs Loft Labs, Inc.
2. Virtual Kubernetes Clusters: Tips and Tricks Rich Burroughs Loft Labs, Inc.
3. 10 Virtual Cluster Tips and Tricks That Will Surprise You Rich Burroughs Loft Labs, Inc.
4. 10 Virtual Cluster Tips and Tricks That Will Surprise You Rich Burroughs Loft Labs, Inc.
5. 10 vcluster Tricks That Will Make You a Cloud Native Rockstar Rich Burroughs Loft Labs, Inc.
6. Hi I’m Rich 6 Rich Burroughs Staff Developer Advocate Loft Labs, Inc. ❏ Creator/host of the Kube Cuddle podcast ❏ Worked in Ops/SRE for 20+ years ❏ I have seen some things ❏ @richburroughs on Twitter/GH/LI
7. @richburroughs 7 What we’ll cover ▸ The why and what of virtual clusters ▸ vcluster Tips and Tricks ▸ Use cases for virtual clusters
8. @richburroughs 8 Multi-tenancy with Kubernetes is hard ▸ Two Kubernetes multi-tenancy models: ▹ Namespace-based isolation ▹ Cluster-based isolation
9. @richburroughs 9 Namespace-based isolation Tenants are restricted to one or more namespaces using RBAC and network policies. ▸ Pros: ▹ Cuts down on cluster sprawl ▹ Less wasted resources ▸ Cons: ▹ Users can’t manage global objects like CRDs ▹ Users may need multiple namespaces ▹ Environments can get very complex
10. @richburroughs 10 Cluster-based isolation Tenants each get their own dedicated cluster. ▸ Pros: ▹ Better isolation ▹ Less complexity in each cluster ▸ Cons: ▹ Difficult to manage a lot of clusters ▹ Wasted resources ▹ Cost
11. @richburroughs 11 11
12. @richburroughs 12 12
13. @richburroughs 13 Virtual Kubernetes clusters A virtual cluster runs inside of a shared, host cluster, but appears to the user as if it’s a standalone, dedicated cluster.
14. @richburroughs 14 vcluster ▸ Open source project launched in April of 2021 ▸ Currently the most popular implementation of virtual clusters ▸ Certified Kubernetes distribution ▸ Fast and easy to use
15. @richburroughs 15 How does vcluster work? ▸ Virtual cluster runs in a namespace on a host cluster ▸ vcluster contains a k8s API server and other tools ▸ Saves state in a DB (SQLlite by default)
16. EKS Cluster Context Namespace in EKS Virtual Cluster Context Virtual Cluster Architecture vcluster runs a Kubernetes control plane inside a container EKS Control Plane API Server Data Store. (etcd, sqlite, …) API Server etcd Scheduler Namespace in virtual cluster deployment pod-1 custom-resource synced-pod-1 vcluster syncer Admin Tenant connects to EKS API server controls EKS context connects to vcluster API server controls vcluster context vcluster-pod Controller Manager Controller Manager
17. @richburroughs 17 Client installation ▸ Go binary ▸ Installs via Homebrew on Macs or download it from GitHub ▸ Signed with cosign (thanks Batuhan Apaydın / @developer-guy!)
18. @richburroughs 18 Demo
20. @richburroughs 20 Tips and Tricks
21. @richburroughs 21 1. vcluster distros ▸ The distro is the Kubernetes installed inside the virtual cluster ▸ Default is k3s which should be fine for most uses ▸ Can specify alternate distros with the --distro flag ▹ k0s ▹ eks (thanks Justin Garrison for the PR!) ▹ k8s (standard Kubernetes)
22. @richburroughs 22 2. Creating virtual clusters ▸ 3 ways to create virtual clusters ▹ vcluster create command ▹ Helm install ▹ Cluster API (CAPI) provider
23. @richburroughs 23 Helm install ▸ Used by vcluster create ▸ Great for GitOps workflows
25. @richburroughs 25 Cluster API provider ▸ Cluster API is used to provision k8s clusters and manage lifecyle ▸ Requires clusterctl ▸ Creates vclusters in the local cluster
28. @richburroughs 28 3. Exposing a vcluster ▸ vcluster connect uses port forwarding for remote clusters ▸ NodePort (used by vcluster connect for local clusters) ▸ Load balancer: vcluster create --expose ▸ Ingress
29. @richburroughs 29 4. Inception ▸ You can run a vcluster inside a vcluster ▸ Mike Tougeron’s talk was about this ▸ I have no idea if there are other legit use cases for this but it’s fun to do
30. @richburroughs 30 5. Isolated Mode ▸ Creates additional isolation for the vcluster ▸ Enable with vcluster create --isolate ▸ Creates these objects: ▹ Pod Security Standard ▹ Resource quota and limit range ▹ Network policy (if the CNI supports them)
31. @richburroughs 31 6. Pausing and resuming ▸ Quick way to suspend workloads ▸ Sets the replicas to 0 ▸ vcluster pause spins down all the pods ▸ vcluster resume brings them back up
32. @richburroughs 32 7. Applying manifests and charts ▸ Apply manifests and/or charts automatically when a vcluster is created ▸ You specify the manifests and charts in the values.yaml file ▸ You can apply YAML, public Helm charts, or even private charts
34. @richburroughs 34 8. Shadow IT ▸ Thanks to Bret Fisher’s audience for this trick ▸ If you have access to a namespace on a shared cluster, you can probably run vcluster ▸ Administrators will see the vcluster pod and the workload pods ▸ Consult your company’s policies before trying this at home
35. @richburroughs 35 9. Plugins ▸ Change the behavior of the syncer ▸ Written in Go ▸ You can do lots of things with plugins
38. @richburroughs 38 10. High Availability ▸ Creates multiple copies of the vcluster components ▸ Works with the k8s distro ▸ You specify the settings in values.yaml ▸ Example: vcluster create rejekts --connect=false --distro k8s -f values.yaml
40. @richburroughs 40 Use cases ▸ Dev environments ▸ CI/CD pipelines ▸ Testing Kubernetes resources
41. @richburroughs 41 Use case 1: Dev environments ▸ Requirements: ▹ Local or remote Kubernetes cluster ▹ Self-service ▹ Fast ▸ Challenges: ▹ Some developers don’t want to be a k8s admin ▹ What if an environment breaks? ▸ How can vcluster help? ▹ Creating and deleting clusters takes seconds ▹ You can easily pause and resume workloads or start over from scratch ▹ Works with local and remote clusters
42. @richburroughs 42 Use case 2: CI/CD pipelines ▸ Requirements: ▹ Creating and destroying clusters ▹ Quick provisioning ▹ Automation ▸ Challenges: ▹ Clusters may be provisioned many times in a test suite ▹ Speed issues can impact feedback time and time to push out changes ▸ How can vcluster help? ▹ You can provision clusters in seconds ▹ You could run tests concurrently in separate virtual clusters ▹ Easy to automate vcluster creation with Helm or CAPI
43. @richburroughs 43 Use case 3: Testing k8s resources ▸ Requirements: ▹ Testing resources across Kubernetes versions ▹ Testing k8s upgrades ▹ Could be ad hoc ▸ Challenges: ▹ Hardware requirements/cost ▹ Provisioning time ▸ How can vcluster help? ▹ You can specify the distro version with --version ▹ Could test multiple versions in parallel
44. @richburroughs 44 There are many more use cases Questions to ask: Does your thing run in a Kubernetes cluster? Do you need to create and destroy clusters quickly? Are you concerned about cost?
45. @richburroughs 45 Resources ▸ Getting started: https://www.vcluster.com ▸ GitHub: https://github.com/loft-sh/vcluster ▸ vcluster SDK: https://github.com/loft-sh/vcluster-sdk ▸ vcluster plugins: https://github.com/loft-sh/vcluster-plugins ▸ Cluster API provider: https://github.com/loft-sh/cluster-api-provider-vcluster ▸ Community Slack: https://slack.loft.sh
46. @richburroughs 46 We’re hiring https://loft.sh/careers/
47. @richburroughs 47 Thank you ▸ Thank you to Oleg and Carl from our engineering team ▸ Also the rest of the vcluster maintainers and contributors ▸ Thanks organizers! ▸ Thank you for watching