Does Kubernetes is really easy than VM for cluster administrator.

Kubernetes becomes more and more popular and few started to migrate service from VM-based infrastructure to container-based infrastructure without considering the difference between VM and containers.
The slides introduce the difference between the VM and container, and what is the challenge for the developer and cluster admin when everything becomes the container-based orchestration.
It covers the management, performance, maintenance...etc

Transcript

1. 你相信服務上 Kubernetes 真的比 VM 好管理? HungWei Chiu 04/26/2024
2. About • HungWei Chiu (hwchiu) • https://www.hwchiu.com
3. KCD Taipei 2024 CALL FOR PAPER • Push Yourself • Both Mandarin and English are acceptable • Linux Foundation will give you a crudely badge for recognize • CFP Deadline: 2024/05/09 • Check COSCUP 2024
4. KCD Taipei 2024 CALL FOR PAPER
5. Agenda • Why Kubernetes • Application deployment model • Pain point of running Kubernetes application • Evaluation
6. Why Kubernetes • Best Practice? • Trend? • Container-based orchestration? • Simple than VM orchestration?
7. Broadcom/VMWare
8. Application Deployment Model • Application on BM • Application/Container on VM on BM • Container on K8s BM • Container on K8s on VM on BM
9. Application Deployment Model
10. K8s On VM Performance • CERN (歐洲核 子 研究委員會) • Virtualization gave 20% overhead, with tuning was reduced to 5%, to bare metal would get this to 0% • How about Cloud Providers? • They have optimized the VM provisioning, better than the common KVM + QEMU. • AWS: Nitro • GCP: Rewrite QEMU https://www.cncf.io/case-studies/cern/
11. Deployment Approaches • VM • On-Prem • KVM (Virus + Libvirtd) • Proxmox • OpenStack • Kubevirt • VMWare • Cloud • VM Service • K8s • Cloud • K8s service • Build K8s on top of VM Service • On-Prem • Distribution • Vanila • Rancher • OKD • EKS Anywhere • Canonical • k3s • K0s • Openshift, Tnazu • K8s • Tools • Kubeadm • Kops • Minikube • Kubespray • Kubeeasz
12. How To Deploy Kubernetes
13. Deployment Approaches • Container on K8s on BM • Better performance • Container on K8s on VM on BM • Better multi tenancy and resource management • But you need a VM orchestration • How to choose?
14. Paint Point of Using Kubernetes • Management • Cluster administrators • Developer • Application performance
15. Management - Cluster Admin • More open source projects to consider • Rich ecosystem • Few projects were designed to work better in Kubernetes • Longhorn • Istio/Cilium • ArgoCD • …etc
16. Management - Cluster Admin • https://github.com/collabnix/kubetools
17. Management - Cluster Admin • Lack of multi-tenancy management • Namespace is weak isolation, not strong as VM cluster. • Apple @ Kubeconf 2022
18. Management - Developer • How to deploy application before Kubernetres ? • SSH to VM to deploy application • Con f igure, Build, Compile and Run • Containerize application and deploy it by Docker • Automated by CI/CD pipeline to ease the efforts for developer
19. Management - Developer • Kubernetes introduces bunch of concepts to both developer and cluster-admin https://www.reddit.com/r/kubernetes/comments/nfbqag/kubernetes_isnt_that_hard_they_said_youll_have_no/
20. Management - Developer • Kubernetes provides the YAML-based format to simply the con f iguration • Deployment (Computing) • Service (Networking) • PVC (Storage) • Con f igMap/Secret (Con f iguration) • HAP/PDB (HA Con f iguration) • …etc • Still a challenge to developers • Developer focus on the application, not k8s operation.
21. Management - Developer • Unfriendly troubleshooting experience • kubectl exec • kubectl logs • kubectl debug • More challenging when you apply f iner-grained permission con f iguration.
22. Application Performance • Four deployment model and we compare two of them today. • Application on BM • Application/Container on VM on BM • Container on K8s BM • Container on K8s on VM on BM
23. Application Performance • From the user view. • VM • Provisioning the machine with 4C8G, and user will deploy application. • Container. • Set the resource to 4C8G to my running application(container) • Both soft and hard limit to 4C8G.
24. Application Performance • From the resource utilization view. • VM • We provisioning the VM with 4C8G • Guest OS requires some resources and application may use at most 3C6G • Container. • Application can fully utilize 4C8G resource. • Container has better resource utilization.
25. Application Performance • However, you will notice lots of Kubernetes articles which discuss the container performance issue. • Symptom: • Spike of P95, P99 latency • … etc
26. Application Performance • It’s about how container perform the soft/hard resource limit. • VM • Provisioning a VM with 4C8G, and the guest OS will handle the rest of resource management for your application. • Container • All containers shares the same host kernel; hence kernel has to protect and control each container to ensure resource management.
27. Application Performance • Container use the group (v1,v2) to limit the CPU performance and throttle the CPU resource when container hits the limit • Request -> Reserve CPU for you • Limit -> Upper bound for your CPU usage • You can specify CPU in two formats • 1 vCPU = 1000ms • 100 ms = 0.1 vCPU
28. Application Performance
29. Application Performance
30. Kubernetes Paint Point – Performance
31. Kubernetes Paint Point – Performance • Assume process needs 80 ms to complete its job
32. Kubernetes Paint Point – Performance • Two requests (80ms each), single thread vs multi threads
33. Kubernetes Paint Point – Performance • You can observe the CPU throttling via following metrics • container_cpu_cfs_throttled_seconds_total • container_cpu_cfs_periods_total • container_cpu_cfs_throttled_periods_total
34. Kubernetes Paint Point – Performance • Throttled happened != high CPU usage. • Kernel bug • Sampling issue. • Metrics are collected every 15/30 seconds (Prometheus con f ig) • Throttling happens in 100 ms time slot, hence the average CPU utilization may not able to re f lect the throttling issue.
35. Exclusive Cores • Application latency and low CPU performance compared to VM • More context switch and higher cache hit • Solution • https://github.com/kubernetes/enhancements/tree/master/keps/sig- node/3570-cpumanager • Use case • Dedicate CPU to process to have high performance, low latency.
36. Exclusive Cores
37. Swap Support • Kubernetes doesn’t support use of swap memory on Linux, until recently version • 1.22 Alpha, 1.28 Beta • https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node- swap • Use case • Improve node stability • Cgroups improved memory management and strongly recommend the use of swap • Long-running applications that swap out memory for startup phase • …etc
38. Deployment Pain Point • Any other pain points of Kubernetes deployment? • Scheduled maintenance • VM • Live migration without affecting user. • K8s • Drain nodes to redeploy all pods and you have to con f igure PDB(Pod Distribution Budget) to ensure service capacity at any time. • Unexpected Node Crash • Recover application(Pod) from node failure.
39. Scheduled Maintenance • VM Live migration • Usually in the layer2 network environment, so VM keeps the same IP address after live migration • Existing connection won’t be affected.
40. Scheduled Maintenance • K8s Drain node • Pod changes its IP after redeployed • Access via service + ingress/istio • Existing connection times out
41. Scheduled Maintenance • If your Kubernetes is deployed on the bare-metal, pod may be redeployed twice during the process • Rolling out the node upgrade for entire cluster
42. Scheduled Maintenance • No such issue in the VM-based environment, since all K8s nodes are provisioned dynamically.
43. Unexpected Node Crash • One bene f it of Kubernetes is the auto-healing, which f ixes your application when something wrong in your application/cluster • The ideal scenario is auto-heal your application when a node becomes unhealthy.
44. Unexpected Node Crash • It works well only for stateless application • Check the KEPS (sig-storage 2268) • https://github.com/kubernetes/enhancements/tree/master/keps/sig- storage/ 2268-non-graceful-shutdown
45. Unexpected Node Crash
46. Unexpected Node Crash
47. Unexpected Node Crash • How to f ix it ? • Stateless • Wait the k8s to redeploy your pods after 340 seconds or faster if modify toleration • Stateful • Manually deletes pods with “force” option and then trigger the redeployed • Implemented the automation for above operation
48. Unexpected Node Crash • You won’t fully experience the bene f its of Kubernetes if you’re migrating bunch of stateful application to K8s initially • Famous debate • Should database be deployed on Kubernetes ?
49. How To Evaluate • Infrastructure environment • Cloud • On-Prem • K8s on VM • K8s on BM • K8s distribution and cluster management tool
50. How To Evaluate • Application Type • Stateless • Stateful • Backing services • DB/Cache/MQ…etc • Move all to K8s ? • Acceptance and enthusiasm for new technologies in your team. • Kubernetes update itself 3-4 times per year • Ecosystem continually advances
51. How To Evaluate • Familiar with K8s • Certi f ication: CKAD/CKA/CKS …etc • Bonus but not a mandatory • Familiarity != know how to use YAML • Instead, you have to understand what issues these K8s features are meant to solve • KEPs • Various design documents and proposals.
52. Summary • Transitioning to Kubernetes • Straightforward but not simple • Different rhythms • Development, deployment and operations • Beware • Containerization != Success