Tweet
Tweet

Slide 1

Slide 1 text

你相信服務上 Kubernetes 真的比 VM 好管理? HungWei Chiu 04/26/2024

Slide 2

Slide 2 text

About • HungWei Chiu (hwchiu) • https://www.hwchiu.com

Slide 3

Slide 3 text

KCD Taipei 2024 CALL FOR PAPER • Push Yourself • Both Mandarin and English are acceptable • Linux Foundation will give you a crudely badge for recognize • CFP Deadline: 2024/05/09 • Check COSCUP 2024

Slide 4

Slide 4 text

KCD Taipei 2024 CALL FOR PAPER

Slide 5

Slide 5 text

Agenda • Why Kubernetes • Application deployment model • Pain point of running Kubernetes application • Evaluation

Slide 6

Slide 6 text

Why Kubernetes • Best Practice? • Trend? • Container-based orchestration? • Simple than VM orchestration?

Slide 7

Slide 7 text

Broadcom/VMWare

Slide 8

Slide 8 text

Application Deployment Model • Application on BM • Application/Container on VM on BM • Container on K8s BM • Container on K8s on VM on BM

Slide 9

Slide 9 text

Application Deployment Model

Slide 10

Slide 10 text

K8s On VM Performance • CERN (歐洲核 子 研究委員會) • Virtualization gave 20% overhead, with tuning was reduced to 5%, to bare metal would get this to 0% • How about Cloud Providers? • They have optimized the VM provisioning, better than the common KVM + QEMU. • AWS: Nitro • GCP: Rewrite QEMU https://www.cncf.io/case-studies/cern/

Slide 11

Slide 11 text

Deployment Approaches • VM • On-Prem • KVM (Virus + Libvirtd) • Proxmox • OpenStack • Kubevirt • VMWare • Cloud • VM Service • K8s • Cloud • K8s service • Build K8s on top of VM Service • On-Prem • Distribution • Vanila • Rancher • OKD • EKS Anywhere • Canonical • k3s • K0s • Openshift, Tnazu • K8s • Tools • Kubeadm • Kops • Minikube • Kubespray • Kubeeasz

Slide 12

Slide 12 text

How To Deploy Kubernetes

Slide 13

Slide 13 text

Deployment Approaches • Container on K8s on BM • Better performance • Container on K8s on VM on BM • Better multi tenancy and resource management • But you need a VM orchestration • How to choose?

Slide 14

Slide 14 text

Paint Point of Using Kubernetes • Management • Cluster administrators • Developer • Application performance

Slide 15

Slide 15 text

Management - Cluster Admin • More open source projects to consider • Rich ecosystem • Few projects were designed to work better in Kubernetes • Longhorn • Istio/Cilium • ArgoCD • …etc

Slide 16

Slide 16 text

Management - Cluster Admin • https://github.com/collabnix/kubetools

Slide 17

Slide 17 text

Management - Cluster Admin • Lack of multi-tenancy management • Namespace is weak isolation, not strong as VM cluster. • Apple @ Kubeconf 2022

Slide 18

Slide 18 text

Management - Developer • How to deploy application before Kubernetres ? • SSH to VM to deploy application • Con f igure, Build, Compile and Run • Containerize application and deploy it by Docker • Automated by CI/CD pipeline to ease the efforts for developer

Slide 19

Slide 19 text

Management - Developer • Kubernetes introduces bunch of concepts to both developer and cluster-admin https://www.reddit.com/r/kubernetes/comments/nfbqag/kubernetes_isnt_that_hard_they_said_youll_have_no/

Slide 20

Slide 20 text

Management - Developer • Kubernetes provides the YAML-based format to simply the con f iguration • Deployment (Computing) • Service (Networking) • PVC (Storage) • Con f igMap/Secret (Con f iguration) • HAP/PDB (HA Con f iguration) • …etc • Still a challenge to developers • Developer focus on the application, not k8s operation.

Slide 21

Slide 21 text

Management - Developer • Unfriendly troubleshooting experience • kubectl exec • kubectl logs • kubectl debug • More challenging when you apply f iner-grained permission con f iguration.

Slide 22

Slide 22 text

Application Performance • Four deployment model and we compare two of them today. • Application on BM • Application/Container on VM on BM • Container on K8s BM • Container on K8s on VM on BM

Slide 23

Slide 23 text

Application Performance • From the user view. • VM • Provisioning the machine with 4C8G, and user will deploy application. • Container. • Set the resource to 4C8G to my running application(container) • Both soft and hard limit to 4C8G.

Slide 24

Slide 24 text

Application Performance • From the resource utilization view. • VM • We provisioning the VM with 4C8G • Guest OS requires some resources and application may use at most 3C6G • Container. • Application can fully utilize 4C8G resource. • Container has better resource utilization.

Slide 25

Slide 25 text

Application Performance • However, you will notice lots of Kubernetes articles which discuss the container performance issue. • Symptom: • Spike of P95, P99 latency • … etc

Slide 26

Slide 26 text

Application Performance • It’s about how container perform the soft/hard resource limit. • VM • Provisioning a VM with 4C8G, and the guest OS will handle the rest of resource management for your application. • Container • All containers shares the same host kernel; hence kernel has to protect and control each container to ensure resource management.

Slide 27

Slide 27 text

Application Performance • Container use the group (v1,v2) to limit the CPU performance and throttle the CPU resource when container hits the limit • Request -> Reserve CPU for you • Limit -> Upper bound for your CPU usage • You can specify CPU in two formats • 1 vCPU = 1000ms • 100 ms = 0.1 vCPU

Slide 28

Slide 28 text

Application Performance

Slide 29

Slide 29 text

Application Performance

Slide 30

Slide 30 text

Kubernetes Paint Point – Performance

Slide 31

Slide 31 text

Kubernetes Paint Point – Performance • Assume process needs 80 ms to complete its job

Slide 32

Slide 32 text

Kubernetes Paint Point – Performance • Two requests (80ms each), single thread vs multi threads

Slide 33

Slide 33 text

Kubernetes Paint Point – Performance • You can observe the CPU throttling via following metrics • container_cpu_cfs_throttled_seconds_total • container_cpu_cfs_periods_total • container_cpu_cfs_throttled_periods_total

Slide 34

Slide 34 text

Kubernetes Paint Point – Performance • Throttled happened != high CPU usage. • Kernel bug • Sampling issue. • Metrics are collected every 15/30 seconds (Prometheus con f ig) • Throttling happens in 100 ms time slot, hence the average CPU utilization may not able to re f lect the throttling issue.

Slide 35

Slide 35 text

Exclusive Cores • Application latency and low CPU performance compared to VM • More context switch and higher cache hit • Solution • https://github.com/kubernetes/enhancements/tree/master/keps/sig- node/3570-cpumanager • Use case • Dedicate CPU to process to have high performance, low latency.

Slide 36

Slide 36 text

Exclusive Cores

Slide 37

Slide 37 text

Swap Support • Kubernetes doesn’t support use of swap memory on Linux, until recently version • 1.22 Alpha, 1.28 Beta • https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node- swap • Use case • Improve node stability • Cgroups improved memory management and strongly recommend the use of swap • Long-running applications that swap out memory for startup phase • …etc

Slide 38

Slide 38 text

Deployment Pain Point • Any other pain points of Kubernetes deployment? • Scheduled maintenance • VM • Live migration without affecting user. • K8s • Drain nodes to redeploy all pods and you have to con f igure PDB(Pod Distribution Budget) to ensure service capacity at any time. • Unexpected Node Crash • Recover application(Pod) from node failure.

Slide 39

Slide 39 text

Scheduled Maintenance • VM Live migration • Usually in the layer2 network environment, so VM keeps the same IP address after live migration • Existing connection won’t be affected.

Slide 40

Slide 40 text

Scheduled Maintenance • K8s Drain node • Pod changes its IP after redeployed • Access via service + ingress/istio • Existing connection times out

Slide 41

Slide 41 text

Scheduled Maintenance • If your Kubernetes is deployed on the bare-metal, pod may be redeployed twice during the process • Rolling out the node upgrade for entire cluster

Slide 42

Slide 42 text

Scheduled Maintenance • No such issue in the VM-based environment, since all K8s nodes are provisioned dynamically.

Slide 43

Slide 43 text

Unexpected Node Crash • One bene f it of Kubernetes is the auto-healing, which f ixes your application when something wrong in your application/cluster • The ideal scenario is auto-heal your application when a node becomes unhealthy.

Slide 44

Slide 44 text

Unexpected Node Crash • It works well only for stateless application • Check the KEPS (sig-storage 2268) • https://github.com/kubernetes/enhancements/tree/master/keps/sig- storage/ 2268-non-graceful-shutdown

Slide 45

Slide 45 text

Unexpected Node Crash

Slide 46

Slide 46 text

Unexpected Node Crash

Slide 47

Slide 47 text

Unexpected Node Crash • How to f ix it ? • Stateless • Wait the k8s to redeploy your pods after 340 seconds or faster if modify toleration • Stateful • Manually deletes pods with “force” option and then trigger the redeployed • Implemented the automation for above operation

Slide 48

Slide 48 text

Unexpected Node Crash • You won’t fully experience the bene f its of Kubernetes if you’re migrating bunch of stateful application to K8s initially • Famous debate • Should database be deployed on Kubernetes ?

Slide 49

Slide 49 text

How To Evaluate • Infrastructure environment • Cloud • On-Prem • K8s on VM • K8s on BM • K8s distribution and cluster management tool

Slide 50

Slide 50 text

How To Evaluate • Application Type • Stateless • Stateful • Backing services • DB/Cache/MQ…etc • Move all to K8s ? • Acceptance and enthusiasm for new technologies in your team. • Kubernetes update itself 3-4 times per year • Ecosystem continually advances

Slide 51

Slide 51 text

How To Evaluate • Familiar with K8s • Certi f ication: CKAD/CKA/CKS …etc • Bonus but not a mandatory • Familiarity != know how to use YAML • Instead, you have to understand what issues these K8s features are meant to solve • KEPs • Various design documents and proposals.

Slide 52

Slide 52 text

Summary • Transitioning to Kubernetes • Straightforward but not simple • Different rhythms • Development, deployment and operations • Beware • Containerization != Success