Why and Why not of enabling swap in Kubernetes

1. Kubernetes 與 Swap 的愛恨情仇 HungWei Chiu(邱宏瑋) 10/24/2024
2. K8s Installation • We have been told to disable swap in your Kubernetes installation, otherwise you will see the following error • Running with swap on is not supported, please disable swap! Or set — fail-swap-on f lag to false • [ERROR Swap]: running with swap on is not supported, Please disable swap
3. Swap • A memory management technique • Move less frequently used memory pages to disk storage, freeing up physical memory for other applications.
4. Swap • Pros: • Enhance memory utilization • Provides additional buffer for running more applications when physical memory is insuf f icient • Increase system stability • Helps prevent system crashes due to memory shortages (OOM)
5. Swap • Cons • Performance degradation • Disk I/O is slower than accessing memory, leading to potential slowdowns • Higher latency • Accessing swapped-out memory pages includes additional delays, affecting application responsiveness.
6. Kubernetes & Swap • Original Issue • https://github.com/kubernetes/kubernetes/issues/53533
7. Kubernetes & Swap • The swap support issue was raised since K8s 1.8 (2017) • 150+- discussion • Concerns • QoS De f inition and isolation • Design Principles
8. Kubernetes & Swap • Concerns • Have to control which pod could use swap • Performance penalty for Guaranteed Pods • Guaranteed pod should have consistent performance
9. Kubernetes & Swap • Concerns • No storage isolation • Disk (I/O) access performance maybe affected by swap • Guaranteed pods should have consistent performance.
10. Kubernetes & Swap • Concerns • No storage isolation • Disk (I/O) access performance maybe affected by swap • Guaranteed pods should have consistent performance.
11. Kubernetes & Swap • Concerns • How to determine the amount of total memory? • If we set the memory.limit to 2G • How about the swap? • Should be speci f ied by user or calculating by kubelet?
12. Kubernetes & Swap • Why Swap? • Cluster administrator • Node-level performance tuning and stability for noisy neighbor issues. • Application developers • Application would bene f it from using swap memory
13. Case 1 • Improved Node Stability • Cgroups v2 improved memory management algorithm, such as oomd which strongly recommend the use of swap. • Having a small amount of swap could improve resource pressure handling and recovery.
14. Case 1 • oomd (systemd-oomd) • From Facebook • Users-pace process • Flexible policy to control the OOM behavior • Needs the swap to make it stable
15. Case2 • Long-running applications • Jave and Node runtime rely on swap for optimal performance. • Initialization logic of applications can be safely swapped out without affecting long-running application resource usage.
16. Case2 • Alternative Solutions • Memory • Request: • Limit: • Lower resource utilization if we have to be guaranteed QoS
17. Case2 • Alternative Solutions • In-place resource update (alpha since 1.27) • Issues • Pod restart • GitOps • Loop • Update yaml + PR + Merged
18. Kubernetes & Swap (User Stories) • Low footprint system • Edge devices with limited memory • Edge compute system/devices with small memory (<2G) • Cluster with nodes <4Gi memory
19. Kubernetes & Swap (User Stories) • Virtualization management overhead • Virtualize k8s workloads such as VMs launched by Kubevirt • VM comes with a management related overhead • Swap help to not request much more memory to deal with. • Live migration to make system available instead of down
20. Kubernetes & Swap (User Stories) • Virtualization management overhead • Virtualize k8s workloads such as VMs launched by Kubevirt • VM comes with a management related overhead • Swap help to not request much more memory to deal with. •
21. Kubernetes & Swap (User Stories) • Live migration to make system available instead of down • When the target server doesn’t have enough memory
22. Kubernetes & Swap • KEP-2400: Node System Swap Support • Phase • 1.22: Alpha • 1.28: Beta1 • 1.30: Beta2 • Number of cases that would bene f it from swap. • Design principles?
23. Kubernetes & Swap • Scenarios • Swap is enabled on a node’s host system, but kubelet doesn’t not permit K8s workloads to use swap -> case 1 • Kubernetes can permit K8s workload scheduled on the node to use some quantity of swap, depending on the con f iguration -> case 2
24. Kubernetes & Swap • Proposal • QoS Issue • Enable swap support only for Burstable QoS Pods • No guaranteed • Performance penalty. • No Best-Effort • Low-priority pods that are f irst to be killed during node press.
25. Kubernetes & Swap • Proposal • Memory limit con f iguration • De f ine a format to calculate • Performance penalty • Take care by yourself :)
26. Kubernetes & Swap • According to the proposal • TotalPodsSwapAvailable • Total - system reserved • ContainerMemoryProportion • Request/Total • Example • Proportion = 0.5, 0.25 • Swap Limit is • 0.5 * 38 = 19 GB • 0.25 * 38 = 9.5 GB
27. Kubernetes & Swap • According to the code (master branch in 10/24/2024) • TotalPodsSwapAvailable • Total • ContainerMemoryProportion • Request/Total • Example • Proportion = 0.5, 0.25 • Swap Limit is • 0.5 * 38 = 19 GB • 0.25 * 38 = 9.5 GB
28. Kubernetes & Swap kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go
29. Kubernetes & Swap google/cadvisor/machine/machine.go
30. Kubernetes & Swap • Proposal is proposal • Code will tell you the truth
31. Kubernetes & Swap • Goals • Kubelet can start up with swap on • Case 1 or 2 • Kubelet is able to set swap utilization to K8s workloads, default to 0 swap. • Use a swap memory for cgroups v2 • I/O isolation support
32. Kubernetes & Swap • Non-Goals • Support non-Linux OS • Provisioning swap from K8s con f iguration. • Setting swappiness from K8s • Allocate swap on per-workload basis with accounting. • Support cgroup v1
33. Kubernetes & Swap(Best Practices) • Disable swap for system critical daemons(kubelet) • Setting the cgroup for the system slice to avoid swap. • cgroup v2 • memory.swap.max = 0 • Setting up the io.latency • cgroup v2 • io.latency to adjust the priority, should higher than workload
34. Kubernetes & Swap(Best Practices) • Control Plane Swap • Only enable swap for worker nodes. • Control plane contains mostly guaranteed QoS Pods • Performance concern
35. Kubernetes & Swap(Best Practices) • Dedicated disk for swap • Using a separate disk for your swap partition. • Workload will interfere the swap performance if all are in the same disk. • Same as previous concern
36. Kubernetes & Swap(Best Practices) • Swap as the default. • Enabling swap on nodes is advanced con f iguration which needs the understanding of the Linux system. • —fail-swap-on=true still the 1st option for most cases of K8s
37. Kubernetes Swap • In 1.28 • Kubelet can’t run on node with swap on by default. • Swap behavior is UnlimitedSwap by default if we enable it. • Two options • LimitedSwap • Only Burstable Pod can use it • UnlimitedSwap • Allow for all pods.
38. Kubernetes Swap • In 1.30 • Kubelet supports to be running on node with swap on. • Swap behavior is NoSwap by default. • NoSwap • NoSwap means no k8s workload can use swap, as no-swap experience, but system can have swap for other use case. • LimitedSwap • Only burstalbe Pods can use it.
39. Summary • Swap and other features (CPU Manager..etc) • Kubernetes support these features • Controlled by application itself • Who will give them up if we can gain bene f it from them? • I do only care my application
40. Summary • More and more features are developed to improve the performance and stability. • Those are natural properties in the virtual machine environment • Container’s lightweight isolation make us rework again. • Still not able to replace all VM’s environment unless you know nothing about OS tuning