Kubernetes Resource Management

LinuxPiter 2019: Kubernetes Resource Management - Alexander Kanevskiy, Krisztian Litkey

Transcript

1. Kubernetes* Resource Management Alexander Kanevskiy Krisztian Litkey Intel, Finland * Other names and brands may be claimed as the property of others.
2. Agenda • The “Noisy neighbor” problem in Kubernetes* • Small detour into the history of hardware • Resources in Kubernetes*: what do we have now • What we can tweak on different levels • CRI Resource Manager • Demo * Other names and brands may be claimed as the property of others.
3. The “Noisy neighbor” problem
4. System devices topology Socket 0 Core 0 Core 1 Core 6 Core 7 Core 2 Core 3 Core 8 Core 9 Core 4 Core 5 Core 10 Core 11 PCIe UPI Socket 1 Core 0 Core 1 Core 6 Core 7 Core 2 Core 3 Core 8 Core 9 Core 4 Core 5 Core 10 Core 11 UPI PCIe Memory Controller Memory Controller Memory Controller Memory Controller $ $ $ $ $ $ $
5. Caches, RDT, MBM Thread 0 Thread 1 Core 0 L1d L1i L2 Thread 0 Thread 1 Core 1 L1d L1i L2 Thread 0 Thread 1 Core 2 L1d L1i L2 Thread 0 Thread 1 Core N L1d L1i L2 L3 Package 0
6. Memory CPU BUS CPU CPU CPU CPU CPU CPU CPU UMA (aka SMP) Uniform Memory Access NUMA Non-uniform Memory access
7. System topology in real world Node 0 Node 2 Node 1 Node 3 Package 0 Core 0 Core 1 Core 5 Core 6 Memory Controller Core 2 Core 7 Memory Controller Core 3 Core 4 Core 8 Core 9 PCIe UPI Package 1 Core 0 Core 1 Core 5 Core 6 Core 2 Core 7 Core 3 Core 4 Core 8 Core 9 UPI PCIe UPI UPI Memory Controller Memory Controller PCIe PCIe UPI UPI DMI DMI Chipset QAT x16 QAT x16 QAT x16 I/O Hub 4x10G NIC
8. Resources in Kubernetes*: Native • Per container – spec.containers[].resources • requests – cpu – memory • limits – cpu – memory • QoS – Best Effort – Burstable – Guaranteed * Other names and brands may be claimed as the property of others.
9. Resources in Kubernetes*: Extended • Extended resources – Fully-qualified names – Outside of “kubernetes.io” domain – Node level • Device Plugin managed resources • Arbitrary advertised by node capacity * Other names and brands may be claimed as the property of others.
10. Resources in Kubernetes*: Metadata • Pod’s Metadata – spec.metadata.labels – spec.metadata.annotations apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ... * Other names and brands may be claimed as the property of others.
11. Resources in Kubernetes* • CPU Manager (1.10+) & Topology Manager (1.16+) Kubelet Topology Manager CPU Manager Device Manager Hints Can Admit Pod ? Device Plugins Hints * Other names and brands may be claimed as the property of others.
12. What we can do: CRI • CPU – CFS: Completely Fair Scheduler: • period, quota, shares • Memory – Limit, OOM Score • cpuset – cpus, mems
13. What we can do: OCI • runc – blkio: weight – CPU realtime period – Kernel memory – Memory reservation – L3 cache schema – Memory Bandwidth schema • OCI spec – blkio: IOPS / bps throttling – HugePage limits – Intel® RDT class – Hooks
14. Hooking into Kubernetes* CRI OCI Kata Kubelet Container Runtime cri-o, containerd, … ... runc + OCI Hooks Kernel * Other names and brands may be claimed as the property of others.
15. CRI Resource Manager • A new Intel project – Container Runtime Interface proxy, sits between CRI Clients and the CRI Runtime – Applies (hardware) resource policies to containers by • modifying proxied container requests, or • generating container update requests, or • triggering extra policy-specific actions during request processing • Benefits – Enable easy prototyping of resource allocation policies – Instrumentation of CRI interface – Drive resource management improvements in upstream Kubernetes • http://github.com/intel/cri-resource-manager
16. CRI Resource Manager Daemon Set CRI-Resource-Manager Kubelet Actual CRI dockershim, containerd, cri-o CRI server CRI client Relay Logs Policy Engine Policy N Policy 1 Kernel cgroups, resctrl libcontainer Dynamic Configuration and Policy API Dynamic Configuration and Policy Agent Kubernetes* API Server Resource File Resource File mounted to container Dynamic Configuration Custom Metrics Tracing Metrics Cache * Other names and brands may be claimed as the property of others.
17. CRI Resource Manager • What do we have now: – Policies: • Static • Static+ • Topology-aware • Work in progress – Intel® RDT: Cache and Memory allocation and monitoring – Block I/O classification and tuning – Better monitoring of resources usage and dynamic rebalancing – Dynamic Configuration and Policy APIs
18. CRI Resource Manager DEMO http://bit.ly/cri-r-m-demo-sp http://bit.ly/cri-r-m-demo-ta
19. Q&A
20. Thank you! alexander.kanevskiy@intel.com krisztian.litkey@intel.com
21. Legal notices and disclaimers • Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at www.intel.com. • Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. • *Other names and brands may be claimed as the property of others. • © Intel Corporation