Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Resource Requests and Limits Under the Hood: The Journey of a Pod Spec

Resource Requests and Limits Under the Hood: The Journey of a Pod Spec

A talk at KubeCon + CloudNativeCon Europe 2021 Virtual

Ad22fcf5773b906c08330f4d57242212?s=128

Kohei Ota

May 06, 2021
Tweet

Transcript

  1. Kaslin Fields, Google Kohei Ota, Hewlett Packard Enterprise Resource Requests

    and Limits Under the Hood: The Journey of a Pod Spec
  2. Self introduction Kohei Ota Architect at Hewlett-Packard Enterprise CNCF Ambassador

    Owner of SIG-Docs Japanese localization Twitter: @inductor__ GitHub: @inductor Kaslin Fields Developer Advocate at Google CNCF Ambassador Member of K8s SIG-ContribEx Comics at kaslin.rocks! Twitter: @kaslinfields GitHub: @kaslin
  3. Your App

  4. Doggy Daycare Analogy

  5. It Takes a Village

  6. Resource Requests & Limits

  7. Here’s an ordinary pod spec apiVersion: v1 kind: Pod metadata:

    name: kubecon-eu-2021 spec: containers: - name: kubecon-eu-2021 image: kubecon:eu-2021 resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" Requests
  8. Requests for planning y x z

  9. I want to put my app in a Pod! The

    Journey of a Request Kubernetes API Developer App Ok, we can help with that.
  10. My app will need 2CPU and 4GB of Memory to

    run properly. Update Cluster State I’ll make note of that so your app’s needs will be met. Record to etcd* *etcd is the key-value store component used by Kubernetes. It functions as a single source of truth for the state of the cluster.
  11. Oh, a new pod is coming that will require 2CPU

    & 4GB of memory. I’ll put that… here. Kubernetes Scheduler Assign Pod to Node
  12. Scheduler Scheduler is a Kubernetes component that evaluates nodes to

    assign a Pod. Resource request is one of the parameters that Scheduler uses when ranking nodes.
  13. Ok, let’s get this new pod settled in! Kubelet on

    Node Create Pod on Node
  14. The Journey of a Request New Pod to assign Assign

    a Node to a Pod Detects that a Pod was assigned to the Node API Server (Control Plane) Hmm… Which node should I put this on… Here ya go, little buddy! Oh, a new request! Scheduler Kubelet (on node)
  15. Requests passing through API Server (Control plane) Scheduler (Control plane)

    Kubelet (Each node) New Pod to assign Assign a Node to a Pod Detects a Pod that assigned to the Node Node evaluation with Resource Requests
  16. Requests passing through Detects that a Pod was assigned to

    the Node Hmm… Which node should I put this on… Pod Requests vs Node Allocatable Scheduler
  17. Requests passing through API Server (Control plane) Scheduler (Control plane)

    Kubelet (Each node) New Pod to assign Assign a Node to a Pod Detects a Pod that assigned to the Node Node evaluation with Resource Requests Pod Requests vs Node Allocatable
  18. Requests summary - Requests - Used at Pod creation -

    Scheduler selects a Node for a Pod to match the resource requirement - CPU request is used in order to limit CPU resource in case they’re used 100% - When CPU is not fully used it’s over-committable - CPU - If over request? → Potential of eviction - Memory - If over request? → Potential of eviction QoS Class
  19. QoS Class in Kubernetes QoS Class Condition Priority (Lower is

    better) Guaranteed limits and optionally requests (not equal to 0) are set for all resources across all containers and they are equal 1 Burstable requests and optionally limits are set (not equal to 0) for one or more resources across one or more containers, and they are not equal 2 BestEffort requests and limits are not set for all of the resources, across all containers 3
  20. apiVersion: v1 kind: Pod metadata: name: kubecon-eu-2021 spec: containers: -

    name: kubecon-eu-2021 image: kubecon:eu-2021 resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" Now let’s talk about limits Limits
  21. Limits enforce.

  22. The Journey of a Pod Limit Kubernetes API Developer App

    I want to make sure my pod doesn’t consume more than 2CPU and 4GB of memory Yes, we can limit your pod’s resource usage.
  23. This new pod needs to be limited to 2CPU and

    4GB. I better make sure the caretaker knows. Kubernetes Scheduler Assign Pod to Node
  24. Ah, this pod is limited. I’ll get only the resources

    it needs from the supplier. Kubelet on Node Send Pod to Kubelet on Node
  25. Hey, I have a pod coming in that needs its

    resources limited. Container Runtime/Linux Kubelet to Container Runtime Kubelet Ok, I can use cgroups to make that happen.
  26. Requests for planning. Limits for enforcing.

  27. Limits summary - Limits - Used to limit resources on

    a Pod by calling cgroups on Linux - CPU - If over limit? → CPU throttling - Memory - If over limit? → Cause OOM
  28. Limits by level API Server (Control plane) Kubelet (Each node)

    CRI Runtime (Each node) Detects a Pod that needs to be assigned to a Node Convert CPU cores to CFS period/quota (milliseconds) Set to OCI spec Pass limits OCI Runtime (Each node) Call Cgroups Cgroups
  29. Limits by level API Server (Control plane) Kubelet (Each node)

    CRI Runtime (Each node) Detects a Pod that needs to be assigned to a Node Convert CPU cores to CFS period/quota (milliseconds) Set to OCI spec Pass limits OCI Runtime (Each node) Call Cgroups Cgroups
  30. Container Primitives

  31. Cgroups

  32. Cgroups? Cgroups(Control groups) allow you to allocate resources — such

    as CPU time(again, not cores!), system memory, network bandwidth, or combinations of these CPU Requests in K8s → cpu.shares in cgroups CPU Limits in K8s → cpu.cfs_period_us & cpu.cfs_quota_us (us = μs) in cgroups Memory Limits in K8s → memory.limit_in_bytes in cgroups
  33. Cgroups? Cgroups(Control groups) allow you to allocate resources — such

    as CPU time(again, not cores!), system memory, network bandwidth, or combinations of these CPU Requests in K8s -> cpu.shares in cgroups CPU Limits in K8s -> cpu.cfs_period_us & cpu.cfs_quota_us (us = μs) in cgroups Memory Limits in K8s -> memory.limit_in_bytes in cgroups Add cpu.shares 2048 cpu.shares is a relative value https://speakerdeck.com/daikurosawa/understanding-cpu-throttling-in-kubernetes-to-improve-application-performance-number-k8sjp
  34. CFS Quota? Period? CFS = “Completely Fair” Scheduler A process

    scheduler in Linux Container isolation is based on cgroups(a Linux kernel functionality) resource limitation Cgroups uses CFS to implement CPU resource restriction CFS scheduling is based on processing time but not core. Scheduling period is every 100ms https://kccncna20.sched.com/event/ek9r/10-more-weird-ways-to-blow-up-your-kubernetes-jian-cheung-joseph-kim-airbnb
  35. CFS Quota? Period? CFS = “Completely Fair” Scheduler A process

    scheduler in Linux Container isolation is based on cgroups(a Linux kernel functionality) resource limitation Cgroups uses CFS to implement CPU resource restriction CFS scheduling is based on processing time but not core. Scheduling period is every 100ms https://kccncna20.sched.com/event/ek9r/10-more-weird-ways-to-blow-up-your-kubernetes-jian-cheung-joseph-kim-airbnb K8s Limits: 500m(0.5core) CFS_Period: 100ms CFS_Quota: 50ms K8s Limits: 2000m(2core) CFS_Period: 100ms CFS_Quota: 200ms How much of CPU resource you can use in every period
  36. CFS Quota? Period? CFS = “Completely Fair” Scheduler A process

    scheduler in Linux Container isolation is based on cgroups(a Linux kernel functionality) resource limitation Cgroups uses CFS to implement CPU resource restriction CFS scheduling is based on processing time but not core. Scheduling period is every 100ms https://kccncna20.sched.com/event/ek9r/10-more-weird-ways-to-blow-up-your-kubernetes-jian-cheung-joseph-kim-airbnb K8s Limits: 500m(0.5core) CFS_Period: 100ms CFS_Quota: 50ms K8s Limits: 2000m(2core) CFS_Period: 100ms CFS_Quota: 200ms How much of CPU resource you can use in every period If there’s no limits… CFS_Quota: -1 (unlimit)
  37. CRI Runtime vs OCI Runtime How container runtime works on

    Kubernetes Kubernetes kubectl run kubectl apply REST API containerd CRI (gRPC) Collection of Kubernetes system components. kube-api-server fetches the kubectl and kubelet talks to the CRI runtime runC CRI runtime executes a OCI runtime binary file with OCI container json spec. OCI runtime spawns the container with CPU/Mem in the spec OCI High level runtime Low level runtime
  38. CRI Runtime vs OCI Runtime How container runtime works on

    Kubernetes Kubernetes kubectl run kubectl apply REST API containerd CRI (gRPC) Collection of Kubernetes system components. kube-api-server fetches the kubectl and kubelet talks to the CRI runtime runC CRI runtime executes a OCI runtime binary file with OCI container json spec. OCI runtime spawns the container with CPU/Mem in the spec OCI High level runtime Low level runtime CRI (High level) Runtimes run with Kubernetes OCI (Low level) Runtimes run with Linux kernel
  39. How do I set the right requests and limits?

  40. Pod Autoscaling Horizontal Pod Autoscaler (HPA) More Fewer Vertical Pod

    Autoscaler (VPA) Change Size ? ? ?
  41. Vertical Pod Autoscaler VPA Modes: Off Initial Auto VPA Recommendations:

    Target Lower Bound Upper Bound Uncapped Target Vertical Pod Autoscaler (VPA)
  42. Conclusion Pod spec is registered in etcd through kube-apiserver kube-scheduler

    fetches newly registered pods from etcd and assign a node to each pod referring to resource requests kubelet fetches assigned pod spec in every sync period and calculate diffs between running containers and pod spec kubelet calls CreateContainer gRPC towards CRI runtime, after converting CPU cores into periods CRI runtime executes OCI runtime binary to create a container with OCI Spec JSON OCI runtime manages cgroups file system (create/delete/update) Vertical Pod Autoscaler (VPA) can provide recommendations for your requests and limits.
  43. None