Understanding CPU throttling in Kubernetes to improve application performance #k8sjp

https://k8sjp.github.io/kubefest-2020/sessions/12/

Transcript

1. Understanding CPU throttling in Kubernetes to improve application performance 2020/06/13 KubeFest Tokyo 2020 @ponde_m
2. I work at @ponde_m @d-kuro
3. Introduction • Kubernetes ͷ Resource Management ʹ͍ͭͯ৭ʑ࿩͠·͕͢ ࠓճ͸ CPU ʹϑΥʔΧεͨ͠࿩Λ͠·͢ • Memory ʹ͍ͭͯͷ࿩͸͠·ͤΜ
4. Agenda • Resource Management in Kubernetes • How Kubernetes Resource Control is implemented • Checking the Throttling • CPU Throttling Impacts and Fixes
5. Understanding CPU throttling in Kubernetes to improve application performance 2020/06/13 KubeFest Tokyo 2020 @ponde_m Understanding CPU throttling in Kubernetes to improve application performance
6. What is Throttling?
7. What is Throttling? • ฉ͍ͨ͜ͱ͸͋Δ͕งғؾͰ࢖͍ͬͯΔ • ͳΜ੍͔ݶ͞Εͯͦ͏͕ͩݴޠԽ͕Ͱ͖ͳ͍
8. What is Throttling? • εϩοτϧ(ӳ: throttle)͸ྲྀମΛ੍ޚ͢ΔػߏͷͻͱͭͰɺྲྀ࿏அ໘ ੵΛมԽͤͯ͞ྲྀྔΛ੍ޚ͢Δ૷ஔͰ͋Δɻओཁͳߏ੒෦඼Ͱ͋Δ หʢόϧϒʣ͸εϩοτϧόϧϒ(ӳ: throttle valve)͋Δ͍͸ߜΓห ͱݺ͹ΕɺหΛૢ࡞͢ΔͨΊͷߏ଄͸εϩοτϧϨόʔ(ӳ: throttle lever)ɺεϩοτϧϖμϧ(ӳ: throttle pedal)ɺΨεϖμϧ(ถ: gas pedal)ɺεϩοτϧάϦοϓ(ӳ: throttle grip)ͳͲͷΑ͏ʹݺ͹Ε Δɻ͋Δ͍͸ૢ࡞෦Λࢦͯ͠εϩοτϧͱུশ͢Δ৔߹΋͋Δɻ https://ja.wikipedia.org/wiki/%E3%82%B9%E3%83%AD%E3%83%83%E3%83%88%E3%83%AB
9. What is Throttling? • εϩοτϧ(ӳ: throttle)͸ྲྀମΛ੍ޚ͢ΔػߏͷͻͱͭͰɺྲྀ࿏அ໘ ੵΛมԽͤͯ͞ྲྀྔΛ੍ޚ͢Δ૷ஔͰ͋Δɻओཁͳߏ੒෦඼Ͱ͋Δ หʢόϧϒʣ͸εϩοτϧόϧϒ(ӳ: throttle valve)͋Δ͍͸ߜΓห ͱݺ͹ΕɺหΛૢ࡞͢ΔͨΊͷߏ଄͸εϩοτϧϨόʔ(ӳ: throttle lever)ɺεϩοτϧϖμϧ(ӳ: throttle pedal)ɺΨεϖμϧ(ถ: gas pedal)ɺεϩοτϧάϦοϓ(ӳ: throttle grip)ͳͲͷΑ͏ʹݺ͹Ε Δɻ͋Δ͍͸ૢ࡞෦Λࢦͯ͠εϩοτϧͱུশ͢Δ৔߹΋͋Δɻ https://ja.wikipedia.org/wiki/%E3%82%B9%E3%83%AD%E3%83%83%E3%83%88%E3%83%AB
10. What is Throttling? • ͳͷͰࠓճͷςʔϚΛݴ͍׵͑Δͱ • “CPU ͷྲྀྔ੍ޚʹ͍ͭͯཧղͯ͠ ΞϓϦέʔγϣϯͷύϑΥʔϚϯεΛ޲্ͤ͞Α͏” • ͱ͍͏ײ͡ʹͳΔ͔ͳͱࢥ͍ͬͯ·͢
11. What is Throttling? • Throttling CPU usage with Linux cgroups http://kennystechtalk.blogspot.com/2015/04/throttling-cpu-usage-with-linux-cgroups.html
12. Resource Management
13. Resource Management • Kubernetes Ͱ͸ Pod ʹׂΓ౰ͯΔ Resource Λࢦఆ͢Δ͜ͱ͕Ͱ͖Δ
14. Requests • ࠷௿ݶ Pod ʹׂΓ౰ͯΔ Resource
15. Limits • Pod ʹׂΓ౰ͯΔ Resource ͷ্ݶ • Limits ʹୡ͢Δͱ CPU ͕ throttle ͞ΕΔ
16. Millicores • 1000 millicores == 1 Core • 2 Core ෼ͷ CPU Λ࢖༻͍ͨ͠৔߹ • 2000m • 1 Core ͷ ¼ ෼ͷ CPU Λ࢖༻͍ͨ͠৔߹ • 250m
17. Scheduling
18. Scheduling • Node ΁ͷ Pod ͷ Scheduling ͸ Requests ΛݩʹߦΘΕΔ Requests 1000m Requests 1000m Requests 1000m Requests 1000m resources: requests: cpu: 1000m limits: cpu: 2000m Schedule 
    $PSF Requests 1000m Requests 1000m Requests 1000m 
    $PSF
19. Scheduling • Node ͷ Resource ΑΓ΋େ͖͍ Requests ͩͱ Pod ͕ schedule ͞Εͳ͍ resources: requests: cpu: 1000m limits: cpu: 2000m schedule resources: requests: cpu: 5000m limits: cpu: 6000m schedule Requests 1000m Requests 1000m Requests 1000m Requests 1000m 
    $PSF 
    $PSF
20. Scheduling • Limits ͸ Scheduling ͷࡍʹߟྀ͞Εͳ͍ͷͰ Pod ͕Limits ·Ͱ Resource Λ࢖͑Δอূ͸ͳ͍ Node ͷར༻Մೳͳ Resource Λ্ճΔࣄ͕͋Δ (overcommitted state) Requests 1000m Requests 1000m Requests 1000m Requests 1000m 
    $PSF Limits 2000m
21. Overcommitted State Limits 2000m Requests 1000m Requests 1000m Requests 1000m Requests 1000m • Overcommitted State ʹͳͬͨࡍʹͲͷΑ͏ʹৼΔ෣͏ͷ͔ • CPU ͷ৔߹͸ CPU Requests ͷ෼ͷ CPU Λऔಘͯ͠࢒ΓΛ throttle ͤ͞Δ throttle!! CPU ͸ѹॖՄೳͳ Resource https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/resources.md 
    $PSF
22. Tips: Capacity and Allocatable • Node ͷ status ʹ͸ Capacity ͱ Allocatable ͱ͍͏2ͭͷ߲໨͕͋Δ • Allocatable • Pod Λಈ͔ͨ͢Ίʹ࢖༻Մೳͳ Resource • Capacity • Node શମͰ࢖༻Մೳͳ Resource • kubectl describe node ͱ͔Ͱ֬ೝͰ͖Δ
23. Tips: Capacity and Allocatable • kube-reserved • kubelet ͳͲͷ Node ͷίϯϙʔωϯτ༻ʹ ༧໿͞Εͨ Resource • system-reserved • ࢒ΓͷγεςϜίϯϙʔωϯτ༻ʹ༧໿͞Εͨ Resource • eviction-thresholds • Eviction ͷᮢ஋
24. Tips: Capacity and Allocatable • kube-reserved • kubelet ͳͲͷ Node ͷίϯϙʔωϯτ༻ʹ ༧໿͞Εͨ Resource • system-reserved • ࢒ΓͷγεςϜίϯϙʔωϯτ༻ʹ༧໿͞Εͨ Resource • eviction-thresholds • Eviction ͷᮢ஋ /PEF
    $BQBDJUZ
    
    LVCFSFTFSWFE
    
    TZTUFNSFTFSWFE
    
    FWJDUJPOUISFTIPME
    
    "MMPDBUBCMF
25. Tips: Capacity and Allocatable • kube-reserved ͱ system-reserved ʹ ؔͯ͠͸σϑΥϧτ஋͕ͳ͍ • Scheduler ͸ kubelet ͳͲͷγεςϜίϯϙʔωϯτ͕ ࢖༻͢Δ Resource ͷ͜ͱΛߟྀͤͣʹ Pod Λ Schedule ͢Δ • Node ͕ෆ҆ఆʹͳΔՄೳੑ͕͋ΔͷͰઃఆ͢Δͱ ⭕ • ֤छύϒϦοΫΫϥ΢υͷ Managed Kubernetes Ͱ͸ ͍͍ײ͡ʹઃఆ͞Ε͍ͯͨΓ͢Δ͜ͱ΋͋Δ
26. How Kubernetes Resource Control is implemented
27. How Kubernetes Resource Control is implemented • Kubernetes Ͱ͸ cgroup Λ༻͍ͯ Resource ͷׂ౰Λ੍ޚ͍ͯ͠Δ • ࠨͷ Pod Λ࢖ͬͯৄࡉΛݟ͍ͯ͘
28. CPU Requests
29. CPU Requests • Pod ͕ Schedule ͞Εͨ Node ্ͰҎԼͷσΟϨΫτϦʹҠಈ͢Δ
30. CPU Requests • Pod ͕ Schedule ͞Εͨ Node ্ͰҎԼͷσΟϨΫτϦʹҠಈ͢Δ
31. CPU Requests • Pod ͕ Schedule ͞Εͨ Node ্ͰҎԼͷσΟϨΫτϦʹҠಈ͢Δ
32. CPU Requests • Pod ͕ Schedule ͞Εͨ Node ্ͰҎԼͷσΟϨΫτϦʹҠಈ͢Δ
33. CPU Requests
34. CPU Requests • CPU Requests ͸ cpu.shares ͱ͍͏ϑΝΠϧʹ൓ө͞ΕΔ • 250 ͱ 256 Ͱ਺ࣈ͕ζϨ͍ͯΔͷ͸ cgroup ͕ CPU 1 Core Λ 1024 ʹ෼ׂ͢Δ͕ Kubernetes Ͱ͸ 1000 ʹ෼ׂ͍ͯ͠ΔͨΊ
35. cpu.shares cpu.shares 1024 cpu.shares 1024 cpu.shares 1024 cpu.shares 1024 cpu.shares 1536 512 cpu.shares 1024 cpu.shares 1024 25% 25% 25% 25% 25% 25% 37.5% 12.5% cpu.shares ͕ ίϯςφؒͰશͯಉ͡৔߹ ۉ౳ʹ CPU Resource͕഑෼͞ΕΔ cpu.shares ͕ ίϯςφؒͰҟͳΔ৔߹͸ ൺ཰ʹԠͯ͡ CPU Resource ͕ ഑෼͞ΕΔ • CPU ഑෼ͷॏΈ • cpu.share ͷ߹ܭʹରͯ͠഑෼͕ܭࢉ͞ΕΔͨΊ૬ରతͳ஋
36. cpu.shares 1536 512 cpu.shares 1024 cpu.shares 1024 cpu.shares cpu.shares 1536 512 cpu.shares 1024 cpu.shares 1024 25% 25% 1536/4096 = 37.5% 12.5% cpu.shares 2048 cpu.shares 2048 ͷ ίϯςφΛ௥Ճ͢Δ 1536/6144 = 25% ໿8.3% ໿16.7% ໿33.3% ໿16.7%
37. cpu.shares • cpu.shares ͸શͯͷίϯςφ͕ϑϧͰ CPU Resource Λ࢖༻͠Α͏ͱͨ͠ࡍͷ഑෼ • cpu.shares ʹج͍ܾͮͯΊΒΕͨCPU Resource ͷ഑෼͸อূ͞ΕΔ • Pod ͷ CPU Requests ʹهࡌͨ͠෼ͷ CPU Resource ͸อূ͞ΕΔ • ΞΠυϧͷίϯςφ͕ଘࡏ͢Δ৔߹͸ۭ͍ͨ෼ͷ CPU Λ ଞͷίϯςφ͕࢖༻͢Δ͜ͱ͕Ͱ͖Δ • Pod ͸ CPU Limits ʹୡ͍ͯ͠ͳ͚Ε͹༨͍ͬͯΔ CPU Resource Λ ͍͍ײ͡ʹ࢖͏͜ͱ͕Ͱ͖Δ
38. CPU Limits
39. CPU Limits • CPU Limits ͸ CFS (Completely Fair Scheduler) ͷ ϝΧχζϜΛ༻͍࣮ͯݱ͞Ε͍ͯΔ • cpu.cfs_period_us ͱ cpu.cfs_quota_us ʹઃఆ͕هࡌ͞Ε͍ͯΔ
40. cpu.cfs_period_us / cpu.cfs_quota_us • Requests ʹ ༻͍ΒΕ͍ͯΔ cpu.shares ͱ͸ҟͳΓ Limits ʹ༻͍Β Ε͍ͯΔ CPU ͷׂ౰͸ظؒʹج͍͍ͮͯΔ • cpu.cfs_period_us • ظؒͷఆٛ, σϑΥϧτ͸ 100000us(100ms) • cpu.cfs_quota_us • ظؒ಺Ͱ࣮ߦͰ͖Δ߹ܭ࣌ؒ
41. cpu.cfs_period_us / cpu.cfs_quota_us • CPU Limits ͸ cpu.cfs_quota_us ʹม׵͞ΕΔ • Լͷྫͩͱ 100ms ͷظؒத 50ms, CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖Δ
42. cpu.cfs_period_us / cpu.cfs_quota_us • CPU Limits: 2000m ͷΑ͏ʹࢦఆͨ͠৔߹͸ cpu.cfs_period_us: 100000 cpu.cfs_quota_us: 200000 ͱͳΔ
43. cpu.cfs_period_us / cpu.cfs_quota_us • CPU Limits ͕ͳ͍৔߹ ͸ cpu.cfs_quota_us ʹ͸ -1 ͕ઃఆ͞ΕΔ • -1 ͕ઃఆ͞ΕΔͱແ੍ݶʹͳΔ
44. Time (ms) cpu.cfs_period_us / cpu.cfs_quota_us • ॲཧΛ׬ྃ͢ΔͨΊʹ 250ms ͷॲཧ͕࣌ؒඞཁͳ γϯάϧεϨουͷΞϓϦέʔγϣϯ 250ms 100ms 200ms 300ms 400ms
45. Time (ms) cpu.cfs_period_us / cpu.cfs_quota_us • cpu.cfs_period_us: 100000 cpu.cfs_quota_us: 50000 ͷ৔߹ 50ms 100ms 200ms 300ms 400ms 50ms 50ms 50ms 50ms throttled throttled throttled throttled
46. Time (ms) cpu.cfs_period_us / cpu.cfs_quota_us • cpu.cfs_period_us: 100000 cpu.cfs_quota_us: 50000 ͷ৔߹ 50ms 100ms 200ms 300ms 400ms 50ms 50ms 50ms 50ms throttled throttled throttled throttled 50ms 100ms 200ms 300ms 400ms 50ms 50ms 50ms 50ms throttled throttled throttled throttled Time (ms) Throttling ͞ΕΔͷͰ ׬ྃ·Ͱʹ 450 ms ͔͔ͬͯ͠·͏
47. Kernel Bugs
48. Kernel Bugs • Kernel ʹ CPU ͷ࢖༻཰͕௿ͯ͘΋ίϯςφ͕ Throttling ͞ΕΔόά͕ଘࡏͨ͠ • Indeed ͷਓ͕ॻ͍ͨϒϩάʹৄࡉ͔Β patch Λग़ͨ͠࿩·Ͱৄ͘͠ॻ͍ͯ͋Δ • εϩοτϦϯάղআ: Ϋϥ΢υʹ͓͚Δ CPU ͷ੍ݶͷमਖ਼ - Indeed ΤϯδχΞϦϯάɾϒϩά https://jp.engineering.indeedblog.com/blog/2019/12/%e3%82%b9%e3%83%ad%e3%83%83%e3%83%88%e3%83%aa%e3%83%b3%e3%82%b0%e3%81%ae%e8%a7%a3%e9%99%a4- %e6%9c%89%e5%8a%b9%e3%81%aa%e4%bf%ae%e6%ad%a3%e3%81%8c%e4%b8%8d%e5%85%b7%e5%90%88%e3%81%ae%e5%8e%9f%e5%9b%a0/
49. Checking the Throttling
50. Checking the Throttling • cpu.stat Λ֬ೝ͢Δ • nr_periods: ܦաࡁΈͷظؒͷ਺ • nr_throttled: nr_periods த Throttling ͷର৅ͱͳͬͨظؒͷ਺ • throttled_time: ૯ Throttle ࣌ؒ (ns)
51. Checking the Throttling • Datadog Ͱ͸ kubernetes.cpu.cfs.throttled.seconds ͱ͍͏ϝτϦΫε͕͋Δ
52. The Impact of CPU Throttling
53. The Impact of CPU Throttling • Error Rate ΍ Response Time ͷѱԽΛট͘͜ͱ͕͋Δ • github.com/hjacobs/kubernetes-failure-stories • Kubernetes ʹؔ͢Δো֐ࣄྫΛ·ͱΊͨ Repository • CPU Throttling ͷࣄྫ͕͍͔ͭ͘ొ৔͢Δ
54. CPU Throttling the Application Pod in Quipper CPU Limits CPU Requests CPU throttled CPU Usage
55. Why was the CPU Throttling gone? ͋ΔมߋΛՃ͑ͯҎ߱ CPU Throttling ͕ͳ͘ͳͬͨ
56. Fix
57. Fix • CPU Limits Λফͨ͠
58. Fix CPU Limits ͜͜Ͱ Limit Λফͨ͠ ͦΕҎ߱ CPU Throttling ͕ͳ͘ͳͬͨ
59. Remove CPU Limits • CPU Limits ͸ઃఆ͢Δͱ CPU ʹۭ͖ Resource ͕͋ͬͯ΋ ͦΕҎ্ར༻͢Δ͜ͱ͕Ͱ͖ͳ͘ͳΔ • ۭ͍ͯΔ CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖ͳ͍ͷ͸ جຊతʹΞϓϦέʔγϣϯʹͱͬͯ͸σϝϦοτ • ա৒ͳར༻Λ཈੍͍ͨ͠৔߹Ҏ֎͸ CPU Limits ͸͍Βͳ͍ • CPU Requests Ͱࢦఆͨ͠෼͸࢖༻Ͱ͖Δ͜ͱ͕อূ͞ΕΔ
60. Remove CPU Limits • ฐࣾͰ͸͜͏͍͏ϙϦγʔͰ Pod ͷ Resource Λઃఆ͠Α͏ ͱ͍͏υΩϡϝϯτΛهࡌͯ͠ӡ༻͍ͯ͠Δ • CPU Limits ͸جຊෆཁͩΑͱ͍͏هࡌΛ͍ͯ͠Δ • CPU Limits Λ࡟আ͢Δ͜ͱʹڧ੍ྗΛಇ͔ͤΔ͜ͱ͸·͍ͩͯ͠ͳ͍ • શͯͷ Pod ͔Β CPU Limits Λ࡟আͨ͠Θ͚Ͱ͸ແ͍͕ CPU Throttling ͷϝτϦΫεΛ watch ͍ͯͯ͠ Throttling ͕ൃੜ͢Ε͹౎౓ରԠΛ͍ͯ͠Δ
61. Remove CPU Limits • CPU Limits ͸ઃఆ͢Δͱ CPU ʹۭ͖ Resource ͕͋ͬͯ΋ ͦΕҎ্ར༻͢Δ͜ͱ͕Ͱ͖ͳ͘ͳΔ • ۭ͍ͯΔ CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖ͳ͍ͷ͸ جຊతʹΞϓϦέʔγϣϯʹͱͬͯ͸σϝϦοτ • ա৒ͳར༻Λ཈੍͍ͨ͠৔߹Ҏ֎͸ CPU Limits ͸͍Βͳ͍ • CPU Requests Ͱࢦఆͨ͠෼͸࢖༻Ͱ͖Δ͜ͱ͕อূ͞ΕΔ
62. Remove CPU Limits • CPU Limits ͸ઃఆ͢Δͱ CPU ʹۭ͖ Resource ͕͋ͬͯ΋ ͦΕҎ্ར༻͢Δ͜ͱ͕Ͱ͖ͳ͘ͳΔ • ۭ͍ͯΔ CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖ͳ͍ͷ͸ جຊతʹΞϓϦέʔγϣϯʹͱͬͯ͸σϝϦοτ • ա৒ͳར༻Λ཈੍͍ͨ͠৔߹Ҏ֎͸ CPU Limits ͸͍Βͳ͍ • CPU Requests Ͱࢦఆͨ͠෼͸࢖༻Ͱ͖Δ͜ͱ͕อূ͞ΕΔ CPU Limits ͕ͳ͍ͱ QoS ͕ Guaranteed ʹͳΒͳ͍
63. QoS Class • Eviction ͷ࣌ʹߟྀ͞ΕΔ Pod ͷ༏ઌ౓ • Guaranteed • Requests/Limits ͕Ұக • Burstable • Requests/Limits ͕ෆҰக • BestEffort • Requests/Limits ͕ͳʹ΋ͳ͍
64. QoS Class • ฐࣾͰ͸ݱঢ় CPU Limits Λ࡟আͨ͜͠ͱͰ QoS Class ͕ Burstable ʹ ઃఆ͞ΕΔ͜ͱͰͳʹ͔ࠔ͍ͬͯΔ͜ͱ͸ͳ͍ • ݩʑ Guaranteed ͳ Pod ͕ຆͲͳ͔ͬͨͱ͍͏ͷ͸͋Δ • Eviction ͸ͦ΋ͦ΋͕ͦΜͳʹൃੜ͍ͯ͠ͳ͍ • ࠔΔέʔε͸ଘࡏ͢Δ
65. Static CPU Manager Policy • Pod ʹ CPU ΛഉଞతʹׂΓ౰ͯΔ͜ͱ͕Ͱ͖Δ • kubelet ͷ Option ͷ cpu-manager-policy=static Ͱ ༗ޮʹ͢Δ͜ͱ͕Ͱ͖Δ • ഉଞతʹ CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖Δ৚݅ • Pod ͷ QoS Class ͕ Guaranteed • Pod ͷ CPU Requests ͕੔਺஋
66. Static CPU Manager Policy • Pod ʹ CPU ΛഉଞతʹׂΓ౰ͯΔ͜ͱ͕Ͱ͖Δ • kubelet ͷ Option ͷ cpu-manager-policy=static Ͱ ༗ޮʹ͢Δ͜ͱ͕Ͱ͖Δ • ഉଞతʹ CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖Δ৚݅ • Pod ͷ QoS Class ͕ Guaranteed • Pod ͷ CPU Requests ͕੔਺஋
67. cpu-cfs-quota=false • kubelet ͷ Option Ͱ cpu-cfs-quota=false Λ౉͢ • CFS Quota ͕ແޮʹͳΔͷͰ CPU Limits ʹΑΔ Throttle ͕ى͖ͳ͘ͳΔ • CPU Limits ࣗମ͸ࢦఆͰ͖ΔͷͰ QoS Class ͸ Guaranteed ʹ ͢Δ͜ͱ͕Ͱ͖Δ
68. cpu-cfs-quota=false • --cpu-manager-policy=static —cpu-cfs-quota=false Λ౉ͯ͠ kubelet Λىಈ͢Δ (ผͰ —kube-reserved ͱ —system-reserved Λࢦఆ͍ͯ͠Δ) • QoS ͕ Guaranteed ͔ͭ CPU Requests ͕੔਺஋ʹ ͳΔΑ͏ͳ Pod Λ༻ҙ • 4 Core ͷ Node த 1 Core Λ Nginx Pod ʹઐ༗ͤ͞Δ
69. cpu-cfs-quota=false • Pod ͷ QoS Class ͕ Guaranteed ͔ͭ cpu.cfs_quota_us ͕ -1 (ແ੍ݶ) ʹͳ͍ͬͯΔ
70. • cpuset.cpus ϑΝΠϧΛ֬ೝ͢Δͱ Nginx ͷ Pod ͕ CPU 1 ͭΛ࢖͏Α͏ʹͳ͍ͬͯΔ͜ͱ͕֬ೝͰ͖Δ Static CPU Manager Policy
71. • cpuset.cpus ϑΝΠϧΛ֬ೝ͢Δͱ Nginx ͷ Pod ͕ CPU 1 ͭΛ࢖͏Α͏ʹͳ͍ͬͯΔ͜ͱ͕֬ೝͰ͖Δ Static CPU Manager Policy cpuset σΟϨΫτϦ cpuset.cpus ϑΝΠϧʹ ίϯςφ͕ΞΫηεͰ͖Δ CPU ͷ൪߸͕ॻ͔Ε͍ͯΔ
72. Static CPU Manager Policy CPU 0 CPU 1 CPU 2 CPU 3 
    $PSF cpuset.cpus = 2 Pod ͸ 2 ൪ͷ CPU ͚ͩ࢖͑Δ
73. • ద౰ͳଞͷ Burstable ͳ Pod ͷ cpuset.cpus ϑΝΠϧΛ֬ೝ͢Δͱ 2 ൪ͷ CPU ͚͕ͩআ֎͞Ε͍ͯΔ Static CPU Manager Policy 0 ~ 1 ൪ͱ 3 ൪ͷ CPU Λ࢖༻͢Δ͜ͱ͕Ͱ͖Δ (Nginx Pod ͕࢖༻͍ͯ͠Δ 2 ൪͸আ֎͞Ε͍ͯΔ)
74. Static CPU Manager Policy CPU 0 CPU 1 CPU 2 CPU 3 
    $PSF cpuset.cpus = 0-1, 3 Pod ͸ 0, 1, 3 ൪ͷ CPU ͚ͩ࢖͑Δ
75. cpu-cfs-quota=false • ஫ҙ఺ͱͯ͠͸ kubelet ͷ Option Ͱࢦఆ͢ΔͷͰ Node ୯ҐͰͷ༗ޮԽ/ແޮԽʹͳΔ • ༗ޮԽ͢Δ Node ͰͲͷ Pod ͕ಈ͍͍ͯΔ͔ͱ͍͏֬ೝΛ͢Δͱ⭕
76. Other Workarounds • Pod ͷ CPU Limits Λ૿΍͢ • ΞϓϦέʔγϣϯͷ Thread ΛݮΒ͢
77. Bood Bye Throttling
78. Conclusion
79. Conclusion • CPU Throttling ʹΑͬͯΞϓϦέʔγϣϯͷύϑΥʔϚϯε΍ ৴པੑʹѱӨڹ͕ͰΔ͜ͱ͕͋Δ • CPU Throttling ͷϞϦλϦϯάΛ͠Α͏ • CPU Limits ͷ࡟আ΍ CFS Quota Λແޮʹ͢Δ͜ͱͰ CPU Throttling ΁ͷରࡦΛߦ͏͜ͱ͕Ͱ͖Δ
80. Thank You for Listening
81. References • Understanding resource limits in kubernetes: cpu time • https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu- time-9eff74d3161b • CPU limits and aggressive throttling in Kubernetes - Omio Engineering - Medium • https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes- c5b20bd8a718 • Understanding Linux Container Scheduling — Squarespace / Engineering • https://engineering.squarespace.com/blog/2017/understanding-linux-container-scheduling
82. References • DockerίϯςφͰར༻Ͱ͖ΔϦιʔε΍ݖݶΛ੍ݶ͢ΔʢDockerͷ࠷৽ػೳΛ࢖ͬͯΈΑ͏ɿୈ3 ճʣ | ͘͞ΒͷφϨοδ • https://knowledge.sakura.ad.jp/5118/ • OpenShiftͷResource requestͱlimit - nekop's blog • https://nekop.hatenablog.com/entry/2017/12/20/182523 • How to Evolve Kubernetes Resource Management Model • https://www.infoq.com/presentations/evolve-kubernetes-resource-manager/
83. References • 3.2. cpu Red Hat Enterprise Linux 6 | Red Hat Customer Portal • https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/6/html/ resource_management_guide/sec-cpu • 3.4. cpuset Red Hat Enterprise Linux 6 | Red Hat Customer Portal • https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/6/html/ resource_management_guide/sec-cpuset • ୈ4ষ CPU Ϛωʔδϟʔͷ࢖༻ OpenShift Container Platform 4.1 | Red Hat Customer Portal • https://access.redhat.com/documentation/ja-jp/openshift_container_platform/4.1/html/ scalability_and_performance/using-cpu-manager
84. References • εϩοτϦϯάղআ: Ϋϥ΢υʹ͓͚Δ CPU ͷ੍ݶͷमਖ਼ - Indeed ΤϯδχΞϦϯάɾϒϩά • https://jp.engineering.indeedblog.com/blog/ 2019/12/%e3%82%b9%e3%83%ad%e3%83%83%e3%83%88%e3%83%aa%e3%83%b3%e3%82%b0 %e8%a7%a3%e9%99%a4- %e3%82%af%e3%83%a9%e3%82%a6%e3%83%89%e3%81%ab%e3%81%8a%e3%81%91%e3%82% 8b-cpu-%e3%81%ae%e5%88%b6%e9%99%90%e3%81%ae/ • εϩοτϦϯάͷղআ: ༗ޮͳमਖ਼͕ෆ۩߹ͷݪҼʹͳͬͯ͠·ͬͨཧ༝ - Indeed ΤϯδχΞϦϯάɾϒϩά • https://jp.engineering.indeedblog.com/blog/ 2019/12/%e3%82%b9%e3%83%ad%e3%83%83%e3%83%88%e3%83%aa%e3%83%b3%e3%82%b0 %e3%81%ae%e8%a7%a3%e9%99%a4- %e6%9c%89%e5%8a%b9%e3%81%aa%e4%bf%ae%e6%ad%a3%e3%81%8c%e4%b8%8d%e5%85% b7%e5%90%88%e3%81%ae%e5%8e%9f%e5%9b%a0/
85. References • Kubernetes best practices: Resource requests and limits | Google Cloud Blog • https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests- and-limits • community/resource-qos.md at master · kubernetes/community • https://github.com/kubernetes/community/blob/master/contributors/design-proposals/ node/resource-qos.md#qos-classes • community/resources.md at master · kubernetes/community • https://github.com/kubernetes/community/blob/master/contributors/design-proposals/ scheduling/resources.md
86. References • Control CPU Management Policies on the Node - Kubernetes • https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ • Feature Highlight: CPU Manager - Kubernetes • https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/ • kubelet - Kubernetes • https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ • Reserve Compute Resources for System Daemons - Kubernetes • https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
87. References • CFS quotas can lead to unnecessary throttling · Issue #67577 · kubernetes/kubernetes • https://github.com/kubernetes/kubernetes/issues/67577 • Throttling: New Developments in Application Performance with CPU Limits - Dave Chiluk, Indeed - YouTube • https://www.youtube.com/watch?v=UE7QX98-kO0 • Throttling CPU usage with Linux cgroups • http://kennystechtalk.blogspot.com/2015/04/throttling-cpu-usage-with-linux-cgroups.html • hjacobs/kubernetes-failure-stories: Compilation of public failure/horror stories related to Kubernetes • https://github.com/hjacobs/kubernetes-failure-stories