図で理解する Descheduler #k8sjp #ymju / Introduction to Descheduler

Kubernetes Meetup Tokyo 19 (Yahoo! JAPAN MEETUP共催) で発表した資料です
https://k8sjp.connpass.com/event/126207/

Transcript

1. ਤͰཧղ͢Δ
   Descheduler
   19/05/31 Kubernetes Meetup Tokyo #19 / Yahoo! JAPAN MEETUP #32

   @ponde_m
   

   View Slide

2. View Slide

3. Descheduler ?
   ?
   

   View Slide

4. Scheduler
   ͷ͓͞Β͍
   

   View Slide

5. Kubernetes ͷΞʔΩςΫνϟ
   API Server
   Contoroller

   Manager
   Scheduler
   etcd
   

   View Slide

6. Kubernetes ͷΞʔΩςΫνϟ
   API Server
   Contoroller

   Manager
   Scheduler
   etcd
   Scheduler
   Pod Λ Node ΁ׂΓ౰ͯΔ
   Scheduler
   

   View Slide

7. Scheduler
   • Pod ͷ࡞੒࣌ʹεέδϡʔϦϯάΛߦ͏
   • Kubernetes Ϋϥελ͸ಈతʹมԽ͢Δ
   • Node ͷϦιʔεʹภΓ͕ग़Δ
   • taints, label ͕มߋ͞Εͯ΋

   εέδϡʔϦϯά͞Εͨ Pod ͸ͦͷ··
   • Node ͕૿͑ͯ΋εέδϡʔϦϯά͞Εͨ Pod ͸ͦͷ··
   • Node Ͱো֐͕ൃੜ͠, 

   Pod ͕ଞͷ Node ʹҠಈͨ͋͠ͱͦͷ··
   

   View Slide

8. Scheduler
   • Pod ͷ࡞੒࣌ʹεέδϡʔϦϯάΛߦ͏
   • Kubernetes Ϋϥελ͸ಈతʹมԽ͢Δ
   • Node ͷϦιʔεʹภΓ͕ग़Δ
   • taints, label ͕มߋ͞Εͯ΋

   εέδϡʔϦϯά͞Εͨ Pod ͸ͦͷ··
   • Node ͕૿͑ͯ΋εέδϡʔϦϯά͞Εͨ Pod ͸ͦͷ··
   • Node Ͱো֐͕ൃੜ͠, 

   Pod ͕ଞͷ Node ʹҠಈͨ͋͠ͱͦͷ··
   • Kubernetes Ϋϥελ͸ಈతʹมԽ͢Δ
   • Node ͷϦιʔεʹภΓ͕ग़Δ
   • taints, label ͕มߋ͞Εͯ΋

   εέδϡʔϦϯά͞Εͨ Pod ͸ͦͷ··
   • Node ͕૿͑ͯ΋εέδϡʔϦϯά͞Εͨ Pod ͸ͦͷ··
   • Node Ͱো֐͕ൃੜ͠, 

   Pod ͕ଞͷ Node ʹҠಈͨ͋͠ͱͦͷ··
   Pod ͷ࡞੒࣌ʹ͔͠

   εέδϡʔϦϯά

   ͯ͘͠Εͳ͍ͷͰ

   ͜ͷΑ͏ͳ໰୊͕ग़ͯ͘Δ
   

   View Slide

9. Node Node
   Node * 2
   

   View Slide

10. Pod
    Pod
    Pod Pod
    Pod Pod
    Pod
    Pod
    Pod Pod
    Pod Pod
    ֤ϊʔυʹಉ͡
    Pod ͕ 6 ͭ
    

    View Slide

11. Node
    Pod
    Pod
    Pod Pod
    Pod Pod
    Pod
    Pod
    Pod Pod
    Pod Pod
    NEW!!
    

    View Slide

12. Node
    Pod
    Pod
    Pod Pod
    Pod Pod
    Pod
    Pod
    Pod Pod
    Pod Pod
    Node
    طʹ Node ʹ

    εέδϡʔϦϯά͞Ε͍ͯΔ
    Pod ͸Ҡಈͯ͘͠Εͳ͍
    

    View Slide

13. Pod
    Pod
    Pod Pod
    Pod
    Pod
    Pod Pod
    Pod
    Pod
    Pod Pod
    ͜͏ͳͬͯ΄͍͠
    

    View Slide

14. Descheduler
    

    View Slide

15. Descheduler
    • github.com/kubernetes-incubator/descheduler
    • ҠಈՄೳͳ Pod Λݟ͚ͭͯ࡟আͯ͘͠ΕΔ
    • ࠶εέδϡʔϧ͸ Scheduler ͕ߦ͏
    • Kubernetes ͷ Job ͱ࣮ͯ͠ߦͤ͞Δ
    

    View Slide

16. Descheduler
    • github.com/kubernetes-incubator/descheduler
    • ҠಈՄೳͳ Pod Λݟ͚ͭͯ࡟আͯ͘͠ΕΔ
    • ࠶εέδϡʔϧ͸ Scheduler ͕ߦ͏
    • Kubernetes ͷ Job ͱ࣮ͯ͠ߦͤ͞Δ
    ҠಈՄೳͳ Pod
    ର৅ͷ Pod Λ
    Ͳ͏΍ܾͬͯΊΔͷ͔
    

    View Slide

17. Policy and Strategies
    • Descheduler Ͱ͸, ࡟আ͢Δ Pod ͷϙϦγʔΛ

    ϑΝΠϧͰఆٛ͢Δ
    • 4 ͭͷετϥςδΛ૊Έ߹Θͤͯઃఆ͢Δ
    • RemoveDuplicates
    • LowNodeUtilization
    • RemovePodsViolatingInterPodAntiAffinity
    • RemovePodsViolatingNodeAffinity
    

    View Slide

18. RemoveDuplicates
    Pod Pod Pod
    • Replica Set / Replication Controller / Deployment / Job

    ʹඥ͍͍ͮͯΔ Pod ͕֤ Node Ͱ 1 ͚ͭͩͰ͋Δ͜ͱΛ֬ೝ͢Δ
    • ෳ਺഑ஔ͞ΕͯͨΒ࡟আ͢Δ
    

    View Slide

19. RemoveDuplicates
    Pod Pod Pod
    

    View Slide

20. RemoveDuplicates
    Pod
    Pod
    Pod
    Pod ͕

    ଞͷ Node ʹҠಈ
    

    View Slide

21. RemoveDuplicates
    Pod
    Pod
    Node
    Pod
    Node ͕ੜ͖ฦͬͯ΋

    Pod ͸໭ͬͯ͜ͳ͍
    

    View Slide

22. RemoveDuplicates
    Pod
    Pod
    Pod
    Pod ͕

    ࠶εέδϡʔϦϯά͞ΕΔ
    Pod
    Run Descheduler
    

    View Slide

23. LowNodeUtilization
    • Ϧιʔεͷ࢖༻཰͕௿͍ Node ΁

    Ϧιʔεͷ࢖༻཰͕ߴ͍ Node ͔Β

    Pod Λ࠶εέδϡʔϧ͢Δ͜ͱΛ໨తͱ͍ͯ͠Δ
    • Pod ͷ Resource requests ͕ܭࢉʹ༻͍ΒΕΔ
    • thresholds ʹઃఆͨ͠ᮢ஋ΛԼճΔͱ

    Node ͷϦιʔε࢖༻཰͕௿͍ͱΈͳ͞ΕΔ
    • targetThresholds ʹઃఆͨ͠ᮢ஋Λ্ճΔͱ

    Node ͷϦιʔε࢖༻཰͕ߴ͍ͱΈͳ͞ΕΔ
    

    View Slide

24. LowNodeUtilization
    • Ϧιʔεͷ࢖༻཰͕௿͍ Node ΁

    Ϧιʔεͷ࢖༻཰͕ߴ͍ Node ͔Β

    Pod Λ࠶εέδϡʔϧ͢Δ
    • thresholds ʹઃఆͨ͠ᮢ஋ΛԼճΔͱ

    Node ͷϦιʔε࢖༻཰͕௿͍ͱΈͳ͞ΕΔ
    • targetThresholds ʹઃఆͨ͠ᮢ஋Λ্ճΔͱ

    Node ͷϦιʔε࢖༻཰͕ߴ͍ͱΈͳ͞ΕΔ
    • thresholds ʹઃఆͨ͠ᮢ஋ΛԼճΔͱ

    Node ͷϦιʔε࢖༻཰͕௿͍ͱΈͳ͞ΕΔ
    CPU / RAM / Pod ਺ ͷᮢ஋

    ԼճΔͱϦιʔε࢖༻཰͕௿͍
    Node ͱ൑ఆ͞ΕΔ
    

    View Slide

25. LowNodeUtilization
    • Ϧιʔεͷ࢖༻཰͕௿͍ Node ΁

    Ϧιʔεͷར༻཰͕ߴ͍ Node ͔Β

    Pod Λ࠶εέδϡʔϧ͢Δ
    • thresholds ʹઃఆͨ͠ᮢ஋ΛԼճΔͱ

    Node ͷϦιʔε࢖༻཰͕௿͍ͱΈͳ͞ΕΔ
    • targetThresholds ʹઃఆͨ͠ᮢ஋Λ্ճΔͱ

    Node ͷϦιʔε࢖༻཰͕ߴ͍ͱΈͳ͞ΕΔ
    CPU / RAM / Pod ਺ ͷᮢ஋

    ্ճΔͱϦιʔε࢖༻཰͕ߴ͍
    Node ͱ൑ఆ͞ΕΔ
    • targetThresholds ʹઃఆͨ͠ᮢ஋Λ্ճΔͱ

    Node ͷϦιʔε࢖༻཰͕ߴ͍ͱΈͳ͞ΕΔ
    

    View Slide

26. LowNodeUtilization
    Pod
    Pod
    Pod
    Pod
    Pod
    Pod
    thresholds:
    “pods": 2
    targetThresholds:
    “pods”: 2
    

    View Slide

27. LowNodeUtilization
    Pod
    Pod
    Pod
    Pod
    Pod
    Pod
    thresholds:
    “pods": 2
    targetThresholds:
    “pods”: 2
    Pod
    Pod
    Pod
    targetThresholds:
    “pods”: 2
    Pod ਺ 2 Λ௒͑ͨΒ

    ௥͍ग़͠ର৅
    

    View Slide

28. LowNodeUtilization
    Pod
    Pod
    Pod
    Pod
    Pod
    Pod
    thresholds:
    “pods": 2
    targetThresholds:
    “pods”: 2
    Pod
    thresholds:
    “pods": 2
    Pod ਺ 2 ະຬͳΒ

    ଞͷ Node ͔Β

    Pod ΛҠಈͤͯ͞΋͍͍
    

    View Slide

29. LowNodeUtilization
    Pod
    Pod
    Pod
    Pod
    Pod
    thresholds:
    “pods": 2
    targetThresholds:
    “pods”: 2
    Pod
    Pod
    Run Descheduler
    

    View Slide

30. RemovePodsViolatingInterPodAntiAffinity
    • Inter-Pod Anti-Affinity ʹ

    ҧ൓͍ͯ͠Δ Pod ͕࡟আ͞ΕΔ
    • Inter-Pod Anti-Affinity
    • ಛఆͷϥϕϧΛ࣋ͬͨ Pod ͸ಉ͡ Node ʹ

    εέδϡʔϦϯά͠ͳ͍Α͏ʹ͢Δ΍ͭ
    

    View Slide

31. Pod C
    RemovePodsViolatingInterPodAntiAffinity
    Pod B
    Pod A
    Node
    

    View Slide

32. Pod C
    RemovePodsViolatingInterPodAntiAffinity
    Pod B
    Pod A
    Node
    Pod C
    Pod B
    Pod A
    Pod ͕ Node Ͱ

    ಈ͍͍ͯΔঢ়ଶͰ
    Pod B ͱ Pod C ʹ

    Pod A ͱಉ͡ϊʔυͰͷ

    ࣮ߦΛېࢭ͢Δ 

    Anti-Affinity rule Λ

    ௥Ճ͢Δ
    

    View Slide

33. Pod C
    RemovePodsViolatingInterPodAntiAffinity
    Pod B
    Pod A
    Node
    Anti-Affinity ͸

    εέδϡʔϧ͞ΕΔ͔࣌͠

    ݟͯ͘Εͳ͍ͷͰ

    Pod ͸Ҡಈͯ͘͠Εͳ͍
    

    View Slide

34. Pod C
    RemovePodsViolatingInterPodAntiAffinity
    Pod B
    Pod A
    Pod ͕

    ࠶εέδϡʔϦϯά͞ΕΔ
    Pod A
    Run Descheduler
    

    View Slide

35. RemovePodsViolatingNodeAffinity
    • Node Affinity ʹҧ൓͍ͯ͠Δ Pod ͕࡟আ͞ΕΔ
    • Node Affinity
    • Pod Λಛఆͷ Node ʹ

    εέδϡʔϦϯά͢ΔΑ͏ʹ͢Δ΍ͭ
    

    View Slide

36. Pod C
    Pod B
    Pod A
    Node
    RemovePodsViolatingNodeAffinity
    node: gopher-a node: gopher-b
    

    View Slide

37. Pod C
    Pod B
    Pod A
    Node
    RemovePodsViolatingNodeAffinity
    node: gopherA node: gopherB
    Pod A
    node: gopher-a
    Pod C
    Pod B
    Pod A ͸
    node: gopher-a
    ͷϥϕϧͷ Node ʹ

    εέδϡʔϧ͢Δ

    Node Affinity
    

    View Slide

38. Pod C
    Pod B
    Pod A
    Node
    RemovePodsViolatingNodeAffinity
    node: gopher-a
    node: gopher-b
    Pod C
    Pod B
    Pod A
    Node
    node: gopher-a
    node: gopher-b
    ϥϕϧͷ෇͚ସ͑Λߦ͏
    

    View Slide

39. RemovePodsViolatingNodeAffinity
    Pod C
    Pod B
    Pod A
    Node
    node: gopher-a
    node: gopher-b
    Node Affinity ͸

    εέδϡʔϧ͞ΕΔ͔࣌͠

    ݟͯ͘Εͳ͍ͷͰ

    Pod ͸Ҡಈͯ͘͠Εͳ͍
    

    View Slide

40. RemovePodsViolatingNodeAffinity
    Pod C
    Pod B
    Pod A
    node: gopher-a
    node: gopher-b
    Pod A
    Run Descheduler Pod ͕

    ࠶εέδϡʔϦϯά͞ΕΔ
    

    View Slide

41. Pod ͷཱͪୀ͖
    • ҎԼͷϧʔϧͰߦΘΕΔ
    • Critical Pods ͸ର৅֎
    • Replica Set / Replication Controller / Deployment / Job

    ʹඥ͍͍ͮͯͳ͍ Pod ͸ର৅֎
    • DaemonSet ͷ Pod ͸ର৅֎
    • ϩʔΧϧετϨʔδΛ࣋ͭ Pod ͸ର৅֎
    • QoS classes ͕ BestEffort ͷ Pod ͸ Guaranteed/ Burstable

    ΑΓ΋༏ઌͯ͠࡟আ͞ΕΔ
    

    View Slide

42. ͓͠·͍
    Gopher ͘Μ͸͔͜͜Β: https://gopherize.me/
    

    View Slide