kube-state-metrics Sharding

https://techfeed.io/events/techfeed-experts-night-7
で発表した資料です。

Transcript

1. kube-state-metrics Sharding
   TechFeed Experts Night#7 ʙ ίϯςφٕज़ΛޠΔ
   

   View Slide

2. I work for
   @ponde_m @d-kuro
   

   View Slide

3. kube-state-metrics ͱ͸
   • Kubernetes ͷ api-server Λ؂ࢹ֤ͯ͠छΦϒδΣΫτͷঢ়ଶͷ
   

   ϝτϦΫεΛఏڙͯ͘͠ΕΔ΍ͭ
   

   View Slide

4. • γϟʔυͱݺ͹ΕΔখ͞ͳνϟϯΫʹσʔλΛ෼ׂ
   
   
   • ෳ਺ͷϚγϯͳͲʹ෼͚ͯอଘ͢Δ͜ͱͰ
   

   େن໛ͳσʔληοτΛ؅ཧ
   
   
   • Ԡ౴଎౓ͷ޲্ͳͲʹ༻͍ΒΕΔ
   Sharding
   

   View Slide

5. • Α͋͘Δσʔλϕʔεͷ Sharding ͷҰྫ
   Sharding
   https://aws.amazon.com/jp/blogs/news/sharding-with-amazon-relational-database-service/
   

   View Slide

6. Kubernetes controller ͷ Sharding
   • controller ͸௨ৗɺϦʔμʔબग़ͰϦʔμʔͱͳͬͨ Pod ͷΈ͕
   

   reconcile Λߦ͏ͱ͍͏Ξϓϩʔνʹͳ͍ͬͯΔ
   https://speakerdeck.com/daikurosawa/leader-election-in-kubernetes-number-k8sjp
   

   View Slide

7. • ͪΐͬͱલʹ࿩୊ʹͳͬͨ΍ͭ
   
   
   • Sharder ͕ϥϕϧΛׂΓৼͬͯ
   

   ͲͷγϟʔυͰॲཧ͢Δ͔Λ
   

   ܾఆ͢Δ
   
   
   • ͬͪ͜ͷ࿩͸ࠓ೔͸͠·ͤΜ
   Kubernetes controller ͷ Sharding
   https://github.com/timebertt/thesis-controller-sharding/blob/master/assets/design-overview.pdf
   

   View Slide

8. kube-state-metrics ͷ Sharding
   • ࠓճͷςʔϚ͸ controller Ͱ͸ͳ͍͚Ͳ
   

   kube-state-metrics ͕ controller Ͱ΋࢖༻Ͱ͖ΔΞϓϩʔνͰ
   

   Sharding Λߦ͏ػೳ͕࣮૷͞Ε͍ͯΔͨΊɺͦͷ঺հΛ͠·͢
   

   View Slide

9. kube-state-metrics ͷ Sharding
   Pod1
   Pod2
   1. CM1 Metrics
   
   
   2. CM3 Metrics
   Export Metrics
   api-server
   1. CM1
   
   
   2. CM2
   
   
   3. CM3
   
   
   4. CM4
   Con
   fi
   gMap List
   1. CM2 Metrics
   
   
   2. CM4 Metrics
   Export Metrics
   

   View Slide

10. kube-state-metrics ͷ Sharding
    https://github.com/kubernetes/kube-state-metrics/pull/613
    

    View Slide

11. kube-state-metrics ͷ Sharding
    https://github.com/kubernetes/kube-state-metrics/pull/613
    γϟʔυʹ෼ׂ͢Δ͜ͱͰ
    
    
    ॲཧ͢ΔΦϒδΣΫτ͕ݮΓ
    
    
    ϨΠςϯγ͕গͳ͘ͳΔ
    

    View Slide

12. ࣮૷
    • kube-state-metrics Λ StatefulSet Ͱಈ͔͢
    
    
    • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β
    

    Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ
    
    
    • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ
    

    Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
    

    View Slide

13. ࣮૷
    • kube-state-metrics Λ StatefulSet Ͱಈ͔͢
    
    
    • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β
    

    Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ
    
    
    • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ
    

    Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
    

    View Slide

14. Pod ໊ͳͲΛҾ਺ʹऔΔ
    

    View Slide

15. Pod ͷ ownerReferences ͔Β
    
    
    StatefulSet Λऔಘ
    

    View Slide

16. StatefulSet ͷϨϓϦΧ਺Λ
    
    
    τʔλϧͷγϟʔυ਺
    
    
    Pod ໊ͷ຤ඌ͔Βγϟʔυͷ
    
    
    ൪߸Λऔಘ
    

    View Slide

17. StatefulSet Λ informer Ͱ watch
    
    
    EventHandler Ͱγϟʔυͷઃఆͱ
    metrics ͷϦηοτΛߦ͏
    

    View Slide

18. ͜͏͢Δ͜ͱͰ HPA ͳͲͰ
    
    
    StatefulSet ͕εέʔϧͨ͠ࡍʹ
    
    
    γϟʔυͷ࠶ׂ౰ͯΛ
    

    ࣗಈͰߦ͏͜ͱ͕Ͱ͖Δ
    

    View Slide

19. ࣮૷
    • kube-state-metrics Λ StatefulSet Ͱಈ͔͢
    
    
    • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β
    

    Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ
    
    
    • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ
    

    Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
    

    View Slide

20. client-go ͷΞʔΩςΫνϟ
    

    View Slide

21. kube-state-metrics ͷΞʔΩςΫνϟ
    kube-state-metrics Ͱ΋ Re
    fl
    ector Λ࢖༻
    
    
    ਤͷΑ͏ʹ Delta FIFO Queue ʹ
    
    
    ΦϒδΣΫτΛ֨ೲ͢ΔͷͰͳ͘
    
    
    ϝτϦΫεΛ֨ೲ͢Δ
    
    
    MetricStore Λ࣮૷͍ͯ͠Δ
    

    View Slide

22. List & Watch ͷ࣮૷ͷதͰ
    
    
    ͲͷγϟʔυͰॲཧ͢Δ͔ͷ
    
    
    ൑ఆΛߦ͍ͬͯΔ
    

    View Slide

23. List & Watch ͷ࣮૷ͷதͰ
    
    
    ͲͷγϟʔυͰॲཧ͢Δ͔ͷ
    
    
    ൑ఆΛߦ͍ͬͯΔ
    

    View Slide

24. Consistent Hash Ͱॲཧ͢ΔγϟʔυΛܾఆ͠ɺ
    
    
    ࣗγϟʔυͱҰக͢Δ৔߹ͷΈॲཧΛߦ͏
    

    View Slide

25. ͜ͷ ListerWatcher Λ༻͍ͯ
    
    
    informer ͳΓΛੜ੒͢Ε͹ controller ͕࡞ΕΔ
    

    View Slide

26. σϝϦοτ
    • StatefulSet ͷϩʔϧΞ΢τʹىҼ͢ΔσϝϦοτ
    
    
    • ϩʔϧΞ΢τ͕ 1 ୆ͣͭߦΘΕΔͨΊ஗͍
    
    
    • Pod ͷ࡟আ͕ઌʹ࣮ࢪ͞ΕΔ
    
    
    • ͜ͷλΠϛϯάͰ Prometheus ͷεΫϨΠϓ͕ൃੜ͢Δͱ
    

    ϝτϦΫεͷܽଛʹܨ͕ΔՄೳੑ͕͋Δ
    
    
    • List & Watch ͷ෦෼͸γϟʔσΟϯάͰ͖ͳ͍ͨΊɺ
    

    ର৅ͱͳΔશΦϒδΣΫτΛਫ਼ࠪ͢Δ෼ͷϦιʔε͕ඞཁͱͳΔ
    

    View Slide

27. ·ͱΊ
    • controller ͷγϟʔσΟϯάʹ࢖͑ͦ͏ͳ
    

    kube-state-metrics Ͱ༻͍ΒΕ͍ͯΔγϟʔσΟϯάख๏Λ঺հ
    
    
    • StatefulSet ͷಛੑΛ্ख͘࢖͍ͬͯΔ
    
    
    • ΦϒδΣΫτͷ UID Ͱ Consistent Hash
    

    View Slide

28. We are hiring
    Ͱ͸
    
    
    Yahoo! JAPAN ޲͚ʹϓϥοτϑΥʔϜΛ࡞Γ͍ͨΤϯδχΞઈࢍืूதͰ͢ʂ
    

    View Slide

29. Thank You for Listening
    

    View Slide