Upgrade to Pro — share decks privately, control downloads, hide ads and more …

kube-state-metrics Sharding

d-kuro
November 02, 2022

kube-state-metrics Sharding

d-kuro

November 02, 2022
Tweet

More Decks by d-kuro

Other Decks in Programming

Transcript

  1. kube-state-metrics Sharding TechFeed Experts Night#7 ʙ ίϯςφٕज़ΛޠΔ

  2. I work for @ponde_m @d-kuro

  3. kube-state-metrics ͱ͸ • Kubernetes ͷ api-server Λ؂ࢹ֤ͯ͠छΦϒδΣΫτͷঢ়ଶͷ 
 ϝτϦΫεΛఏڙͯ͘͠ΕΔ΍ͭ

  4. • γϟʔυͱݺ͹ΕΔখ͞ͳνϟϯΫʹσʔλΛ෼ׂ • ෳ਺ͷϚγϯͳͲʹ෼͚ͯอଘ͢Δ͜ͱͰ 
 େن໛ͳσʔληοτΛ؅ཧ • Ԡ౴଎౓ͷ޲্ͳͲʹ༻͍ΒΕΔ Sharding

  5. • Α͋͘Δσʔλϕʔεͷ Sharding ͷҰྫ Sharding https://aws.amazon.com/jp/blogs/news/sharding-with-amazon-relational-database-service/

  6. Kubernetes controller ͷ Sharding • controller ͸௨ৗɺϦʔμʔબग़ͰϦʔμʔͱͳͬͨ Pod ͷΈ͕ 


    reconcile Λߦ͏ͱ͍͏Ξϓϩʔνʹͳ͍ͬͯΔ https://speakerdeck.com/daikurosawa/leader-election-in-kubernetes-number-k8sjp
  7. • ͪΐͬͱલʹ࿩୊ʹͳͬͨ΍ͭ • Sharder ͕ϥϕϧΛׂΓৼͬͯ 
 ͲͷγϟʔυͰॲཧ͢Δ͔Λ 
 ܾఆ͢Δ •

    ͬͪ͜ͷ࿩͸ࠓ೔͸͠·ͤΜ Kubernetes controller ͷ Sharding https://github.com/timebertt/thesis-controller-sharding/blob/master/assets/design-overview.pdf
  8. kube-state-metrics ͷ Sharding • ࠓճͷςʔϚ͸ controller Ͱ͸ͳ͍͚Ͳ 
 kube-state-metrics ͕

    controller Ͱ΋࢖༻Ͱ͖ΔΞϓϩʔνͰ 
 Sharding Λߦ͏ػೳ͕࣮૷͞Ε͍ͯΔͨΊɺͦͷ঺հΛ͠·͢
  9. kube-state-metrics ͷ Sharding Pod1 Pod2 1. CM1 Metrics 2. CM3

    Metrics Export Metrics api-server 1. CM1 2. CM2 3. CM3 4. CM4 Con fi gMap List 1. CM2 Metrics 2. CM4 Metrics Export Metrics
  10. kube-state-metrics ͷ Sharding https://github.com/kubernetes/kube-state-metrics/pull/613

  11. kube-state-metrics ͷ Sharding https://github.com/kubernetes/kube-state-metrics/pull/613 γϟʔυʹ෼ׂ͢Δ͜ͱͰ ॲཧ͢ΔΦϒδΣΫτ͕ݮΓ ϨΠςϯγ͕গͳ͘ͳΔ

  12. ࣮૷ • kube-state-metrics Λ StatefulSet Ͱಈ͔͢ • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β

    
 Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ 
 Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
  13. ࣮૷ • kube-state-metrics Λ StatefulSet Ͱಈ͔͢ • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β

    
 Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ 
 Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
  14. Pod ໊ͳͲΛҾ਺ʹऔΔ

  15. Pod ͷ ownerReferences ͔Β StatefulSet Λऔಘ

  16. StatefulSet ͷϨϓϦΧ਺Λ τʔλϧͷγϟʔυ਺ Pod ໊ͷ຤ඌ͔Βγϟʔυͷ ൪߸Λऔಘ

  17. StatefulSet Λ informer Ͱ watch EventHandler Ͱγϟʔυͷઃఆͱ metrics ͷϦηοτΛߦ͏

  18. ͜͏͢Δ͜ͱͰ HPA ͳͲͰ StatefulSet ͕εέʔϧͨ͠ࡍʹ γϟʔυͷ࠶ׂ౰ͯΛ 
 ࣗಈͰߦ͏͜ͱ͕Ͱ͖Δ

  19. ࣮૷ • kube-state-metrics Λ StatefulSet Ͱಈ͔͢ • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β

    
 Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ 
 Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
  20. client-go ͷΞʔΩςΫνϟ

  21. kube-state-metrics ͷΞʔΩςΫνϟ kube-state-metrics Ͱ΋ Re fl ector Λ࢖༻ ਤͷΑ͏ʹ Delta

    FIFO Queue ʹ ΦϒδΣΫτΛ֨ೲ͢ΔͷͰͳ͘ ϝτϦΫεΛ֨ೲ͢Δ MetricStore Λ࣮૷͍ͯ͠Δ
  22. List & Watch ͷ࣮૷ͷதͰ ͲͷγϟʔυͰॲཧ͢Δ͔ͷ ൑ఆΛߦ͍ͬͯΔ

  23. List & Watch ͷ࣮૷ͷதͰ ͲͷγϟʔυͰॲཧ͢Δ͔ͷ ൑ఆΛߦ͍ͬͯΔ

  24. Consistent Hash Ͱॲཧ͢ΔγϟʔυΛܾఆ͠ɺ ࣗγϟʔυͱҰக͢Δ৔߹ͷΈॲཧΛߦ͏

  25. ͜ͷ ListerWatcher Λ༻͍ͯ informer ͳΓΛੜ੒͢Ε͹ controller ͕࡞ΕΔ

  26. σϝϦοτ • StatefulSet ͷϩʔϧΞ΢τʹىҼ͢ΔσϝϦοτ • ϩʔϧΞ΢τ͕ 1 ୆ͣͭߦΘΕΔͨΊ஗͍ • Pod

    ͷ࡟আ͕ઌʹ࣮ࢪ͞ΕΔ • ͜ͷλΠϛϯάͰ Prometheus ͷεΫϨΠϓ͕ൃੜ͢Δͱ 
 ϝτϦΫεͷܽଛʹܨ͕ΔՄೳੑ͕͋Δ • List & Watch ͷ෦෼͸γϟʔσΟϯάͰ͖ͳ͍ͨΊɺ 
 ର৅ͱͳΔશΦϒδΣΫτΛਫ਼ࠪ͢Δ෼ͷϦιʔε͕ඞཁͱͳΔ
  27. ·ͱΊ • controller ͷγϟʔσΟϯάʹ࢖͑ͦ͏ͳ 
 kube-state-metrics Ͱ༻͍ΒΕ͍ͯΔγϟʔσΟϯάख๏Λ঺հ • StatefulSet ͷಛੑΛ্ख͘࢖͍ͬͯΔ

    • ΦϒδΣΫτͷ UID Ͱ Consistent Hash
  28. We are hiring Ͱ͸ Yahoo! JAPAN ޲͚ʹϓϥοτϑΥʔϜΛ࡞Γ͍ͨΤϯδχΞઈࢍืूதͰ͢ʂ

  29. Thank You for Listening