kube-state-metrics Sharding

https://techfeed.io/events/techfeed-experts-night-7
で発表した資料です。

Transcript

1. kube-state-metrics Sharding TechFeed Experts Night#7 ʙ ίϯςφٕज़ΛޠΔ
2. I work for @ponde_m @d-kuro
3. kube-state-metrics ͱ͸ • Kubernetes ͷ api-server Λ؂ࢹ֤ͯ͠छΦϒδΣΫτͷঢ়ଶͷ 
 ϝτϦΫεΛఏڙͯ͘͠ΕΔ΍ͭ
4. • γϟʔυͱݺ͹ΕΔখ͞ͳνϟϯΫʹσʔλΛ෼ׂ • ෳ਺ͷϚγϯͳͲʹ෼͚ͯอଘ͢Δ͜ͱͰ 
 େن໛ͳσʔληοτΛ؅ཧ • Ԡ౴଎౓ͷ޲্ͳͲʹ༻͍ΒΕΔ Sharding
5. • Α͋͘Δσʔλϕʔεͷ Sharding ͷҰྫ Sharding https://aws.amazon.com/jp/blogs/news/sharding-with-amazon-relational-database-service/
6. Kubernetes controller ͷ Sharding • controller ͸௨ৗɺϦʔμʔબग़ͰϦʔμʔͱͳͬͨ Pod ͷΈ͕ 
 reconcile Λߦ͏ͱ͍͏Ξϓϩʔνʹͳ͍ͬͯΔ https://speakerdeck.com/daikurosawa/leader-election-in-kubernetes-number-k8sjp
7. • ͪΐͬͱલʹ࿩୊ʹͳͬͨ΍ͭ • Sharder ͕ϥϕϧΛׂΓৼͬͯ 
 ͲͷγϟʔυͰॲཧ͢Δ͔Λ 
 ܾఆ͢Δ • ͬͪ͜ͷ࿩͸ࠓ೔͸͠·ͤΜ Kubernetes controller ͷ Sharding https://github.com/timebertt/thesis-controller-sharding/blob/master/assets/design-overview.pdf
8. kube-state-metrics ͷ Sharding • ࠓճͷςʔϚ͸ controller Ͱ͸ͳ͍͚Ͳ 
 kube-state-metrics ͕ controller Ͱ΋࢖༻Ͱ͖ΔΞϓϩʔνͰ 
 Sharding Λߦ͏ػೳ͕࣮૷͞Ε͍ͯΔͨΊɺͦͷ঺հΛ͠·͢
9. kube-state-metrics ͷ Sharding Pod1 Pod2 1. CM1 Metrics 2. CM3 Metrics Export Metrics api-server 1. CM1 2. CM2 3. CM3 4. CM4 Con fi gMap List 1. CM2 Metrics 2. CM4 Metrics Export Metrics
10. kube-state-metrics ͷ Sharding https://github.com/kubernetes/kube-state-metrics/pull/613
11. kube-state-metrics ͷ Sharding https://github.com/kubernetes/kube-state-metrics/pull/613 γϟʔυʹ෼ׂ͢Δ͜ͱͰ ॲཧ͢ΔΦϒδΣΫτ͕ݮΓ ϨΠςϯγ͕গͳ͘ͳΔ
12. ࣮૷ • kube-state-metrics Λ StatefulSet Ͱಈ͔͢ • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β 
 Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ 
 Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
13. ࣮૷ • kube-state-metrics Λ StatefulSet Ͱಈ͔͢ • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β 
 Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ 
 Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
14. Pod ໊ͳͲΛҾ਺ʹऔΔ
15. Pod ͷ ownerReferences ͔Β StatefulSet Λऔಘ
16. StatefulSet ͷϨϓϦΧ਺Λ τʔλϧͷγϟʔυ਺ Pod ໊ͷ຤ඌ͔Βγϟʔυͷ ൪߸Λऔಘ
17. StatefulSet Λ informer Ͱ watch EventHandler Ͱγϟʔυͷઃఆͱ metrics ͷϦηοτΛߦ͏
18. ͜͏͢Δ͜ͱͰ HPA ͳͲͰ StatefulSet ͕εέʔϧͨ͠ࡍʹ γϟʔυͷ࠶ׂ౰ͯΛ 
 ࣗಈͰߦ͏͜ͱ͕Ͱ͖Δ
19. ࣮૷ • kube-state-metrics Λ StatefulSet Ͱಈ͔͢ • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β 
 Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ 
 Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ
20. client-go ͷΞʔΩςΫνϟ
21. kube-state-metrics ͷΞʔΩςΫνϟ kube-state-metrics Ͱ΋ Re fl ector Λ࢖༻ ਤͷΑ͏ʹ Delta FIFO Queue ʹ ΦϒδΣΫτΛ֨ೲ͢ΔͷͰͳ͘ ϝτϦΫεΛ֨ೲ͢Δ MetricStore Λ࣮૷͍ͯ͠Δ
22. List & Watch ͷ࣮૷ͷதͰ ͲͷγϟʔυͰॲཧ͢Δ͔ͷ ൑ఆΛߦ͍ͬͯΔ
23. List & Watch ͷ࣮૷ͷதͰ ͲͷγϟʔυͰॲཧ͢Δ͔ͷ ൑ఆΛߦ͍ͬͯΔ
24. Consistent Hash Ͱॲཧ͢ΔγϟʔυΛܾఆ͠ɺ ࣗγϟʔυͱҰக͢Δ৔߹ͷΈॲཧΛߦ͏
25. ͜ͷ ListerWatcher Λ༻͍ͯ informer ͳΓΛੜ੒͢Ε͹ controller ͕࡞ΕΔ
26. σϝϦοτ • StatefulSet ͷϩʔϧΞ΢τʹىҼ͢ΔσϝϦοτ • ϩʔϧΞ΢τ͕ 1 ୆ͣͭߦΘΕΔͨΊ஗͍ • Pod ͷ࡟আ͕ઌʹ࣮ࢪ͞ΕΔ • ͜ͷλΠϛϯάͰ Prometheus ͷεΫϨΠϓ͕ൃੜ͢Δͱ 
 ϝτϦΫεͷܽଛʹܨ͕ΔՄೳੑ͕͋Δ • List & Watch ͷ෦෼͸γϟʔσΟϯάͰ͖ͳ͍ͨΊɺ 
 ର৅ͱͳΔશΦϒδΣΫτΛਫ਼ࠪ͢Δ෼ͷϦιʔε͕ඞཁͱͳΔ
27. ·ͱΊ • controller ͷγϟʔσΟϯάʹ࢖͑ͦ͏ͳ 
 kube-state-metrics Ͱ༻͍ΒΕ͍ͯΔγϟʔσΟϯάख๏Λ঺հ • StatefulSet ͷಛੑΛ্ख͘࢖͍ͬͯΔ • ΦϒδΣΫτͷ UID Ͱ Consistent Hash
28. We are hiring Ͱ͸ Yahoo! JAPAN ޲͚ʹϓϥοτϑΥʔϜΛ࡞Γ͍ͨΤϯδχΞઈࢍืूதͰ͢ʂ
29. Thank You for Listening