Upgrade to Pro — share decks privately, control downloads, hide ads and more …

kube-state-metrics Sharding

d-kuro
November 02, 2022

kube-state-metrics Sharding

d-kuro

November 02, 2022
Tweet

More Decks by d-kuro

Other Decks in Programming

Transcript

  1. kube-state-metrics Sharding
    TechFeed Experts Night#7 ʙ ίϯςφٕज़ΛޠΔ

    View Slide

  2. I work for
    @ponde_m @d-kuro

    View Slide

  3. kube-state-metrics ͱ͸
    • Kubernetes ͷ api-server Λ؂ࢹ֤ͯ͠छΦϒδΣΫτͷঢ়ଶͷ

    ϝτϦΫεΛఏڙͯ͘͠ΕΔ΍ͭ

    View Slide

  4. • γϟʔυͱݺ͹ΕΔখ͞ͳνϟϯΫʹσʔλΛ෼ׂ


    • ෳ਺ͷϚγϯͳͲʹ෼͚ͯอଘ͢Δ͜ͱͰ

    େن໛ͳσʔληοτΛ؅ཧ


    • Ԡ౴଎౓ͷ޲্ͳͲʹ༻͍ΒΕΔ
    Sharding

    View Slide

  5. • Α͋͘Δσʔλϕʔεͷ Sharding ͷҰྫ
    Sharding
    https://aws.amazon.com/jp/blogs/news/sharding-with-amazon-relational-database-service/

    View Slide

  6. Kubernetes controller ͷ Sharding
    • controller ͸௨ৗɺϦʔμʔબग़ͰϦʔμʔͱͳͬͨ Pod ͷΈ͕

    reconcile Λߦ͏ͱ͍͏Ξϓϩʔνʹͳ͍ͬͯΔ
    https://speakerdeck.com/daikurosawa/leader-election-in-kubernetes-number-k8sjp

    View Slide

  7. • ͪΐͬͱલʹ࿩୊ʹͳͬͨ΍ͭ


    • Sharder ͕ϥϕϧΛׂΓৼͬͯ

    ͲͷγϟʔυͰॲཧ͢Δ͔Λ

    ܾఆ͢Δ


    • ͬͪ͜ͷ࿩͸ࠓ೔͸͠·ͤΜ
    Kubernetes controller ͷ Sharding
    https://github.com/timebertt/thesis-controller-sharding/blob/master/assets/design-overview.pdf

    View Slide

  8. kube-state-metrics ͷ Sharding
    • ࠓճͷςʔϚ͸ controller Ͱ͸ͳ͍͚Ͳ

    kube-state-metrics ͕ controller Ͱ΋࢖༻Ͱ͖ΔΞϓϩʔνͰ

    Sharding Λߦ͏ػೳ͕࣮૷͞Ε͍ͯΔͨΊɺͦͷ঺հΛ͠·͢

    View Slide

  9. kube-state-metrics ͷ Sharding
    Pod1
    Pod2
    1. CM1 Metrics


    2. CM3 Metrics
    Export Metrics
    api-server
    1. CM1


    2. CM2


    3. CM3


    4. CM4
    Con
    fi
    gMap List
    1. CM2 Metrics


    2. CM4 Metrics
    Export Metrics

    View Slide

  10. kube-state-metrics ͷ Sharding
    https://github.com/kubernetes/kube-state-metrics/pull/613

    View Slide

  11. kube-state-metrics ͷ Sharding
    https://github.com/kubernetes/kube-state-metrics/pull/613
    γϟʔυʹ෼ׂ͢Δ͜ͱͰ


    ॲཧ͢ΔΦϒδΣΫτ͕ݮΓ


    ϨΠςϯγ͕গͳ͘ͳΔ

    View Slide

  12. ࣮૷
    • kube-state-metrics Λ StatefulSet Ͱಈ͔͢


    • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β

    Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ


    • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ

    Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ

    View Slide

  13. ࣮૷
    • kube-state-metrics Λ StatefulSet Ͱಈ͔͢


    • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β

    Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ


    • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ

    Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ

    View Slide

  14. Pod ໊ͳͲΛҾ਺ʹऔΔ

    View Slide

  15. Pod ͷ ownerReferences ͔Β


    StatefulSet Λऔಘ

    View Slide

  16. StatefulSet ͷϨϓϦΧ਺Λ


    τʔλϧͷγϟʔυ਺


    Pod ໊ͷ຤ඌ͔Βγϟʔυͷ


    ൪߸Λऔಘ

    View Slide

  17. StatefulSet Λ informer Ͱ watch


    EventHandler Ͱγϟʔυͷઃఆͱ
    metrics ͷϦηοτΛߦ͏

    View Slide

  18. ͜͏͢Δ͜ͱͰ HPA ͳͲͰ


    StatefulSet ͕εέʔϧͨ͠ࡍʹ


    γϟʔυͷ࠶ׂ౰ͯΛ

    ࣗಈͰߦ͏͜ͱ͕Ͱ͖Δ

    View Slide

  19. ࣮૷
    • kube-state-metrics Λ StatefulSet Ͱಈ͔͢


    • ϨϓϦΧ਺ͱ Pod ͷ຤ඌͷ൪߸͔Β

    Կ൪໨ͷγϟʔυͳͷ͔ݕग़͢Δ


    • Kubernetes ͷΦϒδΣΫτͷ UID Λ༻͍ͯ

    Consistent Hash ͠ɺॲཧ͢ΔγϟʔυΛܾఆ͢Δ

    View Slide

  20. client-go ͷΞʔΩςΫνϟ

    View Slide

  21. kube-state-metrics ͷΞʔΩςΫνϟ
    kube-state-metrics Ͱ΋ Re
    fl
    ector Λ࢖༻


    ਤͷΑ͏ʹ Delta FIFO Queue ʹ


    ΦϒδΣΫτΛ֨ೲ͢ΔͷͰͳ͘


    ϝτϦΫεΛ֨ೲ͢Δ


    MetricStore Λ࣮૷͍ͯ͠Δ

    View Slide

  22. List & Watch ͷ࣮૷ͷதͰ


    ͲͷγϟʔυͰॲཧ͢Δ͔ͷ


    ൑ఆΛߦ͍ͬͯΔ

    View Slide

  23. List & Watch ͷ࣮૷ͷதͰ


    ͲͷγϟʔυͰॲཧ͢Δ͔ͷ


    ൑ఆΛߦ͍ͬͯΔ

    View Slide

  24. Consistent Hash Ͱॲཧ͢ΔγϟʔυΛܾఆ͠ɺ


    ࣗγϟʔυͱҰக͢Δ৔߹ͷΈॲཧΛߦ͏

    View Slide

  25. ͜ͷ ListerWatcher Λ༻͍ͯ


    informer ͳΓΛੜ੒͢Ε͹ controller ͕࡞ΕΔ

    View Slide

  26. σϝϦοτ
    • StatefulSet ͷϩʔϧΞ΢τʹىҼ͢ΔσϝϦοτ


    • ϩʔϧΞ΢τ͕ 1 ୆ͣͭߦΘΕΔͨΊ஗͍


    • Pod ͷ࡟আ͕ઌʹ࣮ࢪ͞ΕΔ


    • ͜ͷλΠϛϯάͰ Prometheus ͷεΫϨΠϓ͕ൃੜ͢Δͱ

    ϝτϦΫεͷܽଛʹܨ͕ΔՄೳੑ͕͋Δ


    • List & Watch ͷ෦෼͸γϟʔσΟϯάͰ͖ͳ͍ͨΊɺ

    ର৅ͱͳΔશΦϒδΣΫτΛਫ਼ࠪ͢Δ෼ͷϦιʔε͕ඞཁͱͳΔ

    View Slide

  27. ·ͱΊ
    • controller ͷγϟʔσΟϯάʹ࢖͑ͦ͏ͳ

    kube-state-metrics Ͱ༻͍ΒΕ͍ͯΔγϟʔσΟϯάख๏Λ঺հ


    • StatefulSet ͷಛੑΛ্ख͘࢖͍ͬͯΔ


    • ΦϒδΣΫτͷ UID Ͱ Consistent Hash

    View Slide

  28. We are hiring
    Ͱ͸


    Yahoo! JAPAN ޲͚ʹϓϥοτϑΥʔϜΛ࡞Γ͍ͨΤϯδχΞઈࢍืूதͰ͢ʂ

    View Slide

  29. Thank You for Listening

    View Slide