$30 off During Our Annual Pro Sale. View Details »

Horizontal Pod Autoscalers / v18.04 #containerdaysjp

Horizontal Pod Autoscalers / v18.04 #containerdaysjp

Kubernetes の Custom MetricsとResource Metrics を使った Multi Metrics でスケールさせる話をしました

at Japan Container Days v18.04

Kodai Sakabe

April 19, 2018
Tweet

More Decks by Kodai Sakabe

Other Decks in Technology

Transcript

  1. HORIZONTAL POD AUTOSCALERS
    — Custom Metrics ͱ Resource Metrics —
    JAPAN CONTAINER DAYS V18.04
    19 April 2018
    Wantedly, Inc. | Kodai Sakabe @koudaiii
    1

    View Slide

  2. CREATE
    A WORLD WHERE WORK
    DRIVES PASSION.
    Wa n t e d l y , I n c .
    2

    View Slide

  3. OUR GOALS
    W a n t e d l y , I n c .
    D i s c o v e r
    Ͱ ͋ ͍
    C o n n e c t
    ͭ ͳ ͕ Γ
    E n g a g e
    ͭ ͳ ͕ Γ Λ ਂ Ί Δ
    S N S
    Ϗ δ ω ε S N S
    C r e a te a w o r l d w h e r e w o r k d r i v e s p a s s i o n
    3

    View Slide

  4. Microservices
    o u r p r o d u c t s
    Wa n te d l y Vi s i t , Wa n te d l y C h a t , Wa n te d l y P e o p l e
    ࡾ ͭ ͷ ϓ ϩ μΫ τ
    QA,Production ؚΊͯ
    αʔϏε਺
    124
    M i c r o s e r v i c e s
    GKE + k8s on AWS
    80+
    K u b e r n e t e s N o d e
    Auto Scale ίϯςφͷ਺
    ͸Մม
    3K+
    C o n t a i n e r s
    4

    View Slide

  5. Relative vs. Absolute metrics
    Custom Metrics and Resource Metrics
    Prometheus Operator and Prometheus
    Custom Metrics API Server
    Kubernetes
    H o w t o u s e H o r i z o n t a l P o d A u t o s c a l e r s
    5

    View Slide

  6. Horizontal Pod
    Autoscalers
    “T h e H o r i z o n t a l P o d A u t o s c a l e r a u t o m a t i c a l l y
    s c a l e s t h e n u m b e r o f p o d s i n a r e p l i c a t i o n
    c o n t r o l l e r , d e p l o y m e n t o r r e p l i c a s e t b a s e d o n
    o b s e r v e d C P U u t i l i z a t i o n ( o r , w i t h c u s t o m
    m e t r i c s s u p p o r t , o n s o m e o t h e r a p p l i c a t i o n -
    p r o v i d e d m e t r i c s ) . N o t e t h a t H o r i z o n t a l P o d
    A u t o s c a l i n g d o e s n o t a p p l y t o o b j e c t s t h a t c a n ’ t
    b e s c a l e d , f o r e x a m p l e , D a e m o n S e t s . “
    εέ ʔϧ ͞ ͤ ͨ ͍ T a r g e t Λ ࢦ ఆ
    Ұ ఆ ؒ ִ ( 3 0 s ) Ͱ C P U ͷ ࢖ ༻ ཰ Λ Ϟχ λ Ϧ ϯ ά
    ར ༻ ऀ ͸ ɺ ༧ Ί r e p l i c a ͷ ਺ ͱ ᮢ ஋ ઃ ఆ
    T a r g e t N u m O f P o d s =
    c e i l ( s u m ( C u r r e n t P o d s C P U U t i l i z a t i o n )
    / T a r g e t )
    6
    h t t p s : // k u b e r n e t e s . i o /d o c s / t a s k s /r u n - a p p l i c a t i o n / h o r i z o n t a l - p o d - a u t o s c a l e /
    h t t p s : //g i t h u b . c o m / k u b e r n e t e s /c o m m u n i t y / b l o b / 1 a 6 5 6 3 2 0 9 0 6 4 5 3 7 9 d b 5 9 d 0 b 5 d 7 7c 5 0 2 d 1 7 1 a e 0 e c /c o n t r i b u t o r s /d e s i g n - p r o p o s a l s /a u t o s c a l i n g / h o r i z o n t a l - p o d - a u t o s c a l e r . m d # a u t o s c a l i n g - a l g o r i t h m

    View Slide

  7. Why?
    ಥ વ ͷ S p i k e ʹ উ ͯ Δ ?
    ϝσΟΞͷಛूͳͲͷ̋̋๒͸ɺීஈͷԿඦഒͷΞΫηεʹͳΔɻͦͷதͰɺ͋ΔαʔϏεͩ
    ͚Πϯελϯε͔ΒScaleͤ͞Δ৔߹ɺؒҧ͍ͳͦ͘͜Ͱ٧·Δɻ
    ಥ વ ͷ ε ύ ΠΫ ʹ ΋ উ ͯ Δ Π ϯ ϑ ϥ
    Microservices Λਪ঑͍ͯ͠·͢ɻग़དྷΔݶΓۉҰͰඵ୯ҐͰScaleΛͰ͖ΔΑ͏ʹ͢Δඞཁɻ
    M i c r o s e r v i c e s Λ ͢ Δ ্ Ͱ S c a l e ͷ ՝ ୊ Λ ແ ͘ ͢
    Scaleઃఆ͕खؒͰɺؾ͕ͭ͘ͱ Distributed Monolith* ʹͳΔɻ୭Ͱ΋ Scale Ͱ͖ΔΑ͏ʹ
    ͢Δɻ
    ৽ ن α ʔ Ϗ ε ΋ ൓ ө Ͱ ͖ Δ G e n e r a l ͳ ઃ ఆ Λ
    7
    * https://www.simplethread.com/youre-not-actually-building-microservices/

    View Slide

  8. Performance
    Tuning
    l o a d t e s t i n g
    ґ ଘ ؔ ܎ ͕ ͋ Δ α ʔ Ϗ ε Մ ࢹ Խ
    γ φ Ϧ Φ ࡞ ੒
    ෛ ՙ ૷ ஔ
    εέ ʔϧ ͢ Δ ج ४ ఆ ٛ
    K u b e r n e t e s Խ
    H o r i z o n t a l P o d A u t o s c a l e r s
    μο γ ϡ Ϙ ʔ υ Ͱ ؅ ཧ
    Τ ϯ υ ϙ Π ϯ τ ຖ ʹ Ϙ τϧ ω ο Ϋ Λ ௵ ͢
    8

    View Slide

  9. Relative vs. Absolute
    M e t r i c s ͷ ύ ϥ μ Π Ϝ γ ϑ τ
    “We chose values of the target metric to be relative (e.g. 90% of requested CPU resource) rather than
    absolute (e.g. 0.6 core) for the following reason. If we choose absolute metric, user will need to
    guarantee that the target is lower than the request. Otherwise, overloaded pods may not be able to
    consume more than the autoscaler's absolute target utilization, thereby preventing the autoscaler
    from seeing high enough utilization to trigger it to scale up. This may be especially troublesome when
    user changes requested resources for a pod because they would need to also change the autoscaler
    utilization threshold. Therefore, we decided to choose relative metric. For user, it is enough to set it to a
    value smaller than 100%, and further changes of requested resources will not invalidate it.”
    ࠓ·Ͱ͍͔ͭ͘ݻఆͷᮢ஋ϞχλϦϯάઃఆ΍ϝτϦΫεΛऔ͍ͬͯͨͷΛશͯɺׂ߹ʹมߋ͠·ͨ͠ɻ͜ΕͰ͍
    ͭ͘ίϯςφ͕ಈ͍͍ͯΔ͔೺Ѳ͢Δඞཁ͸ͳ͘ɺશମͷ
    ݸ਺͔ΒԿ%ҡ࣋Ͱ͖͍ͯΔͷ͔Λ؅ཧ͢Δख๏ʹύϥμΠϜγϑτ͠·ͨ͠ɻ
    9
    https://github.com/kubernetes/community/blob/1a65632090645379db59d0b5d77c502d171ae0ec/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#relative-vs-absolute-metrics

    View Slide

  10. o n K u b e r n e t e s M e t r i c s & A u t o s c a l e r s C u s t o m m e t r i c s
    εέ ʔϧ ౷ Ұ
    α ʔ Ϗ ε σΟε Χ
    ό Ϧ
    Π ϯε λ ϯε ͷ ε
    έ ʔϧ ୤ ٫
    H o r i z o n t a l P o d
    A u to s c a l e r s v 1
    C P U ࢖ ༻ ཰
    μο γ ϡ Ϙʔ υ
    C P U
    M e m o r y
    R e q u e s t
    Q u e r y
    g o r o u t i n e
    e tc . .
    Microservice ͷ Scale
    ߋ ʹ ౿ Έ ࠐ Μ ͩ ε έ ʔ ϥ Ϗ Ϧ ς Ο Λ
    ί ϯ ς φ ෦ ෼ · Ͱ ؅ ཧ ͕ ඞ ཁ ͩ ͬ ͨ ͷ ͕ɺ ػ ց ͕ ϝ ϯ ς φ ϯε Λ ͓ ͜ ͳ ͏ ͜ ͱ Ͱɺ Π ϯ ϑ ϥ ࣄ ଶ ͷ ೳ ྗ ͕ ্ ͕ Γ · ͠
    ͨ ɻ ͞ Β ʹ εέ ʔ ϥ Ϗ Ϧ ςΟ Λ ߴ Ί Δ ͨ Ί ʹ ࣍ ΁ ͷ ε ς οϓ ͱ ͯ͠ H o r i z o n t a l P o d A u to s c a l e r s Λ ౿ Έ ࠐ ΜͰ
    ࢖ ͑Δ Α ͏ ʹ ͠ ͨ ͍ ͱ ͍ ͏ ε ς ʔ δ ΁ ཱ ͭ ͜ ͱ ʹ ͳ Γ · ͠ ͨ ɻ
    S T A R T
    Ͳ ͜ · Ͱ ෛ ՙ ͔ ͚ ͨ Β མ
    ͪ Δ ?
    क Δ ΂ ͖ ΋ ͷ ͸ ?
    10

    View Slide

  11. * Heapster: a application which
    gathers metrics, writes them to
    metrics storage "sinks", and exposes
    the resource metrics API from in-
    memory storage.
    * Metrics Server: a lighter-weight in-
    memory server specifically for the
    resource metrics API.
    R E S O U R C E M E T R I C S
    * Prometheus Adapter. An
    implementation of the custom
    metrics API that attempts to
    support arbitrary metrics following
    a set label and naming scheme.
    * Google Stackdriver
    C U S T O M M E T R I C S
    Multiple Metrics
    ೋ ͭ ͷ M e t r i c s Λ ࢖ ͬ ͨ
    11
    https://github.com/kubernetes/metrics/blob/dba7415d399b043dd64026e26a26aeb03dbbc220/IMPLEMENTATIONS.md

    View Slide

  12. S e t u p k 8 s P r o m e t h e u s C u s t o m m e t r i c s A P I S e r v e r
    o p t i o n ༗ ޮ Խ
    M e t r i c s S e r v e r
    Pr o m e t h e u s
    Pr o m e t h e u s
    O p e r a to r
    k 8 s - p r o m e t h e u s -
    a d a p te r
    S e r v i c e M o n i to r
    Using k8s-prometheus-adapter
    C u s t o m M e t r i c s ༗ ޮ Խ
    G K E ͷ ৔ ߹ ͸ ɺ S t a c k D r i v e r ͕ ࢖ ͑ · ͢ ͕ɺ ࠓ ճ ͸ ฐ ࣾ ؀ ڥ Ͱ ͋ Δ k u b e r n e te s / ko p s ؀ ڥ Ͱ ͷ ྫ ʹ ͳ Γ · ͢ɻ
    ͦ ͷ ࡍ ʹ ඞ ཁ ͳ ख ॱ ͕ େ ͖ ͘ 4 S T E P ඞ ཁ Ͱ ͢ɻ
    S e t A p p l i c a t i o n
    /m e t r i c s ௥ Ճ
    S e r v i c e ௥ Ճ
    H PA ͷ ઃ ఆ
    12
    Set up k8s: https://github.com/kubernetes/kops/blob/1924ce0d50ce59f839dde91c642e6753222ae14d/docs/horizontal_pod_autoscaling.md#cluster-configuration
    Prometheus https://github.com/luxas/kubeadm-workshop/tree/592921a9754939b6db435c99e4f9377ddd5a98df#deploying-the-prometheus-operator-for-monitoring-services-in-the-cluster
    Custom Metrics api Server https://github.com/luxas/kubeadm-workshop/tree/592921a9754939b6db435c99e4f9377ddd5a98df#deploying-a-custom-metrics-api-server-and-a-sample-app

    View Slide

  13. X
    Prometheus
    Service Monitor
    namespaceSelector: ʮany: trueʯʹ
    ͢Δ͜ͱͰશͯͷnamespace ͕ର৅
    endpoints: ʹ͸ɺݟʹߦ͘αʔϏεͷ
    port ͷ name Λࢦఆ
    13
    https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html

    View Slide

  14. Custom Metrics
    API Server
    k 8 s - p r o m e t h e u s - a d a p t e r
    P r o m e t h e u s ͱ k 8 s A P I ͷ ڮ ౉ ͠
    ͜ ͷ ઃ ఆ Α Γ ɺ
    H PA Ϛ χ ϑ Σ ε τ
    S e r v i c e Ϛ χ ϑ Σ ε τ
    /m e t r i c s ௥ Ճ
    14
    h t t p s : //g i t h u b . c o m / D i r e c t X M a n 1 2 / k 8 s - p r o m e t h e u s - a d a p t e r/ t r e e / 8 3 a 7d d 5 d 5 e 6 9 6 9 b b f 5 3 8 0 d 4 1 c 9 c 0 e 2 b 8 3 4 3 2 7 3 3 2 # e x a m p l e
    h t t p s : //g i t h u b . c o m / l u x a s / k u b e a d m - w o r k s h o p / t r e e / 5 9 2 9 2 1 a 9 7 5 4 9 3 9 b 6 d b 4 3 5 c 9 9 e 4 f 9 3 7 7d d d 5 a 9 8 d f # d e p l o y i n g - a - c u s t o m - m e t r i c s - a p i - s e r v e r - a n d - a - s a m p l e - a p p

    View Slide

  15. Application
    C u s t o m M e t r i c s Λ ར ༻
    Ϋ ϥε λ ͷ η ο τΞ οϓ ͕ ऴ Θ Δ ͱ ɺ ࣮ ࡍ ʹ α ʔ Ϗ ε ʹ ൓ ө ͞ ͤ ͯ ͍ ͖ · ͢
    ServiceMoniter Ͱࢦఆͨ͠
    Service Λ·ͣ͸ϝτϦΫεΛड
    ͚औΔઐ༻ͱͯ͠࡞੒͠·͢
    S e r v i c e ࡞ ੒
    ͲͷΑ͏ʹ Custom Metrics ͱ
    Resource Metrics Λ࢖ͬͯε
    έʔϧͤ͞Δ͔ΛHPA ϚχϑΣ
    ετΛ࡞੒͠·͢
    H PA ࡞ ੒
    /metrics ࣮૷
    or
    nginx ͕લஈʹ͍Δ৔߹
    nginx_exporter Λ sidecar ༻ҙ
    /m e t r i c s ࣮ ૷
    15
    https://github.com/discordianfish/nginx_exporter

    View Slide

  16. X
    Prometheus
    nginx_exporter
    n g i n x _ e x p o r t e r ͸ ɺ /m e t r i c s Λ ։ ์
    ؂ ࢹ ͠ ͨ ͍ n g i n x ʹ ର ͯ͠ /n g i n x _ s t a t u s ʹ
    ΞΫ η ε ͠ ஋ Λ औ ಘ ͯ͠ P r o m e t h e u s ༻ ʹ ग़ ྗ
    P r o m e t h e u s - > n g i n x _ e x p o r t e r - >
    n g i n x ( n g i n x _ s t a t u s )
    16
    https://github.com/discordianfish/nginx_exporter

    View Slide

  17. X
    Monitoring
    η ο τ Ξ ο ϓ ޙ ࢼ ݧ
    $ hey -n 20000 -c 8
    ϐʔΫʹୡ͢Δલʹεέʔϧ͢Δ
    17

    View Slide

  18. ·ͱΊ
    C o n c l u s i o n
    ෳ਺ͷϢʔεʔέʔεʹԠͯ͡εέʔϧग़དྷΔ
    μογϡϘʔυ΍ෛՙςετΛݩʹෛՙ͕ߴ·Δલʹ Scale ͓ͯ͘͠ͱ͍ͬͨ͜ͱ΋Ͱग़དྷΔ
    C u s t o m M e t r i c s ͱ R e s o u r c e M e t r i c s Ͱ S c a l e ͷ ෯ Λ ࣋ ͨ ͤ Δ
    Prometheus Λ௨ͯ͠ɺ goroutine ͷ਺΍ಛఆͷΤϯυϙΠϯτ౳༷ʑͳܗͰ Custom
    Metrics ͕࢖͑ΔΑ͏ʹͳΔɻ
    P r o m e t h e u s / C u s t o m M e t r i c s A P I S e r v e r / k 8 s ͷ ઃ ఆ
    Prometheus ͷϥΠϒϥϦΛೖΕΔ͜ͱͰɺΞϓϦέʔγϣϯʹ `/metrics` Λ௥Ճ͢Δͱ͍
    ͏ํ๏ͱ Nginx Λલஈʹ഑ஔ͍ͯ͠Δ৔߹͸ɺnginx_exporter Λ sidecar ʹೖΕΔ͜ͱͰ
    http ͷϦΫΤετɺίωΫγϣϯ਺͔Β࢝ΊΒΕ·͢
    S e r v i c e / H o r i z o n t a l P o d A u t o S c a l e / n g i n x _ e x p o r t e r
    18

    View Slide

  19. * Distributed Tracing αʔϏεؒ௨৴
    ՄࢹԽ
    * Trafic Monitoring RequestͷՄࢹԽ
    * Fault Tolerant ϦτϥΠճ਺ͱλΠϜ
    Ξ΢τ
    * Circuit Breaker Failure rate ͔Β
    error Λฦ͢
    * Mirroring ϦΫΤετͷϛϥʔϦϯά
    S e r v i c e m e s h
    Serivce ʹରͯ͠ͷεέʔϧΛର৅ͱ͠
    ·͕ͨ͠ɺMicroservices શମͰεέʔ
    ϧ͢Δ͜ͱ΋ग़དྷΔͱΑΓɺॊೈʹε
    έʔϧՄೳʹͳΓ·͢ɻ
    Istio ʹ͸ Prometheus ͕ಋೖ͞ΕΔ
    ͨΊ্ख͘ Istio Λ௨ͯ͠ Scale ग़དྷͳ
    ͍͔ݕূதͰ͢ɻ
    M i c r o s e r v i s e s શ ମ ͷ εέ ʔϧ
    Istio
    ࣍ ΁ ͷ ௅ ઓ 19
    https://istio.io

    View Slide