HORIZONTAL POD AUTOSCALERS — Custom Metrics ͱ Resource Metrics — JAPAN CONTAINER DAYS V18.04 19 April 2018 Wantedly, Inc. | Kodai Sakabe @koudaiii 1

CREATE A WORLD WHERE WORK DRIVES PASSION. Wa n t e d l y , I n c . 2

OUR GOALS W a n t e d l y , I n c . D i s c o v e r Ͱ ͋ ͍ C o n n e c t ͭ ͳ ͕ Γ E n g a g e ͭ ͳ ͕ Γ Λ ਂ Ί Δ S N S Ϗ δ ω ε S N S C r e a te a w o r l d w h e r e w o r k d r i v e s p a s s i o n 3

Microservices o u r p r o d u c t s Wa n te d l y Vi s i t , Wa n te d l y C h a t , Wa n te d l y P e o p l e ࡾ ͭ ͷ ϓ ϩ μΫ τ QA,Production ؚΊͯ αʔϏε਺ 124 M i c r o s e r v i c e s GKE + k8s on AWS 80+ K u b e r n e t e s N o d e Auto Scale ίϯςφͷ਺ ͸Մม 3K+ C o n t a i n e r s 4

Relative vs. Absolute metrics Custom Metrics and Resource Metrics Prometheus Operator and Prometheus Custom Metrics API Server Kubernetes H o w t o u s e H o r i z o n t a l P o d A u t o s c a l e r s 5

Horizontal Pod Autoscalers “T h e H o r i z o n t a l P o d A u t o s c a l e r a u t o m a t i c a l l y s c a l e s t h e n u m b e r o f p o d s i n a r e p l i c a t i o n c o n t r o l l e r , d e p l o y m e n t o r r e p l i c a s e t b a s e d o n o b s e r v e d C P U u t i l i z a t i o n ( o r , w i t h c u s t o m m e t r i c s s u p p o r t , o n s o m e o t h e r a p p l i c a t i o n - p r o v i d e d m e t r i c s ) . N o t e t h a t H o r i z o n t a l P o d A u t o s c a l i n g d o e s n o t a p p l y t o o b j e c t s t h a t c a n ’ t b e s c a l e d , f o r e x a m p l e , D a e m o n S e t s . “ εέ ʔϧ ͞ ͤ ͨ ͍ T a r g e t Λ ࢦ ఆ Ұ ఆ ؒ ִ ( 3 0 s ) Ͱ C P U ͷ ࢖ ༻ ཰ Λ Ϟχ λ Ϧ ϯ ά ར ༻ ऀ ͸ ɺ ༧ Ί r e p l i c a ͷ ਺ ͱ ᮢ ஋ ઃ ఆ T a r g e t N u m O f P o d s = c e i l ( s u m ( C u r r e n t P o d s C P U U t i l i z a t i o n ) / T a r g e t ) 6 h t t p s : // k u b e r n e t e s . i o /d o c s / t a s k s /r u n - a p p l i c a t i o n / h o r i z o n t a l - p o d - a u t o s c a l e / h t t p s : //g i t h u b . c o m / k u b e r n e t e s /c o m m u n i t y / b l o b / 1 a 6 5 6 3 2 0 9 0 6 4 5 3 7 9 d b 5 9 d 0 b 5 d 7 7c 5 0 2 d 1 7 1 a e 0 e c /c o n t r i b u t o r s /d e s i g n - p r o p o s a l s /a u t o s c a l i n g / h o r i z o n t a l - p o d - a u t o s c a l e r . m d # a u t o s c a l i n g - a l g o r i t h m

Why? ಥ વ ͷ S p i k e ʹ উ ͯ Δ ? ϝσΟΞͷಛूͳͲͷ̋̋๒͸ɺීஈͷԿඦഒͷΞΫηεʹͳΔɻͦͷதͰɺ͋ΔαʔϏεͩ ͚Πϯελϯε͔ΒScaleͤ͞Δ৔߹ɺؒҧ͍ͳͦ͘͜Ͱ٧·Δɻ ಥ વ ͷ ε ύ ΠΫ ʹ ΋ উ ͯ Δ Π ϯ ϑ ϥ Microservices Λਪ঑͍ͯ͠·͢ɻग़དྷΔݶΓۉҰͰඵ୯ҐͰScaleΛͰ͖ΔΑ͏ʹ͢Δඞཁɻ M i c r o s e r v i c e s Λ ͢ Δ ্ Ͱ S c a l e ͷ ՝ ୊ Λ ແ ͘ ͢ Scaleઃఆ͕खؒͰɺؾ͕ͭ͘ͱ Distributed Monolith* ʹͳΔɻ୭Ͱ΋ Scale Ͱ͖ΔΑ͏ʹ ͢Δɻ ৽ ن α ʔ Ϗ ε ΋ ൓ ө Ͱ ͖ Δ G e n e r a l ͳ ઃ ఆ Λ 7 * https://www.simplethread.com/youre-not-actually-building-microservices/

Performance Tuning l o a d t e s t i n g ґ ଘ ؔ ܎ ͕ ͋ Δ α ʔ Ϗ ε Մ ࢹ Խ γ φ Ϧ Φ ࡞ ੒ ෛ ՙ ૷ ஔ εέ ʔϧ ͢ Δ ج ४ ఆ ٛ K u b e r n e t e s Խ H o r i z o n t a l P o d A u t o s c a l e r s μο γ ϡ Ϙ ʔ υ Ͱ ؅ ཧ Τ ϯ υ ϙ Π ϯ τ ຖ ʹ Ϙ τϧ ω ο Ϋ Λ ௵ ͢ 8

Relative vs. Absolute M e t r i c s ͷ ύ ϥ μ Π Ϝ γ ϑ τ “We chose values of the target metric to be relative (e.g. 90% of requested CPU resource) rather than absolute (e.g. 0.6 core) for the following reason. If we choose absolute metric, user will need to guarantee that the target is lower than the request. Otherwise, overloaded pods may not be able to consume more than the autoscaler's absolute target utilization, thereby preventing the autoscaler from seeing high enough utilization to trigger it to scale up. This may be especially troublesome when user changes requested resources for a pod because they would need to also change the autoscaler utilization threshold. Therefore, we decided to choose relative metric. For user, it is enough to set it to a value smaller than 100%, and further changes of requested resources will not invalidate it.” ࠓ·Ͱ͍͔ͭ͘ݻఆͷᮢ஋ϞχλϦϯάઃఆ΍ϝτϦΫεΛऔ͍ͬͯͨͷΛશͯɺׂ߹ʹมߋ͠·ͨ͠ɻ͜ΕͰ͍ ͭ͘ίϯςφ͕ಈ͍͍ͯΔ͔೺Ѳ͢Δඞཁ͸ͳ͘ɺશମͷ ݸ਺͔ΒԿ%ҡ࣋Ͱ͖͍ͯΔͷ͔Λ؅ཧ͢Δख๏ʹύϥμΠϜγϑτ͠·ͨ͠ɻ 9 https://github.com/kubernetes/community/blob/1a65632090645379db59d0b5d77c502d171ae0ec/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#relative-vs-absolute-metrics

o n K u b e r n e t e s M e t r i c s & A u t o s c a l e r s C u s t o m m e t r i c s εέ ʔϧ ౷ Ұ α ʔ Ϗ ε σΟε Χ ό Ϧ Π ϯε λ ϯε ͷ ε έ ʔϧ ୤ ٫ H o r i z o n t a l P o d A u to s c a l e r s v 1 C P U ࢖ ༻ ཰ μο γ ϡ Ϙʔ υ C P U M e m o r y R e q u e s t Q u e r y g o r o u t i n e e tc . . Microservice ͷ Scale ߋ ʹ ౿ Έ ࠐ Μ ͩ ε έ ʔ ϥ Ϗ Ϧ ς Ο Λ ί ϯ ς φ ෦ ෼ · Ͱ ؅ ཧ ͕ ඞ ཁ ͩ ͬ ͨ ͷ ͕ɺ ػ ց ͕ ϝ ϯ ς φ ϯε Λ ͓ ͜ ͳ ͏ ͜ ͱ Ͱɺ Π ϯ ϑ ϥ ࣄ ଶ ͷ ೳ ྗ ͕ ্ ͕ Γ · ͠ ͨ ɻ ͞ Β ʹ εέ ʔ ϥ Ϗ Ϧ ςΟ Λ ߴ Ί Δ ͨ Ί ʹ ࣍ ΁ ͷ ε ς οϓ ͱ ͯ͠ H o r i z o n t a l P o d A u to s c a l e r s Λ ౿ Έ ࠐ ΜͰ ࢖ ͑Δ Α ͏ ʹ ͠ ͨ ͍ ͱ ͍ ͏ ε ς ʔ δ ΁ ཱ ͭ ͜ ͱ ʹ ͳ Γ · ͠ ͨ ɻ S T A R T Ͳ ͜ · Ͱ ෛ ՙ ͔ ͚ ͨ Β མ ͪ Δ ? क Δ ΂ ͖ ΋ ͷ ͸ ? 10

* Heapster: a application which gathers metrics, writes them to metrics storage "sinks", and exposes the resource metrics API from in- memory storage. * Metrics Server: a lighter-weight in- memory server specifically for the resource metrics API. R E S O U R C E M E T R I C S * Prometheus Adapter. An implementation of the custom metrics API that attempts to support arbitrary metrics following a set label and naming scheme. * Google Stackdriver C U S T O M M E T R I C S Multiple Metrics ೋ ͭ ͷ M e t r i c s Λ ࢖ ͬ ͨ 11 https://github.com/kubernetes/metrics/blob/dba7415d399b043dd64026e26a26aeb03dbbc220/IMPLEMENTATIONS.md

S e t u p k 8 s P r o m e t h e u s C u s t o m m e t r i c s A P I S e r v e r o p t i o n ༗ ޮ Խ M e t r i c s S e r v e r Pr o m e t h e u s Pr o m e t h e u s O p e r a to r k 8 s - p r o m e t h e u s - a d a p te r S e r v i c e M o n i to r Using k8s-prometheus-adapter C u s t o m M e t r i c s ༗ ޮ Խ G K E ͷ ৔ ߹ ͸ ɺ S t a c k D r i v e r ͕ ࢖ ͑ · ͢ ͕ɺ ࠓ ճ ͸ ฐ ࣾ ؀ ڥ Ͱ ͋ Δ k u b e r n e te s / ko p s ؀ ڥ Ͱ ͷ ྫ ʹ ͳ Γ · ͢ɻ ͦ ͷ ࡍ ʹ ඞ ཁ ͳ ख ॱ ͕ େ ͖ ͘ 4 S T E P ඞ ཁ Ͱ ͢ɻ S e t A p p l i c a t i o n /m e t r i c s ௥ Ճ S e r v i c e ௥ Ճ H PA ͷ ઃ ఆ 12 Set up k8s: https://github.com/kubernetes/kops/blob/1924ce0d50ce59f839dde91c642e6753222ae14d/docs/horizontal_pod_autoscaling.md#cluster-configuration Prometheus https://github.com/luxas/kubeadm-workshop/tree/592921a9754939b6db435c99e4f9377ddd5a98df#deploying-the-prometheus-operator-for-monitoring-services-in-the-cluster Custom Metrics api Server https://github.com/luxas/kubeadm-workshop/tree/592921a9754939b6db435c99e4f9377ddd5a98df#deploying-a-custom-metrics-api-server-and-a-sample-app

X Prometheus Service Monitor namespaceSelector: ʮany: trueʯʹ ͢Δ͜ͱͰશͯͷnamespace ͕ର৅ endpoints: ʹ͸ɺݟʹߦ͘αʔϏεͷ port ͷ name Λࢦఆ 13 https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html

Custom Metrics API Server k 8 s - p r o m e t h e u s - a d a p t e r P r o m e t h e u s ͱ k 8 s A P I ͷ ڮ ౉ ͠ ͜ ͷ ઃ ఆ Α Γ ɺ H PA Ϛ χ ϑ Σ ε τ S e r v i c e Ϛ χ ϑ Σ ε τ /m e t r i c s ௥ Ճ 14 h t t p s : //g i t h u b . c o m / D i r e c t X M a n 1 2 / k 8 s - p r o m e t h e u s - a d a p t e r/ t r e e / 8 3 a 7d d 5 d 5 e 6 9 6 9 b b f 5 3 8 0 d 4 1 c 9 c 0 e 2 b 8 3 4 3 2 7 3 3 2 # e x a m p l e h t t p s : //g i t h u b . c o m / l u x a s / k u b e a d m - w o r k s h o p / t r e e / 5 9 2 9 2 1 a 9 7 5 4 9 3 9 b 6 d b 4 3 5 c 9 9 e 4 f 9 3 7 7d d d 5 a 9 8 d f # d e p l o y i n g - a - c u s t o m - m e t r i c s - a p i - s e r v e r - a n d - a - s a m p l e - a p p

Application C u s t o m M e t r i c s Λ ར ༻ Ϋ ϥε λ ͷ η ο τΞ οϓ ͕ ऴ Θ Δ ͱ ɺ ࣮ ࡍ ʹ α ʔ Ϗ ε ʹ ൓ ө ͞ ͤ ͯ ͍ ͖ · ͢ ServiceMoniter Ͱࢦఆͨ͠ Service Λ·ͣ͸ϝτϦΫεΛड ͚औΔઐ༻ͱͯ͠࡞੒͠·͢ S e r v i c e ࡞ ੒ ͲͷΑ͏ʹ Custom Metrics ͱ Resource Metrics Λ࢖ͬͯε έʔϧͤ͞Δ͔ΛHPA ϚχϑΣ ετΛ࡞੒͠·͢ H PA ࡞ ੒ /metrics ࣮૷ or nginx ͕લஈʹ͍Δ৔߹ nginx_exporter Λ sidecar ༻ҙ /m e t r i c s ࣮ ૷ 15 https://github.com/discordianfish/nginx_exporter

X Prometheus nginx_exporter n g i n x _ e x p o r t e r ͸ ɺ /m e t r i c s Λ ։ ์ ؂ ࢹ ͠ ͨ ͍ n g i n x ʹ ର ͯ͠ /n g i n x _ s t a t u s ʹ ΞΫ η ε ͠ ஋ Λ औ ಘ ͯ͠ P r o m e t h e u s ༻ ʹ ग़ ྗ P r o m e t h e u s - > n g i n x _ e x p o r t e r - > n g i n x ( n g i n x _ s t a t u s ) 16 https://github.com/discordianfish/nginx_exporter

X Monitoring η ο τ Ξ ο ϓ ޙ ࢼ ݧ $ hey -n 20000 -c 8 ϐʔΫʹୡ͢Δલʹεέʔϧ͢Δ 17

·ͱΊ C o n c l u s i o n ෳ਺ͷϢʔεʔέʔεʹԠͯ͡εέʔϧग़དྷΔ μογϡϘʔυ΍ෛՙςετΛݩʹෛՙ͕ߴ·Δલʹ Scale ͓ͯ͘͠ͱ͍ͬͨ͜ͱ΋Ͱग़དྷΔ C u s t o m M e t r i c s ͱ R e s o u r c e M e t r i c s Ͱ S c a l e ͷ ෯ Λ ࣋ ͨ ͤ Δ Prometheus Λ௨ͯ͠ɺ goroutine ͷ਺΍ಛఆͷΤϯυϙΠϯτ౳༷ʑͳܗͰ Custom Metrics ͕࢖͑ΔΑ͏ʹͳΔɻ P r o m e t h e u s / C u s t o m M e t r i c s A P I S e r v e r / k 8 s ͷ ઃ ఆ Prometheus ͷϥΠϒϥϦΛೖΕΔ͜ͱͰɺΞϓϦέʔγϣϯʹ `/metrics` Λ௥Ճ͢Δͱ͍ ͏ํ๏ͱ Nginx Λલஈʹ഑ஔ͍ͯ͠Δ৔߹͸ɺnginx_exporter Λ sidecar ʹೖΕΔ͜ͱͰ http ͷϦΫΤετɺίωΫγϣϯ਺͔Β࢝ΊΒΕ·͢ S e r v i c e / H o r i z o n t a l P o d A u t o S c a l e / n g i n x _ e x p o r t e r 18

* Distributed Tracing αʔϏεؒ௨৴ ՄࢹԽ * Trafic Monitoring RequestͷՄࢹԽ * Fault Tolerant ϦτϥΠճ਺ͱλΠϜ Ξ΢τ * Circuit Breaker Failure rate ͔Β error Λฦ͢ * Mirroring ϦΫΤετͷϛϥʔϦϯά S e r v i c e m e s h Serivce ʹରͯ͠ͷεέʔϧΛର৅ͱ͠ ·͕ͨ͠ɺMicroservices શମͰεέʔ ϧ͢Δ͜ͱ΋ग़དྷΔͱΑΓɺॊೈʹε έʔϧՄೳʹͳΓ·͢ɻ Istio ʹ͸ Prometheus ͕ಋೖ͞ΕΔ ͨΊ্ख͘ Istio Λ௨ͯ͠ Scale ग़དྷͳ ͍͔ݕূதͰ͢ɻ M i c r o s e r v i s e s શ ମ ͷ εέ ʔϧ Istio ࣍ ΁ ͷ ௅ ઓ 19 https://istio.io