OUR GOALS W a n t e d l y , I n c . D i s c o v e r Ͱ ͋ ͍ C o n n e c t ͭ ͳ ͕ Γ E n g a g e ͭ ͳ ͕ Γ Λ ਂ Ί Δ S N S Ϗ δ ω ε S N S C r e a te a w o r l d w h e r e w o r k d r i v e s p a s s i o n 3
Microservices o u r p r o d u c t s Wa n te d l y Vi s i t , Wa n te d l y C h a t , Wa n te d l y P e o p l e ࡾ ͭ ͷ ϓ ϩ μΫ τ QA,Production ؚΊͯ αʔϏε 124 M i c r o s e r v i c e s GKE + k8s on AWS 80+ K u b e r n e t e s N o d e Auto Scale ίϯςφͷ Մม 3K+ C o n t a i n e r s 4
Relative vs. Absolute metrics Custom Metrics and Resource Metrics Prometheus Operator and Prometheus Custom Metrics API Server Kubernetes H o w t o u s e H o r i z o n t a l P o d A u t o s c a l e r s 5
Horizontal Pod Autoscalers “T h e H o r i z o n t a l P o d A u t o s c a l e r a u t o m a t i c a l l y s c a l e s t h e n u m b e r o f p o d s i n a r e p l i c a t i o n c o n t r o l l e r , d e p l o y m e n t o r r e p l i c a s e t b a s e d o n o b s e r v e d C P U u t i l i z a t i o n ( o r , w i t h c u s t o m m e t r i c s s u p p o r t , o n s o m e o t h e r a p p l i c a t i o n - p r o v i d e d m e t r i c s ) . N o t e t h a t H o r i z o n t a l P o d A u t o s c a l i n g d o e s n o t a p p l y t o o b j e c t s t h a t c a n ’ t b e s c a l e d , f o r e x a m p l e , D a e m o n S e t s . “ εέ ʔϧ ͞ ͤ ͨ ͍ T a r g e t Λ ࢦ ఆ Ұ ఆ ؒ ִ ( 3 0 s ) Ͱ C P U ͷ ༻ Λ Ϟχ λ Ϧ ϯ ά ར ༻ ऀ ɺ ༧ Ί r e p l i c a ͷ ͱ ᮢ ઃ ఆ T a r g e t N u m O f P o d s = c e i l ( s u m ( C u r r e n t P o d s C P U U t i l i z a t i o n ) / T a r g e t ) 6 h t t p s : // k u b e r n e t e s . i o /d o c s / t a s k s /r u n - a p p l i c a t i o n / h o r i z o n t a l - p o d - a u t o s c a l e / h t t p s : //g i t h u b . c o m / k u b e r n e t e s /c o m m u n i t y / b l o b / 1 a 6 5 6 3 2 0 9 0 6 4 5 3 7 9 d b 5 9 d 0 b 5 d 7 7c 5 0 2 d 1 7 1 a e 0 e c /c o n t r i b u t o r s /d e s i g n - p r o p o s a l s /a u t o s c a l i n g / h o r i z o n t a l - p o d - a u t o s c a l e r . m d # a u t o s c a l i n g - a l g o r i t h m
Why? ಥ વ ͷ S p i k e ʹ উ ͯ Δ ? ϝσΟΞͷಛूͳͲͷ̋̋๒ɺීஈͷԿඦഒͷΞΫηεʹͳΔɻͦͷதͰɺ͋ΔαʔϏεͩ ͚Πϯελϯε͔ΒScaleͤ͞Δ߹ɺؒҧ͍ͳͦ͘͜Ͱ٧·Δɻ ಥ વ ͷ ε ύ ΠΫ ʹ উ ͯ Δ Π ϯ ϑ ϥ Microservices Λਪ͍ͯ͠·͢ɻग़དྷΔݶΓۉҰͰඵ୯ҐͰScaleΛͰ͖ΔΑ͏ʹ͢Δඞཁɻ M i c r o s e r v i c e s Λ ͢ Δ ্ Ͱ S c a l e ͷ ՝ Λ ແ ͘ ͢ Scaleઃఆ͕खؒͰɺؾ͕ͭ͘ͱ Distributed Monolith* ʹͳΔɻ୭Ͱ Scale Ͱ͖ΔΑ͏ʹ ͢Δɻ ৽ ن α ʔ Ϗ ε ө Ͱ ͖ Δ G e n e r a l ͳ ઃ ఆ Λ 7 * https://www.simplethread.com/youre-not-actually-building-microservices/
Performance Tuning l o a d t e s t i n g ґ ଘ ؔ ͕ ͋ Δ α ʔ Ϗ ε Մ ࢹ Խ γ φ Ϧ Φ ࡞ ෛ ՙ ஔ εέ ʔϧ ͢ Δ ج ४ ఆ ٛ K u b e r n e t e s Խ H o r i z o n t a l P o d A u t o s c a l e r s μο γ ϡ Ϙ ʔ υ Ͱ ཧ Τ ϯ υ ϙ Π ϯ τ ຖ ʹ Ϙ τϧ ω ο Ϋ Λ ௵ ͢ 8
Relative vs. Absolute M e t r i c s ͷ ύ ϥ μ Π Ϝ γ ϑ τ “We chose values of the target metric to be relative (e.g. 90% of requested CPU resource) rather than absolute (e.g. 0.6 core) for the following reason. If we choose absolute metric, user will need to guarantee that the target is lower than the request. Otherwise, overloaded pods may not be able to consume more than the autoscaler's absolute target utilization, thereby preventing the autoscaler from seeing high enough utilization to trigger it to scale up. This may be especially troublesome when user changes requested resources for a pod because they would need to also change the autoscaler utilization threshold. Therefore, we decided to choose relative metric. For user, it is enough to set it to a value smaller than 100%, and further changes of requested resources will not invalidate it.” ࠓ·Ͱ͍͔ͭ͘ݻఆͷᮢϞχλϦϯάઃఆϝτϦΫεΛऔ͍ͬͯͨͷΛશͯɺׂ߹ʹมߋ͠·ͨ͠ɻ͜ΕͰ͍ ͭ͘ίϯςφ͕ಈ͍͍ͯΔ͔Ѳ͢Δඞཁͳ͘ɺશମͷ ݸ͔ΒԿ%ҡ࣋Ͱ͖͍ͯΔͷ͔Λཧ͢Δख๏ʹύϥμΠϜγϑτ͠·ͨ͠ɻ 9 https://github.com/kubernetes/community/blob/1a65632090645379db59d0b5d77c502d171ae0ec/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#relative-vs-absolute-metrics
o n K u b e r n e t e s M e t r i c s & A u t o s c a l e r s C u s t o m m e t r i c s εέ ʔϧ ౷ Ұ α ʔ Ϗ ε σΟε Χ ό Ϧ Π ϯε λ ϯε ͷ ε έ ʔϧ ٫ H o r i z o n t a l P o d A u to s c a l e r s v 1 C P U ༻ μο γ ϡ Ϙʔ υ C P U M e m o r y R e q u e s t Q u e r y g o r o u t i n e e tc . . Microservice ͷ Scale ߋ ʹ ౿ Έ ࠐ Μ ͩ ε έ ʔ ϥ Ϗ Ϧ ς Ο Λ ί ϯ ς φ ෦ · Ͱ ཧ ͕ ඞ ཁ ͩ ͬ ͨ ͷ ͕ɺ ػ ց ͕ ϝ ϯ ς φ ϯε Λ ͓ ͜ ͳ ͏ ͜ ͱ Ͱɺ Π ϯ ϑ ϥ ࣄ ଶ ͷ ೳ ྗ ͕ ্ ͕ Γ · ͠ ͨ ɻ ͞ Β ʹ εέ ʔ ϥ Ϗ Ϧ ςΟ Λ ߴ Ί Δ ͨ Ί ʹ ࣍ ͷ ε ς οϓ ͱ ͯ͠ H o r i z o n t a l P o d A u to s c a l e r s Λ ౿ Έ ࠐ ΜͰ ͑Δ Α ͏ ʹ ͠ ͨ ͍ ͱ ͍ ͏ ε ς ʔ δ ཱ ͭ ͜ ͱ ʹ ͳ Γ · ͠ ͨ ɻ S T A R T Ͳ ͜ · Ͱ ෛ ՙ ͔ ͚ ͨ Β མ ͪ Δ ? क Δ ͖ ͷ ? 10
* Heapster: a application which gathers metrics, writes them to metrics storage "sinks", and exposes the resource metrics API from in- memory storage. * Metrics Server: a lighter-weight in- memory server specifically for the resource metrics API. R E S O U R C E M E T R I C S * Prometheus Adapter. An implementation of the custom metrics API that attempts to support arbitrary metrics following a set label and naming scheme. * Google Stackdriver C U S T O M M E T R I C S Multiple Metrics ೋ ͭ ͷ M e t r i c s Λ ͬ ͨ 11 https://github.com/kubernetes/metrics/blob/dba7415d399b043dd64026e26a26aeb03dbbc220/IMPLEMENTATIONS.md
S e t u p k 8 s P r o m e t h e u s C u s t o m m e t r i c s A P I S e r v e r o p t i o n ༗ ޮ Խ M e t r i c s S e r v e r Pr o m e t h e u s Pr o m e t h e u s O p e r a to r k 8 s - p r o m e t h e u s - a d a p te r S e r v i c e M o n i to r Using k8s-prometheus-adapter C u s t o m M e t r i c s ༗ ޮ Խ G K E ͷ ߹ ɺ S t a c k D r i v e r ͕ ͑ · ͢ ͕ɺ ࠓ ճ ฐ ࣾ ڥ Ͱ ͋ Δ k u b e r n e te s / ko p s ڥ Ͱ ͷ ྫ ʹ ͳ Γ · ͢ɻ ͦ ͷ ࡍ ʹ ඞ ཁ ͳ ख ॱ ͕ େ ͖ ͘ 4 S T E P ඞ ཁ Ͱ ͢ɻ S e t A p p l i c a t i o n /m e t r i c s Ճ S e r v i c e Ճ H PA ͷ ઃ ఆ 12 Set up k8s: https://github.com/kubernetes/kops/blob/1924ce0d50ce59f839dde91c642e6753222ae14d/docs/horizontal_pod_autoscaling.md#cluster-configuration Prometheus https://github.com/luxas/kubeadm-workshop/tree/592921a9754939b6db435c99e4f9377ddd5a98df#deploying-the-prometheus-operator-for-monitoring-services-in-the-cluster Custom Metrics api Server https://github.com/luxas/kubeadm-workshop/tree/592921a9754939b6db435c99e4f9377ddd5a98df#deploying-a-custom-metrics-api-server-and-a-sample-app
X Prometheus Service Monitor namespaceSelector: ʮany: trueʯʹ ͢Δ͜ͱͰશͯͷnamespace ͕ର endpoints: ʹɺݟʹߦ͘αʔϏεͷ port ͷ name Λࢦఆ 13 https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html
Custom Metrics API Server k 8 s - p r o m e t h e u s - a d a p t e r P r o m e t h e u s ͱ k 8 s A P I ͷ ڮ ͠ ͜ ͷ ઃ ఆ Α Γ ɺ H PA Ϛ χ ϑ Σ ε τ S e r v i c e Ϛ χ ϑ Σ ε τ /m e t r i c s Ճ 14 h t t p s : //g i t h u b . c o m / D i r e c t X M a n 1 2 / k 8 s - p r o m e t h e u s - a d a p t e r/ t r e e / 8 3 a 7d d 5 d 5 e 6 9 6 9 b b f 5 3 8 0 d 4 1 c 9 c 0 e 2 b 8 3 4 3 2 7 3 3 2 # e x a m p l e h t t p s : //g i t h u b . c o m / l u x a s / k u b e a d m - w o r k s h o p / t r e e / 5 9 2 9 2 1 a 9 7 5 4 9 3 9 b 6 d b 4 3 5 c 9 9 e 4 f 9 3 7 7d d d 5 a 9 8 d f # d e p l o y i n g - a - c u s t o m - m e t r i c s - a p i - s e r v e r - a n d - a - s a m p l e - a p p
Application C u s t o m M e t r i c s Λ ར ༻ Ϋ ϥε λ ͷ η ο τΞ οϓ ͕ ऴ Θ Δ ͱ ɺ ࣮ ࡍ ʹ α ʔ Ϗ ε ʹ ө ͞ ͤ ͯ ͍ ͖ · ͢ ServiceMoniter Ͱࢦఆͨ͠ Service Λ·ͣϝτϦΫεΛड ͚औΔઐ༻ͱͯ͠࡞͠·͢ S e r v i c e ࡞ ͲͷΑ͏ʹ Custom Metrics ͱ Resource Metrics Λͬͯε έʔϧͤ͞Δ͔ΛHPA ϚχϑΣ ετΛ࡞͠·͢ H PA ࡞ /metrics ࣮ or nginx ͕લஈʹ͍Δ߹ nginx_exporter Λ sidecar ༻ҙ /m e t r i c s ࣮ 15 https://github.com/discordianfish/nginx_exporter
X Prometheus nginx_exporter n g i n x _ e x p o r t e r ɺ /m e t r i c s Λ ։ ์ ࢹ ͠ ͨ ͍ n g i n x ʹ ର ͯ͠ /n g i n x _ s t a t u s ʹ ΞΫ η ε ͠ Λ औ ಘ ͯ͠ P r o m e t h e u s ༻ ʹ ग़ ྗ P r o m e t h e u s - > n g i n x _ e x p o r t e r - > n g i n x ( n g i n x _ s t a t u s ) 16 https://github.com/discordianfish/nginx_exporter
·ͱΊ C o n c l u s i o n ෳͷϢʔεʔέʔεʹԠͯ͡εέʔϧग़དྷΔ μογϡϘʔυෛՙςετΛݩʹෛՙ͕ߴ·Δલʹ Scale ͓ͯ͘͠ͱ͍ͬͨ͜ͱͰग़དྷΔ C u s t o m M e t r i c s ͱ R e s o u r c e M e t r i c s Ͱ S c a l e ͷ ෯ Λ ࣋ ͨ ͤ Δ Prometheus Λ௨ͯ͠ɺ goroutine ͷಛఆͷΤϯυϙΠϯτ༷ʑͳܗͰ Custom Metrics ͕͑ΔΑ͏ʹͳΔɻ P r o m e t h e u s / C u s t o m M e t r i c s A P I S e r v e r / k 8 s ͷ ઃ ఆ Prometheus ͷϥΠϒϥϦΛೖΕΔ͜ͱͰɺΞϓϦέʔγϣϯʹ `/metrics` ΛՃ͢Δͱ͍ ͏ํ๏ͱ Nginx Λલஈʹஔ͍ͯ͠Δ߹ɺnginx_exporter Λ sidecar ʹೖΕΔ͜ͱͰ http ͷϦΫΤετɺίωΫγϣϯ͔Β࢝ΊΒΕ·͢ S e r v i c e / H o r i z o n t a l P o d A u t o S c a l e / n g i n x _ e x p o r t e r 18
* Distributed Tracing αʔϏεؒ௨৴ ՄࢹԽ * Trafic Monitoring RequestͷՄࢹԽ * Fault Tolerant ϦτϥΠճͱλΠϜ Ξτ * Circuit Breaker Failure rate ͔Β error Λฦ͢ * Mirroring ϦΫΤετͷϛϥʔϦϯά S e r v i c e m e s h Serivce ʹରͯ͠ͷεέʔϧΛରͱ͠ ·͕ͨ͠ɺMicroservices શମͰεέʔ ϧ͢Δ͜ͱग़དྷΔͱΑΓɺॊೈʹε έʔϧՄೳʹͳΓ·͢ɻ Istio ʹ Prometheus ͕ಋೖ͞ΕΔ ͨΊ্ख͘ Istio Λ௨ͯ͠ Scale ग़དྷͳ ͍͔ݕূதͰ͢ɻ M i c r o s e r v i s e s શ ମ ͷ εέ ʔϧ Istio ࣍ ͷ ઓ 19 https://istio.io
koudaiii
sansantech
tomoaki25
fedepaol
bengo4com
wakamatsu_takumu
h1kita
murachiakira
line_developers_tw
orgachem
lemiorhan
tyrmanuz
akkie76
dougneiner
reverentgeek
trallard
danielanewman
kastner
jlugia
addyosmani
chriscoyier
kevinliebholz
chrislema
mraible
jacobian