Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Autoscaling in Kubernetes
Atsushi Tanaka
September 09, 2018
Programming
3
840
Autoscaling in Kubernetes
Atsushi Tanaka
September 09, 2018
Tweet
Share
More Decks by Atsushi Tanaka
See All by Atsushi Tanaka
Kubernetes Cluster Migration
bgpat
4
3.4k
k8sとNginxでオートスケール / Autoscaling with k8s and Nginx
bgpat
2
830
GCPのgemにコントリビュートした話
bgpat
0
460
at_exit
bgpat
0
540
OpenCensus with Datadog APM
bgpat
0
1.4k
Distributed Tracing for Microservices
bgpat
5
2.7k
PrometheusでKubernetesの高速オートスケールを実現した話
bgpat
2
1.2k
Other Decks in Programming
See All in Programming
io22 extended What's new in app performance
veronikapj
0
340
設計の学び方:自分流のススメ
masuda220
PRO
10
7.2k
ES2022の新機能
smt7174
0
260
Imperative is dead, long live Declarative! | Appdevcon
prof18
0
110
Deep Dive Into Google Zanzibar and its Concepts for Authorization Scenarios
dschenkelman
1
140
Jetpack Composeでの画面遷移
iwata_n
0
180
Node.jsデザインパターンを読んで
mmmommm
0
2.8k
Licences open source : entre guerre de clochers et radicalité
pylapp
2
510
Oracle REST Data Service: APEX Office Hours
thatjeffsmith
0
780
I/O Extended 2022 in Android ~ Whats new in Android development tools
pluu
0
560
#JJUG_CCC 「サポート」は製品開発? - JDBCライブラリ屋さんが実践する攻めのテクニカルサポートとJavaエンジニアのキャリアについて -
cdataj
0
420
大規模プロダクトにLinterを導入し運用している話
hirokiotsuka
0
230
Featured
See All Featured
Visualization
eitanlees
125
11k
5 minutes of I Can Smell Your CMS
philhawksworth
196
18k
Side Projects
sachag
450
37k
The Brand Is Dead. Long Live the Brand.
mthomps
46
2.7k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
119
28k
Atom: Resistance is Futile
akmur
255
20k
Build your cross-platform service in a week with App Engine
jlugia
219
17k
Gamification - CAS2011
davidbonilla
75
3.9k
Fontdeck: Realign not Redesign
paulrobertlloyd
73
4.1k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
12
920
Faster Mobile Websites
deanohume
294
28k
A Tale of Four Properties
chriscoyier
149
21k
Transcript
©2018 Wantedly, Inc. Autoscaling in Kubernetes KubernetesͱΦʔτεέʔϧ ,VCFSOFUFT LT ษڧձ!ຊϚΠΫϩιϑτגࣜձࣾ
9.Sep.2018 - Atsushi Tanaka - @bgpat
©2018 Wantedly, Inc. • Atsushi Tanaka ాதಞࢤ • Infrastructure
Engineer at Wantedly, Inc. • Interested in k8s, Docker and Golang About @bgpat
©2018 Wantedly, Inc. Φʔτεέʔϧͷ
©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"
©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"
©2018 Wantedly, Inc. ෛՙʹ߹ΘͤͯࣗಈͰαʔόʔΛ֦ு͢Δ
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w
"EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w
"EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ ਫฏεέʔϧ ਨεέʔϧ
©2018 Wantedly, Inc. εϖοΫ Scale Out Scale In Scale
Down Scale Up
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w
"EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ /PEF 1PE
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. w ϝτϦΫε͕ઃఆΛ͍͑ͯͳ͍͔ࢹ w 1PEͷΛ૿͢ݮΒ͢ EFQMPZNFOUͷSFQMJDBTͷΛมߋ ࣮ࡍʹεέʔϧΛߦ͏ͷεέδϡʔϥʔ Horizontal
Pod Autoscaler
©2018 Wantedly, Inc. /PEF /PEF /PEF Kubernetesͷεέʔϧ 1PE 4FSWJDF 1PE
1PE 1PE 1PE %FQMPZNFOU 3FQMJDB4FU Service Pod ʹରͯ͠ ϦΫΤετΛϩʔυόϥϯε͢Δ Deployment Pod ͷΛ Ұఆʹอͭ
©2018 Wantedly, Inc. εέʔϧ͢Δج४ w $16༻ͷΈ BVUPTDBMJOHW w Ϧιʔεʹඥ͘ϝτϦΫε
BVUPTDBMJOHW w Ϧιʔεʹඥ͔ͳ͍ϝτϦΫε LTW Horizontal Pod Autoscaler
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. autoscaling/v2 ෳͷϝτϦΫεΛઃఆՄೳ ͲΕ͔ͻͱͭͰᮢΛ͑Δͱεέʔϧ͢Δ • Resource: CPUɾϝϞϦ ͷ༻
• Object: Kubernetes ͕ཧ͢ΔϦιʔεͷϝτϦΫε • Pods: Pod ʹؔ͢ΔϝτϦΫε • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε Horizontal Pod Autoscaler
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. API ΛͬͯϝτϦΫεΛऔಘ͢Δ • Resource Metrics API •
Custom Metrics API • External Metrics API Horizontal Pod Autoscaler
©2018 Wantedly, Inc. • Metrics Server or Heapster • Pod
Λ Node ʹஔ͢Δͱ͖ʹΘΕΔ • औΕΔϝτϦΫε CPU ͱ ϝϞϦ ͷΈ Resource Metrics API
©2018 Wantedly, Inc. • CPU, ϝϞϦ༻ྔҎ֎ͷϝτϦΫε͕औΕΔ • API αʔόʔΛ༻ҙͯ͠Ϋϥελʹొ͠ͳ͍ͱ͍͚ͳ͍ •
Prometheus • Azure • Stackdriver Custom & External Metrics API
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. Pod Λεέδϡʔϧ͢ΔͨΊͷϊʔυ͕Γͳ͘ͳΔͱ ΫϥελΛߏ͢ΔϊʔυͷΛ૿͢ ରԠ͍ͯ͠Δ CloudProvider • AWS
• Azure • GCP Cluster Autoscaler
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. • Pod ͷਨεέʔϧ • resource.limit ͱ resource.request
Λมߋ • Node ͷεέδϡʔϦϯά࣌ʹӨڹ • ᮢࣗಈઃఆ • HPAͱͷซ༻Մೳ Vertical Pod Autoscaler
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. • Pod ͷதʹϦιʔεΛࢹ͢ΔίϯςφΛཱͯͯ (sidecar) ༻ྔʹ߹Θͤͯཁٻ͢ΔϦιʔεΛ૿ݮͤ͞Δ • ૿ݮ͢Δྔϊʔυͷʹґଘ
• ΫϥελΛߏ͢Δίϯϙʔωϯτʹ༻͍ΒΕ͍ͯΔ • cluster-autoscaler • metrics-server Addon Resizer
©2018 Wantedly, Inc. ͔͜͜Βࣄྫհ
©2018 Wantedly, Inc. • kops Ͱ AWS ্ʹߏங • ϊʔυ:
40 • αʔϏε: 70ݸ • ΞϓϦέʔγϣϯͷ࣮ • Ruby on Rails • Goݴޠ • Python (ػցֶश) • C++, Rust (ը૾ॲཧ) WantedlyͷKubernetesΫϥελ
©2018 Wantedly, Inc. • HPA + Custom Metrics • Prometheus
• k8s-prometheus-adapter • Prometheus Operator • nginx-exporter • Cluster Autoscaler WantedlyͰ͍ͬͯΔΦʔτεέʔϧ
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU manage manage metrics
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. Prometheus Operator https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html
©2018 Wantedly, Inc. Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata:
name: service-monitor namespace: custom-metrics labels: service-monitor: prometheus spec: selector: matchLabels: custom-metrics: prometheus namespaceSelector: any: true endpoints: - port: http - port: https custom-metrics: prometheus ͷϥϕϧ͕͍͍ͯΔαʔϏε Λࢹ
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. Nginxͷstub_status location /basic_status { stub_status; } Active
connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ͪίωΫγϣϯΛ͏
©2018 Wantedly, Inc. Active connections: 291 server accepts handled requests
16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 nginx-exporter # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU create create metrics
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. HPA apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name:
wantedly spec: maxReplicas: 50 minReplicas: 20 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: wantedly metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName: nginx_connections_current{state="writing"} targetAverageValue: 2.5 CPU ͱ Nginx ͷ ͪίωΫγϣϯͰεέʔϧ
©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ
©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ
©2018 Wantedly, Inc. • αʔόʔͷίετ͕͑ΒΕͨ • EC2 30 → Pod
20 (EC2 7) • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ) • Ξϥʔτ͕ݮͬͨ • ᮢܾΊ͕େม • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍Λ୳͢ KubernetesͷΦʔτεέʔϧΛಋೖͯ͠
©2018 Wantedly, Inc. • KubernetesͰ͑ΔΦʔτεέʔϧͷ • εέʔϧͷΈ • ઃఆͷํ •
࣮ࡍʹͬͯΈͨ • ߏྫ • Ͳ͏ͳ͔ͬͨ ·ͱΊ
©2018 Wantedly, Inc. • Custom Metrics Λಋೖ͢Δ • CPU͚ͩͷͱ͖ΑΓޮΑ͘Φʔτεέʔϧ͕͑Δ •
ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ • ͘͢εέʔϧͯ͠ىಈ͕͍ͱεύΠΫʹউͯͳ͍ • API Server ͷΦϓγϣϯΛมߋ͢Δ • --horizontal-pod-autoscaler-sync-period • ϞχλϦϯάͰ͖ΔΈ • Wantedly Ͱ DataDog Λ͍ͬͯΔ KubernetesͷΦʔτεέʔϧಋೖͷTips