Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Autoscaling in Kubernetes
Atsushi Tanaka
September 09, 2018
Programming
3
910
Autoscaling in Kubernetes
Atsushi Tanaka
September 09, 2018
Tweet
Share
More Decks by Atsushi Tanaka
See All by Atsushi Tanaka
Terraform と Kubernetes の共存による IaC の実践
bgpat
0
300
Kubernetes Cluster Migration
bgpat
4
3.7k
k8sとNginxでオートスケール / Autoscaling with k8s and Nginx
bgpat
2
930
GCPのgemにコントリビュートした話
bgpat
0
520
at_exit
bgpat
0
580
OpenCensus with Datadog APM
bgpat
0
1.5k
Distributed Tracing for Microservices
bgpat
5
2.9k
PrometheusでKubernetesの高速オートスケールを実現した話
bgpat
2
1.3k
Other Decks in Programming
See All in Programming
AWSとCPUのムフフな関係
cmdemura
0
470
10年以上続くプロダクトの フロントエンド刷新プロジェクトのふりかえり
yotahada3
2
340
An Advanced Introduction to R
nicetak
0
1.8k
TSX First な Zero-Runtime SSG potato4d/dodai とその仕組み / owned static site generator #kyotojs
potato4d
0
320
新卒でサービス立ち上げから Hasuraを使って3年経った振り返り
yutorin
0
230
Enumを自動で網羅的にテストしてみた
estie
0
1.3k
Unity+C#で学ぶ! メモリレイアウトとvtableのすゝめ 〜動的ポリモーフィズムを実現する仕組み〜
rossam
1
210
Amazon QuickSightのアップデート -re:Invent 2022の復習&2022年ハイライト-
shogo452
0
230
CDKでValidationする本当の方法 / cdk-validation
gotok365
1
210
ポケモンで学ぶiOS 16弾丸ツアー 🚅
giginet
PRO
1
610
フロントエンドで学んだことをデータ分析で使ってみた話
daichi_igarashi
0
180
Git Rebase
bkuhlmann
10
1.2k
Featured
See All Featured
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
22
1.7k
KATA
mclloyd
12
9.7k
4 Signs Your Business is Dying
shpigford
171
20k
Docker and Python
trallard
30
1.9k
Debugging Ruby Performance
tmm1
67
11k
YesSQL, Process and Tooling at Scale
rocio
159
12k
A Modern Web Designer's Workflow
chriscoyier
689
180k
Writing Fast Ruby
sferik
613
58k
Why Our Code Smells
bkeepers
PRO
326
55k
The Invisible Side of Design
smashingmag
292
48k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
1.2k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
152
13k
Transcript
©2018 Wantedly, Inc. Autoscaling in Kubernetes KubernetesͱΦʔτεέʔϧ ,VCFSOFUFT LT ษڧձ!ຊϚΠΫϩιϑτגࣜձࣾ
9.Sep.2018 - Atsushi Tanaka - @bgpat
©2018 Wantedly, Inc. • Atsushi Tanaka ాதಞࢤ • Infrastructure
Engineer at Wantedly, Inc. • Interested in k8s, Docker and Golang About @bgpat
©2018 Wantedly, Inc. Φʔτεέʔϧͷ
©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"
©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"
©2018 Wantedly, Inc. ෛՙʹ߹ΘͤͯࣗಈͰαʔόʔΛ֦ு͢Δ
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w
"EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w
"EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ ਫฏεέʔϧ ਨεέʔϧ
©2018 Wantedly, Inc. εϖοΫ Scale Out Scale In Scale
Down Scale Up
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w
"EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ /PEF 1PE
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. w ϝτϦΫε͕ઃఆΛ͍͑ͯͳ͍͔ࢹ w 1PEͷΛ૿͢ݮΒ͢ EFQMPZNFOUͷSFQMJDBTͷΛมߋ ࣮ࡍʹεέʔϧΛߦ͏ͷεέδϡʔϥʔ Horizontal
Pod Autoscaler
©2018 Wantedly, Inc. /PEF /PEF /PEF Kubernetesͷεέʔϧ 1PE 4FSWJDF 1PE
1PE 1PE 1PE %FQMPZNFOU 3FQMJDB4FU Service Pod ʹରͯ͠ ϦΫΤετΛϩʔυόϥϯε͢Δ Deployment Pod ͷΛ Ұఆʹอͭ
©2018 Wantedly, Inc. εέʔϧ͢Δج४ w $16༻ͷΈ BVUPTDBMJOHW w Ϧιʔεʹඥ͘ϝτϦΫε
BVUPTDBMJOHW w Ϧιʔεʹඥ͔ͳ͍ϝτϦΫε LTW Horizontal Pod Autoscaler
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. HPA autoscaling/v1
©2018 Wantedly, Inc. autoscaling/v2 ෳͷϝτϦΫεΛઃఆՄೳ ͲΕ͔ͻͱͭͰᮢΛ͑Δͱεέʔϧ͢Δ • Resource: CPUɾϝϞϦ ͷ༻
• Object: Kubernetes ͕ཧ͢ΔϦιʔεͷϝτϦΫε • Pods: Pod ʹؔ͢ΔϝτϦΫε • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε Horizontal Pod Autoscaler
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. HPA autoscaling/v2
©2018 Wantedly, Inc. API ΛͬͯϝτϦΫεΛऔಘ͢Δ • Resource Metrics API •
Custom Metrics API • External Metrics API Horizontal Pod Autoscaler
©2018 Wantedly, Inc. • Metrics Server or Heapster • Pod
Λ Node ʹஔ͢Δͱ͖ʹΘΕΔ • औΕΔϝτϦΫε CPU ͱ ϝϞϦ ͷΈ Resource Metrics API
©2018 Wantedly, Inc. • CPU, ϝϞϦ༻ྔҎ֎ͷϝτϦΫε͕औΕΔ • API αʔόʔΛ༻ҙͯ͠Ϋϥελʹొ͠ͳ͍ͱ͍͚ͳ͍ •
Prometheus • Azure • Stackdriver Custom & External Metrics API
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. Pod Λεέδϡʔϧ͢ΔͨΊͷϊʔυ͕Γͳ͘ͳΔͱ ΫϥελΛߏ͢ΔϊʔυͷΛ૿͢ ରԠ͍ͯ͠Δ CloudProvider • AWS
• Azure • GCP Cluster Autoscaler
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. • Pod ͷਨεέʔϧ • resource.limit ͱ resource.request
Λมߋ • Node ͷεέδϡʔϦϯά࣌ʹӨڹ • ᮢࣗಈઃఆ • HPAͱͷซ༻Մೳ Vertical Pod Autoscaler
©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w
7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ͑ΔΦʔτεέʔϧ
©2018 Wantedly, Inc. • Pod ͷதʹϦιʔεΛࢹ͢ΔίϯςφΛཱͯͯ (sidecar) ༻ྔʹ߹Θͤͯཁٻ͢ΔϦιʔεΛ૿ݮͤ͞Δ • ૿ݮ͢Δྔϊʔυͷʹґଘ
• ΫϥελΛߏ͢Δίϯϙʔωϯτʹ༻͍ΒΕ͍ͯΔ • cluster-autoscaler • metrics-server Addon Resizer
©2018 Wantedly, Inc. ͔͜͜Βࣄྫհ
©2018 Wantedly, Inc. • kops Ͱ AWS ্ʹߏங • ϊʔυ:
40 • αʔϏε: 70ݸ • ΞϓϦέʔγϣϯͷ࣮ • Ruby on Rails • Goݴޠ • Python (ػցֶश) • C++, Rust (ը૾ॲཧ) WantedlyͷKubernetesΫϥελ
©2018 Wantedly, Inc. • HPA + Custom Metrics • Prometheus
• k8s-prometheus-adapter • Prometheus Operator • nginx-exporter • Cluster Autoscaler WantedlyͰ͍ͬͯΔΦʔτεέʔϧ
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU manage manage metrics
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. Prometheus Operator https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html
©2018 Wantedly, Inc. Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata:
name: service-monitor namespace: custom-metrics labels: service-monitor: prometheus spec: selector: matchLabels: custom-metrics: prometheus namespaceSelector: any: true endpoints: - port: http - port: https custom-metrics: prometheus ͷϥϕϧ͕͍͍ͯΔαʔϏε Λࢹ
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. Nginxͷstub_status location /basic_status { stub_status; } Active
connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ͪίωΫγϣϯΛ͏
©2018 Wantedly, Inc. Active connections: 291 server accepts handled requests
16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 nginx-exporter # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU create create metrics
©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom
Metrics ͷߏ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
©2018 Wantedly, Inc. HPA apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name:
wantedly spec: maxReplicas: 50 minReplicas: 20 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: wantedly metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName: nginx_connections_current{state="writing"} targetAverageValue: 2.5 CPU ͱ Nginx ͷ ͪίωΫγϣϯͰεέʔϧ
©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ
©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ
©2018 Wantedly, Inc. • αʔόʔͷίετ͕͑ΒΕͨ • EC2 30 → Pod
20 (EC2 7) • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ) • Ξϥʔτ͕ݮͬͨ • ᮢܾΊ͕େม • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍Λ୳͢ KubernetesͷΦʔτεέʔϧΛಋೖͯ͠
©2018 Wantedly, Inc. • KubernetesͰ͑ΔΦʔτεέʔϧͷ • εέʔϧͷΈ • ઃఆͷํ •
࣮ࡍʹͬͯΈͨ • ߏྫ • Ͳ͏ͳ͔ͬͨ ·ͱΊ
©2018 Wantedly, Inc. • Custom Metrics Λಋೖ͢Δ • CPU͚ͩͷͱ͖ΑΓޮΑ͘Φʔτεέʔϧ͕͑Δ •
ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ • ͘͢εέʔϧͯ͠ىಈ͕͍ͱεύΠΫʹউͯͳ͍ • API Server ͷΦϓγϣϯΛมߋ͢Δ • --horizontal-pod-autoscaler-sync-period • ϞχλϦϯάͰ͖ΔΈ • Wantedly Ͱ DataDog Λ͍ͬͯΔ KubernetesͷΦʔτεέʔϧಋೖͷTips