Slide 1

Slide 1 text

©2018 Wantedly, Inc. Autoscaling in Kubernetes KubernetesͱΦʔτεέʔϧ ,VCFSOFUFT LT ษڧձ!೔ຊϚΠΫϩιϑτגࣜձࣾ 9.Sep.2018 - Atsushi Tanaka - @bgpat

Slide 2

Slide 2 text

©2018 Wantedly, Inc. • Atsushi Tanaka ాதಞࢤ • Infrastructure Engineer
 at Wantedly, Inc. • Interested in k8s, Docker and Golang About @bgpat

Slide 3

Slide 3 text

©2018 Wantedly, Inc. Φʔτεέʔϧͷ࿩

Slide 4

Slide 4 text

©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"

Slide 5

Slide 5 text

©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"

Slide 6

Slide 6 text

©2018 Wantedly, Inc. ෛՙʹ߹ΘͤͯࣗಈͰαʔόʔΛ֦ு͢Δ

Slide 7

Slide 7 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

Slide 8

Slide 8 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ ਫฏεέʔϧ ਨ௚εέʔϧ

Slide 9

Slide 9 text

©2018 Wantedly, Inc. εϖοΫ ୆਺ Scale Out Scale In Scale Down Scale Up

Slide 10

Slide 10 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ /PEF 1PE

Slide 11

Slide 11 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

Slide 12

Slide 12 text

©2018 Wantedly, Inc. w ϝτϦΫε͕ઃఆ஋Λ௒͍͑ͯͳ͍͔؂ࢹ w 1PEͷ਺Λ૿΍͢ݮΒ͢ EFQMPZNFOUͷSFQMJDBTͷ஋Λมߋ ࣮ࡍʹεέʔϧΛߦ͏ͷ͸εέδϡʔϥʔ Horizontal Pod Autoscaler

Slide 13

Slide 13 text

©2018 Wantedly, Inc. /PEF /PEF /PEF Kubernetesͷεέʔϧ 1PE 4FSWJDF 1PE 1PE 1PE 1PE %FQMPZNFOU
 3FQMJDB4FU Service ͸ Pod ʹରͯ͠
 ϦΫΤετΛϩʔυόϥϯε͢Δ Deployment ͸ Pod ͷ਺Λ
 Ұఆʹอͭ

Slide 14

Slide 14 text

©2018 Wantedly, Inc. εέʔϧ͢Δج४ w $16࢖༻཰ͷΈ BVUPTDBMJOHW w Ϧιʔεʹඥ෇͘ϝτϦΫε BVUPTDBMJOHW w Ϧιʔεʹඥ෇͔ͳ͍ϝτϦΫε LTW Horizontal Pod Autoscaler

Slide 15

Slide 15 text

©2018 Wantedly, Inc. HPA autoscaling/v1

Slide 16

Slide 16 text

©2018 Wantedly, Inc. HPA autoscaling/v1

Slide 17

Slide 17 text

©2018 Wantedly, Inc. HPA autoscaling/v1

Slide 18

Slide 18 text

©2018 Wantedly, Inc. HPA autoscaling/v1

Slide 19

Slide 19 text

©2018 Wantedly, Inc. autoscaling/v2 ෳ਺ͷϝτϦΫεΛઃఆՄೳ ͲΕ͔ͻͱͭͰ΋ᮢ஋Λ௒͑Δͱεέʔϧ͢Δ • Resource: CPUɾϝϞϦ ͷ࢖༻཰ • Object: Kubernetes ͕؅ཧ͢ΔϦιʔεͷϝτϦΫε • Pods: Pod ʹؔ͢ΔϝτϦΫε • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε Horizontal Pod Autoscaler

Slide 20

Slide 20 text

©2018 Wantedly, Inc. HPA autoscaling/v2

Slide 21

Slide 21 text

©2018 Wantedly, Inc. HPA autoscaling/v2

Slide 22

Slide 22 text

©2018 Wantedly, Inc. HPA autoscaling/v2

Slide 23

Slide 23 text

©2018 Wantedly, Inc. HPA autoscaling/v2

Slide 24

Slide 24 text

©2018 Wantedly, Inc. HPA autoscaling/v2

Slide 25

Slide 25 text

©2018 Wantedly, Inc. HPA autoscaling/v2

Slide 26

Slide 26 text

©2018 Wantedly, Inc. API Λ࢖ͬͯϝτϦΫεΛऔಘ͢Δ • Resource Metrics API • Custom Metrics API • External Metrics API Horizontal Pod Autoscaler

Slide 27

Slide 27 text

©2018 Wantedly, Inc. • Metrics Server or Heapster • Pod Λ Node ʹ഑ஔ͢Δͱ͖ʹ࢖ΘΕΔ • औΕΔϝτϦΫε͸ CPU ͱ ϝϞϦ ͷΈ Resource Metrics API

Slide 28

Slide 28 text

©2018 Wantedly, Inc. • CPU, ϝϞϦ࢖༻ྔҎ֎ͷϝτϦΫε͕औΕΔ • API αʔόʔΛ༻ҙͯ͠Ϋϥελʹొ࿥͠ͳ͍ͱ͍͚ͳ͍ • Prometheus • Azure • Stackdriver Custom & External Metrics API

Slide 29

Slide 29 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

Slide 30

Slide 30 text

©2018 Wantedly, Inc. Pod Λεέδϡʔϧ͢ΔͨΊͷϊʔυ͕଍Γͳ͘ͳΔͱ ΫϥελΛߏ੒͢Δϊʔυͷ਺Λ૿΍͢ ରԠ͍ͯ͠Δ CloudProvider • AWS • Azure • GCP Cluster Autoscaler

Slide 31

Slide 31 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

Slide 32

Slide 32 text

©2018 Wantedly, Inc. • Pod ͷਨ௚εέʔϧ • resource.limit ͱ resource.request Λมߋ • Node ΁ͷεέδϡʔϦϯά࣌ʹӨڹ • ᮢ஋͸ࣗಈઃఆ • HPAͱͷซ༻΋Մೳ Vertical Pod Autoscaler

Slide 33

Slide 33 text

©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1" w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

Slide 34

Slide 34 text

©2018 Wantedly, Inc. • Pod ͷதʹϦιʔεΛ؂ࢹ͢ΔίϯςφΛཱͯͯ (sidecar)
 ࢖༻ྔʹ߹Θͤͯཁٻ͢ΔϦιʔεΛ૿ݮͤ͞Δ • ૿ݮ͢Δྔ͸ϊʔυͷ਺ʹґଘ • ΫϥελΛߏ੒͢Δίϯϙʔωϯτʹ༻͍ΒΕ͍ͯΔ • cluster-autoscaler • metrics-server Addon Resizer

Slide 35

Slide 35 text

©2018 Wantedly, Inc. ͔͜͜Βࣄྫ঺հ

Slide 36

Slide 36 text

©2018 Wantedly, Inc. • kops Ͱ AWS ্ʹߏங • ϊʔυ਺: ໿40୆ • αʔϏε਺: ໿70ݸ • ΞϓϦέʔγϣϯͷ࣮૷ • Ruby on Rails • Goݴޠ • Python (ػցֶश) • C++, Rust (ը૾ॲཧ) WantedlyͷKubernetesΫϥελ

Slide 37

Slide 37 text

©2018 Wantedly, Inc. • HPA + Custom Metrics • Prometheus • k8s-prometheus-adapter • Prometheus Operator • nginx-exporter • Cluster Autoscaler WantedlyͰ࢖͍ͬͯΔΦʔτεέʔϧ

Slide 38

Slide 38 text

©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU manage manage metrics

Slide 39

Slide 39 text

©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

Slide 40

Slide 40 text

©2018 Wantedly, Inc. Prometheus Operator https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html

Slide 41

Slide 41 text

©2018 Wantedly, Inc. Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: service-monitor namespace: custom-metrics labels: service-monitor: prometheus spec: selector: matchLabels: custom-metrics: prometheus namespaceSelector: any: true endpoints: - port: http - port: https custom-metrics: prometheus ͷϥϕϧ͕෇͍͍ͯΔαʔϏε Λ؂ࢹ

Slide 42

Slide 42 text

©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

Slide 43

Slide 43 text

©2018 Wantedly, Inc. Nginxͷstub_status location /basic_status { stub_status; } Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ଴ͪίωΫγϣϯ਺Λ࢖͏

Slide 44

Slide 44 text

©2018 Wantedly, Inc. Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 nginx-exporter # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1

Slide 45

Slide 45 text

©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

Slide 46

Slide 46 text

©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU create create metrics

Slide 47

Slide 47 text

©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

Slide 48

Slide 48 text

©2018 Wantedly, Inc. HPA apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: wantedly spec: maxReplicas: 50 minReplicas: 20 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: wantedly metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName: nginx_connections_current{state="writing"} targetAverageValue: 2.5 CPU ͱ Nginx ͷ
 ଴ͪίωΫγϣϯͰεέʔϧ

Slide 49

Slide 49 text

©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ਺

Slide 50

Slide 50 text

©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ਺

Slide 51

Slide 51 text

©2018 Wantedly, Inc. • αʔόʔͷίετ͕཈͑ΒΕͨ • EC2 30୆ → Pod 20୆ (EC2 7୆෼) • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ) • Ξϥʔτ͕ݮͬͨ • ᮢ஋ܾΊ͕େม • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍஋Λ୳͢ KubernetesͷΦʔτεέʔϧΛಋೖͯ͠

Slide 52

Slide 52 text

©2018 Wantedly, Inc. • KubernetesͰ࢖͑ΔΦʔτεέʔϧͷ࿩ • εέʔϧͷ࢓૊Έ • ઃఆͷ࢓ํ • ࣮ࡍʹ࢖ͬͯΈͨ • ߏ੒ྫ • Ͳ͏ͳ͔ͬͨ ·ͱΊ

Slide 53

Slide 53 text

©2018 Wantedly, Inc. • Custom Metrics Λಋೖ͢Δ • CPU͚ͩͷͱ͖ΑΓޮ཰Α͘Φʔτεέʔϧ͕࢖͑Δ • ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ • ͢͹΍͘εέʔϧͯ͠΋ىಈ͕஗͍ͱεύΠΫʹ͸উͯͳ͍ • API Server ͷΦϓγϣϯΛมߋ͢Δ • --horizontal-pod-autoscaler-sync-period • ϞχλϦϯάͰ͖Δ࢓૊Έ • Wantedly Ͱ͸ DataDog Λ࢖͍ͬͯΔ KubernetesͷΦʔτεέʔϧಋೖͷTips