Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Autoscaling in Kubernetes

C3b68d6841dcd343be9685f1cdaf8ac9?s=47 Atsushi Tanaka
September 09, 2018
Programming
3
760

Autoscaling in Kubernetes

C3b68d6841dcd343be9685f1cdaf8ac9?s=128

Atsushi Tanaka

September 09, 2018
Tweet

More Decks by Atsushi Tanaka

See All by Atsushi Tanaka
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
4
3.1k
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
2
710
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
0
390
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
0
460
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
0
1.1k
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
5
2.6k
C3b68d6841dcd343be9685f1cdaf8ac9?s=48 bgpat
2
1.1k

Other Decks in Programming

See All in Programming
57e5f749a4ee962a6cfb2197c160c577?s=48 maeda8
0
170
D63b0683b8fdd02c71c802691f97c04c?s=48 jogannaoki
0
1.2k
72a2082c6a4dd79ad68befb3db911616?s=48 mraible
PRO
2
310
Ee4cc84c4dedde009c013c6d5f1a2e84?s=48 tatchnicolas
2
220
Ad4c3c738e57a9bbcb800871e81c232d?s=48 itosho
0
100
B3b2139e4f2c0eca4efe2379fcebc1c5?s=48 afilina
PRO
3
420
E86b46be0f4e61db6b28becc5493bbab?s=48 januswel
0
1.1k
8a820699dff43e6b707a488b592e7a21?s=48 choplin
2
610
6e704ad00f166b420eba3d174713c82b?s=48 urushiyama
0
310
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
160
1bc80e2eee2adeaa8bb577798d92e9d0?s=48 antonarhipov
0
130
7602f2751868682b296171f58589c851?s=48 lyrixx
2
100

Featured

See All Featured
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
35
3.8k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
42
1.5k
A415db3ac9e9668acbc1fb36eead84ac?s=48 mthomps
44
2.5k
88bc30284c6a424b63a92aad27d0ba36?s=48 jnunemaker
PRO
43
4.9k
F383c6a4dc55e331bbe2987b622cee6b?s=48 stephaniewalter
263
11k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
318
19k
78b475797a14c84799063c7cd073962f?s=48 holman
451
140k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
6
750
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
372
43k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
78
270k
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
72
3.9k
E195ae45320d9202eaa01c9f1d31a416?s=48 marktimemedia
15
710

Transcript

  1. ©2018 Wantedly, Inc. Autoscaling in Kubernetes KubernetesͱΦʔτεέʔϧ ,VCFSOFUFT LT ษڧձ!೔ຊϚΠΫϩιϑτגࣜձࣾ

    9.Sep.2018 - Atsushi Tanaka - @bgpat

  2. ©2018 Wantedly, Inc. • Atsushi Tanaka ాதಞࢤ  • Infrastructure

    Engineer
 at Wantedly, Inc. • Interested in k8s, Docker and Golang About @bgpat

  3. ©2018 Wantedly, Inc. Φʔτεέʔϧͷ࿩

  4. ©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"

  5. ©2018 Wantedly, Inc. https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ "εέʔϧ"

  6. ©2018 Wantedly, Inc. ෛՙʹ߹ΘͤͯࣗಈͰαʔόʔΛ֦ு͢Δ

  7. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w

    "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

  8. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w

    "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ ਫฏεέʔϧ ਨ௚εέʔϧ

  9. ©2018 Wantedly, Inc. εϖοΫ ୆਺ Scale Out Scale In Scale

    Down Scale Up

  10. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w

    "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ /PEF 1PE

  11. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

  12. ©2018 Wantedly, Inc. w ϝτϦΫε͕ઃఆ஋Λ௒͍͑ͯͳ͍͔؂ࢹ w 1PEͷ਺Λ૿΍͢ݮΒ͢ EFQMPZNFOUͷSFQMJDBTͷ஋Λมߋ ࣮ࡍʹεέʔϧΛߦ͏ͷ͸εέδϡʔϥʔ Horizontal

    Pod Autoscaler

  13. ©2018 Wantedly, Inc. /PEF /PEF /PEF Kubernetesͷεέʔϧ 1PE 4FSWJDF 1PE

    1PE 1PE 1PE %FQMPZNFOU
 3FQMJDB4FU Service ͸ Pod ʹରͯ͠
 ϦΫΤετΛϩʔυόϥϯε͢Δ Deployment ͸ Pod ͷ਺Λ
 Ұఆʹอͭ

  14. ©2018 Wantedly, Inc. εέʔϧ͢Δج४ w $16࢖༻཰ͷΈ BVUPTDBMJOHW  w Ϧιʔεʹඥ෇͘ϝτϦΫε

    BVUPTDBMJOHW  w Ϧιʔεʹඥ෇͔ͳ͍ϝτϦΫε LTW Horizontal Pod Autoscaler

  15. ©2018 Wantedly, Inc. HPA autoscaling/v1

  16. ©2018 Wantedly, Inc. HPA autoscaling/v1

  17. ©2018 Wantedly, Inc. HPA autoscaling/v1

  18. ©2018 Wantedly, Inc. HPA autoscaling/v1

  19. ©2018 Wantedly, Inc. autoscaling/v2 ෳ਺ͷϝτϦΫεΛઃఆՄೳ ͲΕ͔ͻͱͭͰ΋ᮢ஋Λ௒͑Δͱεέʔϧ͢Δ • Resource: CPUɾϝϞϦ ͷ࢖༻཰

    • Object: Kubernetes ͕؅ཧ͢ΔϦιʔεͷϝτϦΫε • Pods: Pod ʹؔ͢ΔϝτϦΫε • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε Horizontal Pod Autoscaler

  20. ©2018 Wantedly, Inc. HPA autoscaling/v2

  21. ©2018 Wantedly, Inc. HPA autoscaling/v2

  22. ©2018 Wantedly, Inc. HPA autoscaling/v2

  23. ©2018 Wantedly, Inc. HPA autoscaling/v2

  24. ©2018 Wantedly, Inc. HPA autoscaling/v2

  25. ©2018 Wantedly, Inc. HPA autoscaling/v2

  26. ©2018 Wantedly, Inc. API Λ࢖ͬͯϝτϦΫεΛऔಘ͢Δ • Resource Metrics API •

    Custom Metrics API • External Metrics API Horizontal Pod Autoscaler

  27. ©2018 Wantedly, Inc. • Metrics Server or Heapster • Pod

    Λ Node ʹ഑ஔ͢Δͱ͖ʹ࢖ΘΕΔ • औΕΔϝτϦΫε͸ CPU ͱ ϝϞϦ ͷΈ Resource Metrics API

  28. ©2018 Wantedly, Inc. • CPU, ϝϞϦ࢖༻ྔҎ֎ͷϝτϦΫε͕औΕΔ • API αʔόʔΛ༻ҙͯ͠Ϋϥελʹొ࿥͠ͳ͍ͱ͍͚ͳ͍ •

    Prometheus • Azure • Stackdriver Custom & External Metrics API

  29. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

  30. ©2018 Wantedly, Inc. Pod Λεέδϡʔϧ͢ΔͨΊͷϊʔυ͕଍Γͳ͘ͳΔͱ ΫϥελΛߏ੒͢Δϊʔυͷ਺Λ૿΍͢ ରԠ͍ͯ͠Δ CloudProvider • AWS

    • Azure • GCP Cluster Autoscaler

  31. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

  32. ©2018 Wantedly, Inc. • Pod ͷਨ௚εέʔϧ • resource.limit ͱ resource.request

    Λมߋ • Node ΁ͷεέδϡʔϦϯά࣌ʹӨڹ • ᮢ஋͸ࣗಈઃఆ • HPAͱͷซ༻΋Մೳ Vertical Pod Autoscaler

  33. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ

  34. ©2018 Wantedly, Inc. • Pod ͷதʹϦιʔεΛ؂ࢹ͢ΔίϯςφΛཱͯͯ (sidecar)
 ࢖༻ྔʹ߹Θͤͯཁٻ͢ΔϦιʔεΛ૿ݮͤ͞Δ • ૿ݮ͢Δྔ͸ϊʔυͷ਺ʹґଘ

    • ΫϥελΛߏ੒͢Δίϯϙʔωϯτʹ༻͍ΒΕ͍ͯΔ • cluster-autoscaler • metrics-server Addon Resizer

  35. ©2018 Wantedly, Inc. ͔͜͜Βࣄྫ঺հ

  36. ©2018 Wantedly, Inc. • kops Ͱ AWS ্ʹߏங • ϊʔυ਺:

    ໿40୆ • αʔϏε਺: ໿70ݸ • ΞϓϦέʔγϣϯͷ࣮૷ • Ruby on Rails • Goݴޠ • Python (ػցֶश) • C++, Rust (ը૾ॲཧ) WantedlyͷKubernetesΫϥελ

  37. ©2018 Wantedly, Inc. • HPA + Custom Metrics • Prometheus

    • k8s-prometheus-adapter • Prometheus Operator • nginx-exporter • Cluster Autoscaler WantedlyͰ࢖͍ͬͯΔΦʔτεέʔϧ

  38. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU manage manage metrics

  39. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

  40. ©2018 Wantedly, Inc. Prometheus Operator https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html

  41. ©2018 Wantedly, Inc. Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata:

    name: service-monitor namespace: custom-metrics labels: service-monitor: prometheus spec: selector: matchLabels: custom-metrics: prometheus namespaceSelector: any: true endpoints: - port: http - port: https custom-metrics: prometheus ͷϥϕϧ͕෇͍͍ͯΔαʔϏε Λ؂ࢹ

  42. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

  43. ©2018 Wantedly, Inc. Nginxͷstub_status location /basic_status { stub_status; } Active

    connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ଴ͪίωΫγϣϯ਺Λ࢖͏

  44. ©2018 Wantedly, Inc. Active connections: 291 server accepts handled requests

    16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 nginx-exporter # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1

  45. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

  46. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU create create metrics

  47. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage

  48. ©2018 Wantedly, Inc. HPA apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name:

    wantedly spec: maxReplicas: 50 minReplicas: 20 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: wantedly metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName: nginx_connections_current{state="writing"} targetAverageValue: 2.5 CPU ͱ Nginx ͷ
 ଴ͪίωΫγϣϯͰεέʔϧ

  49. ©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ਺

  50. ©2018 Wantedly, Inc. ΦʔτεέʔϧͱϦΫΤετ਺

  51. ©2018 Wantedly, Inc. • αʔόʔͷίετ͕཈͑ΒΕͨ • EC2 30୆ → Pod

    20୆ (EC2 7୆෼) • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ) • Ξϥʔτ͕ݮͬͨ • ᮢ஋ܾΊ͕େม • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍஋Λ୳͢ KubernetesͷΦʔτεέʔϧΛಋೖͯ͠

  52. ©2018 Wantedly, Inc. • KubernetesͰ࢖͑ΔΦʔτεέʔϧͷ࿩ • εέʔϧͷ࢓૊Έ • ઃఆͷ࢓ํ •

    ࣮ࡍʹ࢖ͬͯΈͨ • ߏ੒ྫ • Ͳ͏ͳ͔ͬͨ ·ͱΊ

  53. ©2018 Wantedly, Inc. • Custom Metrics Λಋೖ͢Δ • CPU͚ͩͷͱ͖ΑΓޮ཰Α͘Φʔτεέʔϧ͕࢖͑Δ •

    ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ • ͢͹΍͘εέʔϧͯ͠΋ىಈ͕஗͍ͱεύΠΫʹ͸উͯͳ͍ • API Server ͷΦϓγϣϯΛมߋ͢Δ • --horizontal-pod-autoscaler-sync-period • ϞχλϦϯάͰ͖Δ࢓૊Έ • Wantedly Ͱ͸ DataDog Λ࢖͍ͬͯΔ KubernetesͷΦʔτεέʔϧಋೖͷTips