Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Autoscaling in Kubernetes

Atsushi Tanaka
September 09, 2018

Autoscaling in Kubernetes

Atsushi Tanaka

September 09, 2018
Tweet

More Decks by Atsushi Tanaka

Other Decks in Programming

Transcript

  1. ©2018 Wantedly, Inc. • Atsushi Tanaka ాதಞࢤ  • Infrastructure

    Engineer
 at Wantedly, Inc. • Interested in k8s, Docker and Golang About @bgpat
  2. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w

    "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ ਫฏεέʔϧ ਨ௚εέʔϧ
  3. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  4. ©2018 Wantedly, Inc. /PEF /PEF /PEF Kubernetesͷεέʔϧ 1PE 4FSWJDF 1PE

    1PE 1PE 1PE %FQMPZNFOU
 3FQMJDB4FU Service ͸ Pod ʹରͯ͠
 ϦΫΤετΛϩʔυόϥϯε͢Δ Deployment ͸ Pod ͷ਺Λ
 Ұఆʹอͭ
  5. ©2018 Wantedly, Inc. εέʔϧ͢Δج४ w $16࢖༻཰ͷΈ BVUPTDBMJOHW  w Ϧιʔεʹඥ෇͘ϝτϦΫε

    BVUPTDBMJOHW  w Ϧιʔεʹඥ෇͔ͳ͍ϝτϦΫε LTW Horizontal Pod Autoscaler
  6. ©2018 Wantedly, Inc. autoscaling/v2 ෳ਺ͷϝτϦΫεΛઃఆՄೳ ͲΕ͔ͻͱͭͰ΋ᮢ஋Λ௒͑Δͱεέʔϧ͢Δ • Resource: CPUɾϝϞϦ ͷ࢖༻཰

    • Object: Kubernetes ͕؅ཧ͢ΔϦιʔεͷϝτϦΫε • Pods: Pod ʹؔ͢ΔϝτϦΫε • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε Horizontal Pod Autoscaler
  7. ©2018 Wantedly, Inc. API Λ࢖ͬͯϝτϦΫεΛऔಘ͢Δ • Resource Metrics API •

    Custom Metrics API • External Metrics API Horizontal Pod Autoscaler
  8. ©2018 Wantedly, Inc. • Metrics Server or Heapster • Pod

    Λ Node ʹ഑ஔ͢Δͱ͖ʹ࢖ΘΕΔ • औΕΔϝτϦΫε͸ CPU ͱ ϝϞϦ ͷΈ Resource Metrics API
  9. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  10. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  11. ©2018 Wantedly, Inc. • Pod ͷਨ௚εέʔϧ • resource.limit ͱ resource.request

    Λมߋ • Node ΁ͷεέδϡʔϦϯά࣌ʹӨڹ • ᮢ஋͸ࣗಈઃఆ • HPAͱͷซ༻΋Մೳ Vertical Pod Autoscaler
  12. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  13. ©2018 Wantedly, Inc. • kops Ͱ AWS ্ʹߏங • ϊʔυ਺:

    ໿40୆ • αʔϏε਺: ໿70ݸ • ΞϓϦέʔγϣϯͷ࣮૷ • Ruby on Rails • Goݴޠ • Python (ػցֶश) • C++, Rust (ը૾ॲཧ) WantedlyͷKubernetesΫϥελ
  14. ©2018 Wantedly, Inc. • HPA + Custom Metrics • Prometheus

    • k8s-prometheus-adapter • Prometheus Operator • nginx-exporter • Cluster Autoscaler WantedlyͰ࢖͍ͬͯΔΦʔτεέʔϧ
  15. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU manage manage metrics
  16. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  17. ©2018 Wantedly, Inc. Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata:

    name: service-monitor namespace: custom-metrics labels: service-monitor: prometheus spec: selector: matchLabels: custom-metrics: prometheus namespaceSelector: any: true endpoints: - port: http - port: https custom-metrics: prometheus ͷϥϕϧ͕෇͍͍ͯΔαʔϏε Λ؂ࢹ
  18. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  19. ©2018 Wantedly, Inc. Nginxͷstub_status location /basic_status { stub_status; } Active

    connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ଴ͪίωΫγϣϯ਺Λ࢖͏
  20. ©2018 Wantedly, Inc. Active connections: 291 server accepts handled requests

    16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 nginx-exporter # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1
  21. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  22. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU create create metrics
  23. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  24. ©2018 Wantedly, Inc. HPA apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name:

    wantedly spec: maxReplicas: 50 minReplicas: 20 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: wantedly metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName: nginx_connections_current{state="writing"} targetAverageValue: 2.5 CPU ͱ Nginx ͷ
 ଴ͪίωΫγϣϯͰεέʔϧ
  25. ©2018 Wantedly, Inc. • αʔόʔͷίετ͕཈͑ΒΕͨ • EC2 30୆ → Pod

    20୆ (EC2 7୆෼) • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ) • Ξϥʔτ͕ݮͬͨ • ᮢ஋ܾΊ͕େม • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍஋Λ୳͢ KubernetesͷΦʔτεέʔϧΛಋೖͯ͠
  26. ©2018 Wantedly, Inc. • Custom Metrics Λಋೖ͢Δ • CPU͚ͩͷͱ͖ΑΓޮ཰Α͘Φʔτεέʔϧ͕࢖͑Δ •

    ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ • ͢͹΍͘εέʔϧͯ͠΋ىಈ͕஗͍ͱεύΠΫʹ͸উͯͳ͍ • API Server ͷΦϓγϣϯΛมߋ͢Δ • --horizontal-pod-autoscaler-sync-period • ϞχλϦϯάͰ͖Δ࢓૊Έ • Wantedly Ͱ͸ DataDog Λ࢖͍ͬͯΔ KubernetesͷΦʔτεέʔϧಋೖͷTips