Autoscaling in Kubernetes

Transcript

1. ©2018 Wantedly, Inc.
   Autoscaling in Kubernetes
   KubernetesͱΦʔτεέʔϧ
   ,VCFSOFUFT LT
   ษڧձ!೔ຊϚΠΫϩιϑτגࣜձࣾ
   9.Sep.2018 - Atsushi Tanaka - @bgpat
   

   View Slide

2. ©2018 Wantedly, Inc.
   • Atsushi Tanaka ాதಞࢤ
   
   • Infrastructure Engineer

   at Wantedly, Inc.
   • Interested in k8s, Docker and Golang
   About @bgpat
   

   View Slide

3. ©2018 Wantedly, Inc.
   Φʔτεέʔϧͷ࿩
   

   View Slide

4. ©2018 Wantedly, Inc.
   https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ
   "εέʔϧ"
   

   View Slide

5. ©2018 Wantedly, Inc.
   https://ja.wikipedia.org/wiki/εέʔϥϏϦςΟ
   "εέʔϧ"
   

   View Slide

6. ©2018 Wantedly, Inc.
   ෛՙʹ߹ΘͤͯࣗಈͰαʔόʔΛ֦ு͢Δ
   

   View Slide

7. ©2018 Wantedly, Inc.
   w )PSJ[POUBM1PE"VUPTDBMFS
   w $MVTUFS"VUPTDBMFS
   w 7FSUJDBM1PE"VUPTDBMFS
   w "EEPO3FTJ[FS
   KubernetesͰ࢖͑ΔΦʔτεέʔϧ
   

   View Slide

8. ©2018 Wantedly, Inc.
   w )PSJ[POUBM1PE"VUPTDBMFS
   w $MVTUFS"VUPTDBMFS
   w 7FSUJDBM1PE"VUPTDBMFS
   w "EEPO3FTJ[FS
   KubernetesͰ࢖͑ΔΦʔτεέʔϧ
   ਫฏεέʔϧ
   ਨ௚εέʔϧ
   

   View Slide

9. ©2018 Wantedly, Inc.
   εϖοΫ
   ୆਺
   Scale Out
   Scale In
   Scale Down
   Scale Up
   

   View Slide

10. ©2018 Wantedly, Inc.
    w )PSJ[POUBM1PE"VUPTDBMFS
    w $MVTUFS"VUPTDBMFS
    w 7FSUJDBM1PE"VUPTDBMFS
    w "EEPO3FTJ[FS
    KubernetesͰ࢖͑ΔΦʔτεέʔϧ
    /PEF
    1PE
    

    View Slide

11. ©2018 Wantedly, Inc.
    w )PSJ[POUBM1PE"VUPTDBMFS )1"
    
    w $MVTUFS"VUPTDBMFS
    w 7FSUJDBM1PE"VUPTDBMFS
    w "EEPO3FTJ[FS
    KubernetesͰ࢖͑ΔΦʔτεέʔϧ
    

    View Slide

12. ©2018 Wantedly, Inc.
    w ϝτϦΫε͕ઃఆ஋Λ௒͍͑ͯͳ͍͔؂ࢹ
    w 1PEͷ਺Λ૿΍͢ݮΒ͢
    EFQMPZNFOUͷSFQMJDBTͷ஋Λมߋ
    ࣮ࡍʹεέʔϧΛߦ͏ͷ͸εέδϡʔϥʔ
    Horizontal Pod Autoscaler
    

    View Slide

13. ©2018 Wantedly, Inc.
    /PEF
    /PEF
    /PEF
    Kubernetesͷεέʔϧ
    1PE
    4FSWJDF
    1PE
    1PE
    1PE
    1PE
    %FQMPZNFOU

    3FQMJDB4FU
    Service ͸ Pod ʹରͯ͠

    ϦΫΤετΛϩʔυόϥϯε͢Δ
    Deployment ͸ Pod ͷ਺Λ

    Ұఆʹอͭ
    

    View Slide

14. ©2018 Wantedly, Inc.
    εέʔϧ͢Δج४
    w $16࢖༻཰ͷΈ BVUPTDBMJOHW
    
    w Ϧιʔεʹඥ෇͘ϝτϦΫε BVUPTDBMJOHW
    
    w Ϧιʔεʹඥ෇͔ͳ͍ϝτϦΫε LTW
    
    Horizontal Pod Autoscaler
    

    View Slide

15. ©2018 Wantedly, Inc.
    HPA autoscaling/v1
    

    View Slide

16. ©2018 Wantedly, Inc.
    HPA autoscaling/v1
    

    View Slide

17. ©2018 Wantedly, Inc.
    HPA autoscaling/v1
    

    View Slide

18. ©2018 Wantedly, Inc.
    HPA autoscaling/v1
    

    View Slide

19. ©2018 Wantedly, Inc.
    autoscaling/v2
    ෳ਺ͷϝτϦΫεΛઃఆՄೳ
    ͲΕ͔ͻͱͭͰ΋ᮢ஋Λ௒͑Δͱεέʔϧ͢Δ
    • Resource: CPUɾϝϞϦ ͷ࢖༻཰
    • Object: Kubernetes ͕؅ཧ͢ΔϦιʔεͷϝτϦΫε
    • Pods: Pod ʹؔ͢ΔϝτϦΫε
    • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε
    Horizontal Pod Autoscaler
    

    View Slide

20. ©2018 Wantedly, Inc.
    HPA autoscaling/v2
    

    View Slide

21. ©2018 Wantedly, Inc.
    HPA autoscaling/v2
    

    View Slide

22. ©2018 Wantedly, Inc.
    HPA autoscaling/v2
    

    View Slide

23. ©2018 Wantedly, Inc.
    HPA autoscaling/v2
    

    View Slide

24. ©2018 Wantedly, Inc.
    HPA autoscaling/v2
    

    View Slide

25. ©2018 Wantedly, Inc.
    HPA autoscaling/v2
    

    View Slide

26. ©2018 Wantedly, Inc.
    API Λ࢖ͬͯϝτϦΫεΛऔಘ͢Δ
    • Resource Metrics API
    • Custom Metrics API
    • External Metrics API
    Horizontal Pod Autoscaler
    

    View Slide

27. ©2018 Wantedly, Inc.
    • Metrics Server or Heapster
    • Pod Λ Node ʹ഑ஔ͢Δͱ͖ʹ࢖ΘΕΔ
    • औΕΔϝτϦΫε͸ CPU ͱ ϝϞϦ ͷΈ
    Resource Metrics API
    

    View Slide

28. ©2018 Wantedly, Inc.
    • CPU, ϝϞϦ࢖༻ྔҎ֎ͷϝτϦΫε͕औΕΔ
    • API αʔόʔΛ༻ҙͯ͠Ϋϥελʹొ࿥͠ͳ͍ͱ͍͚ͳ͍
    • Prometheus
    • Azure
    • Stackdriver
    Custom & External Metrics API
    

    View Slide

29. ©2018 Wantedly, Inc.
    w )PSJ[POUBM1PE"VUPTDBMFS )1"
    
    w $MVTUFS"VUPTDBMFS
    w 7FSUJDBM1PE"VUPTDBMFS
    w "EEPO3FTJ[FS
    KubernetesͰ࢖͑ΔΦʔτεέʔϧ
    

    View Slide

30. ©2018 Wantedly, Inc.
    Pod Λεέδϡʔϧ͢ΔͨΊͷϊʔυ͕଍Γͳ͘ͳΔͱ
    ΫϥελΛߏ੒͢Δϊʔυͷ਺Λ૿΍͢
    ରԠ͍ͯ͠Δ CloudProvider
    • AWS
    • Azure
    • GCP
    Cluster Autoscaler
    

    View Slide

31. ©2018 Wantedly, Inc.
    w )PSJ[POUBM1PE"VUPTDBMFS )1"
    
    w $MVTUFS"VUPTDBMFS
    w 7FSUJDBM1PE"VUPTDBMFS
    w "EEPO3FTJ[FS
    KubernetesͰ࢖͑ΔΦʔτεέʔϧ
    

    View Slide

32. ©2018 Wantedly, Inc.
    • Pod ͷਨ௚εέʔϧ
    • resource.limit ͱ resource.request Λมߋ
    • Node ΁ͷεέδϡʔϦϯά࣌ʹӨڹ
    • ᮢ஋͸ࣗಈઃఆ
    • HPAͱͷซ༻΋Մೳ
    Vertical Pod Autoscaler
    

    View Slide

33. ©2018 Wantedly, Inc.
    w )PSJ[POUBM1PE"VUPTDBMFS )1"
    
    w $MVTUFS"VUPTDBMFS
    w 7FSUJDBM1PE"VUPTDBMFS
    w "EEPO3FTJ[FS
    KubernetesͰ࢖͑ΔΦʔτεέʔϧ
    

    View Slide

34. ©2018 Wantedly, Inc.
    • Pod ͷதʹϦιʔεΛ؂ࢹ͢ΔίϯςφΛཱͯͯ (sidecar)

    ࢖༻ྔʹ߹Θͤͯཁٻ͢ΔϦιʔεΛ૿ݮͤ͞Δ
    • ૿ݮ͢Δྔ͸ϊʔυͷ਺ʹґଘ
    • ΫϥελΛߏ੒͢Δίϯϙʔωϯτʹ༻͍ΒΕ͍ͯΔ
    • cluster-autoscaler
    • metrics-server
    Addon Resizer
    

    View Slide

35. ©2018 Wantedly, Inc.
    ͔͜͜Βࣄྫ঺հ
    

    View Slide

36. ©2018 Wantedly, Inc.
    • kops Ͱ AWS ্ʹߏங
    • ϊʔυ਺: ໿40୆
    • αʔϏε਺: ໿70ݸ
    • ΞϓϦέʔγϣϯͷ࣮૷
    • Ruby on Rails
    • Goݴޠ
    • Python (ػցֶश)
    • C++, Rust (ը૾ॲཧ)
    WantedlyͷKubernetesΫϥελ
    

    View Slide

37. ©2018 Wantedly, Inc.
    • HPA + Custom Metrics
    • Prometheus
    • k8s-prometheus-adapter
    • Prometheus Operator
    • nginx-exporter
    • Cluster Autoscaler
    WantedlyͰ࢖͍ͬͯΔΦʔτεέʔϧ
    

    View Slide

38. ©2018 Wantedly, Inc.
    1SPNFUIFVT
    1SPNFUIFVT
    1PE
    1PE
    HPA + Custom Metrics ͷߏ੒
    .BTUFS/PEF
    $VTUPN.FUSJDT
    "1*4FSWFS
    ,VCFSOFUFT
    "1*4FSWFS
    1PE
    "QQMJDBUJPO
    /HJOY
    OHJOY
    FYQPSUFS
    proxy
    )1"
    metrics
    proxy
    stub_status
    1SPNFUIFVT
    1SPNFUIFVT
    0QFSBUPS
    manage
    metrics
    %FQMPZNFOU
    scale
    3FQMJDB4FU
    manage
    manage
    metrics
    

    View Slide

39. ©2018 Wantedly, Inc.
    1SPNFUIFVT
    1SPNFUIFVT
    1PE
    1PE
    HPA + Custom Metrics ͷߏ੒
    .BTUFS/PEF
    $VTUPN.FUSJDT
    "1*4FSWFS
    ,VCFSOFUFT
    "1*4FSWFS
    1PE
    "QQMJDBUJPO
    /HJOY
    OHJOY
    FYQPSUFS
    proxy
    )1"
    metrics
    proxy
    stub_status
    1SPNFUIFVT
    1SPNFUIFVT
    0QFSBUPS
    manage
    metrics
    %FQMPZNFOU
    scale
    3FQMJDB4FU
    metrics
    manage
    manage
    

    View Slide

40. ©2018 Wantedly, Inc.
    Prometheus Operator
    https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html
    

    View Slide

41. ©2018 Wantedly, Inc.
    Prometheus Operator
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
    name: service-monitor
    namespace: custom-metrics
    labels:
    service-monitor: prometheus
    spec:
    selector:
    matchLabels:
    custom-metrics: prometheus
    namespaceSelector:
    any: true
    endpoints:
    - port: http
    - port: https
    custom-metrics: prometheus
    ͷϥϕϧ͕෇͍͍ͯΔαʔϏε
    Λ؂ࢹ
    

    View Slide

42. ©2018 Wantedly, Inc.
    1SPNFUIFVT
    1SPNFUIFVT
    1PE
    1PE
    HPA + Custom Metrics ͷߏ੒
    .BTUFS/PEF
    $VTUPN.FUSJDT
    "1*4FSWFS
    ,VCFSOFUFT
    "1*4FSWFS
    1PE
    "QQMJDBUJPO
    /HJOY
    OHJOY
    FYQPSUFS
    proxy
    )1"
    metrics
    proxy
    stub_status
    1SPNFUIFVT
    1SPNFUIFVT
    0QFSBUPS
    manage
    metrics
    %FQMPZNFOU
    scale
    3FQMJDB4FU
    metrics
    manage
    manage
    

    View Slide

43. ©2018 Wantedly, Inc.
    Nginxͷstub_status
    location /basic_status {
    stub_status;
    }
    Active connections: 291
    server accepts handled requests
    16630948 16630948 31070465
    Reading: 6 Writing: 179 Waiting: 106
    ଴ͪίωΫγϣϯ਺Λ࢖͏
    

    View Slide

44. ©2018 Wantedly, Inc.
    Active connections: 291
    server accepts handled requests
    16630948 16630948 31070465
    Reading: 6 Writing: 179 Waiting: 106
    nginx-exporter
    # HELP nginx_connections_current Number of
    connections currently processed by nginx
    # TYPE nginx_connections_current gauge
    nginx_connections_current{state="active"} 2
    nginx_connections_current{state="reading"} 0
    nginx_connections_current{state="waiting"} 1
    nginx_connections_current{state="writing"} 1
    

    View Slide

45. ©2018 Wantedly, Inc.
    1SPNFUIFVT
    1SPNFUIFVT
    1PE
    1PE
    HPA + Custom Metrics ͷߏ੒
    .BTUFS/PEF
    $VTUPN.FUSJDT
    "1*4FSWFS
    ,VCFSOFUFT
    "1*4FSWFS
    1PE
    "QQMJDBUJPO
    /HJOY
    OHJOY
    FYQPSUFS
    proxy
    )1"
    metrics
    proxy
    stub_status
    1SPNFUIFVT
    1SPNFUIFVT
    0QFSBUPS
    manage
    metrics
    %FQMPZNFOU
    scale
    3FQMJDB4FU
    metrics
    manage
    manage
    

    View Slide

46. ©2018 Wantedly, Inc.
    1SPNFUIFVT
    1SPNFUIFVT
    1PE
    1PE
    HPA + Custom Metrics ͷߏ੒
    .BTUFS/PEF
    $VTUPN.FUSJDT
    "1*4FSWFS
    ,VCFSOFUFT
    "1*4FSWFS
    1PE
    "QQMJDBUJPO
    /HJOY
    OHJOY
    FYQPSUFS
    proxy
    )1"
    metrics
    proxy
    stub_status
    1SPNFUIFVT
    1SPNFUIFVT
    0QFSBUPS
    manage
    metrics
    %FQMPZNFOU
    scale
    3FQMJDB4FU
    create
    create
    metrics
    

    View Slide

47. ©2018 Wantedly, Inc.
    1SPNFUIFVT
    1SPNFUIFVT
    1PE
    1PE
    HPA + Custom Metrics ͷߏ੒
    .BTUFS/PEF
    $VTUPN.FUSJDT
    "1*4FSWFS
    ,VCFSOFUFT
    "1*4FSWFS
    1PE
    "QQMJDBUJPO
    /HJOY
    OHJOY
    FYQPSUFS
    proxy
    )1"
    metrics
    proxy
    stub_status
    1SPNFUIFVT
    1SPNFUIFVT
    0QFSBUPS
    manage
    metrics
    %FQMPZNFOU
    scale
    3FQMJDB4FU
    metrics
    manage
    manage
    

    View Slide

48. ©2018 Wantedly, Inc.
    HPA
    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
    name: wantedly
    spec:
    maxReplicas: 50
    minReplicas: 20
    scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: wantedly
    metrics:
    - type: Resource
    resource:
    name: cpu
    targetAverageUtilization: 50
    - type: Pods
    pods:
    metricName: nginx_connections_current{state="writing"}
    targetAverageValue: 2.5
    CPU ͱ Nginx ͷ

    ଴ͪίωΫγϣϯͰεέʔϧ
    

    View Slide

49. ©2018 Wantedly, Inc.
    ΦʔτεέʔϧͱϦΫΤετ਺
    

    View Slide

50. ©2018 Wantedly, Inc.
    ΦʔτεέʔϧͱϦΫΤετ਺
    

    View Slide

51. ©2018 Wantedly, Inc.
    • αʔόʔͷίετ͕཈͑ΒΕͨ
    • EC2 30୆ → Pod 20୆ (EC2 7୆෼)
    • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ)
    • Ξϥʔτ͕ݮͬͨ
    • ᮢ஋ܾΊ͕େม
    • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍஋Λ୳͢
    KubernetesͷΦʔτεέʔϧΛಋೖͯ͠
    

    View Slide

52. ©2018 Wantedly, Inc.
    • KubernetesͰ࢖͑ΔΦʔτεέʔϧͷ࿩
    • εέʔϧͷ࢓૊Έ
    • ઃఆͷ࢓ํ
    • ࣮ࡍʹ࢖ͬͯΈͨ
    • ߏ੒ྫ
    • Ͳ͏ͳ͔ͬͨ
    ·ͱΊ
    

    View Slide

53. ©2018 Wantedly, Inc.
    • Custom Metrics Λಋೖ͢Δ
    • CPU͚ͩͷͱ͖ΑΓޮ཰Α͘Φʔτεέʔϧ͕࢖͑Δ
    • ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ
    • ͢͹΍͘εέʔϧͯ͠΋ىಈ͕஗͍ͱεύΠΫʹ͸উͯͳ͍
    • API Server ͷΦϓγϣϯΛมߋ͢Δ
    • --horizontal-pod-autoscaler-sync-period
    • ϞχλϦϯάͰ͖Δ࢓૊Έ
    • Wantedly Ͱ͸ DataDog Λ࢖͍ͬͯΔ
    KubernetesͷΦʔτεέʔϧಋೖͷTips
    

    View Slide