Cluster Autoscaler: Dynamic & Automatic Reconfiguration

Presented at Kubernetes Meetup Tokyo #6: https://k8sjp.connpass.com/event/60324/

Transcript

1. Cluster Autoscaler
   Dynamic & Automatic Reconfiguration
   @mumoshu,
   primary maintainer of kube-aws/contributor of cluster-
   autoscaler
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 1
   

   View Slide

2. Կͷ࿩
   • K8S on AWSͰΦʔτεέʔϦϯάΛ͢Δ
   • pod͡Όͳͯ͘nodeͷ΄͏
   • cluster-autoscaler(CA)Λ࢖͏
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 2
   

   View Slide

3. Cluster Autoscaler: FAQ
   • Ϋϥ΢υϓϩόΠμඪ४ͷΦʔτεέʔϦϯάͱͷҧ͍?
   • AWS AutoScaling: based on CPU/RequestCount(ALB)
   • Cluster Autoscaler: Autoscaling based on total resource
   requests
   • e.g. add node(s) when CA sees pending pods
   • Pod࡞Γ͗ͯᷓ͢ΕͨΒϊʔυ૿΍͢Α!
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 3
   

   View Slide

4. Why: I was too lazy to operate CA
   • K8SΫϥελͷӡ༻΋΍Διϑτ΢ΣΞΤϯδχΞ
   • Φʔτεέʔϧ͸ར༻͍ͨ͠
   • CAͷӡ༻͕஍ຯʹΊΜͲ͍͘͞
   I'm responsible to run several production K8S clusters on AWS.
   I'd like to enable autoscaling on them but I'm too lazy to
   operate CA. I'm basically a software who loves sketching &
   coding, not operating !
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 4
   

   View Slide

5. Why: Syncing node pool config between cloud<->CA
   • ASG on AWSͱCA on K8Sͷઃఆ͸ผ
   • Cloud Provider: Node Pool Min/Max Size
   • CA: Node Pool Min/Max Size
   • Change(s) on AWS-side requires updating/restarting CA
   • ໃ६͕͋Δͱਖ਼͘͠ಈ͔ͳ͍
   • AWSଆͰઃఆΛม͑ͨΒɺCAͷઃఆΛม͑ͯ࠶ىಈ͕ඞཁ
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 5
   

   View Slide

6. Node Pools/ϊʔυϓʔϧ
   • AutoScaling Group(AWS)
   aws autoscaling update-auto-scaling-group \
   --auto-scaling-group-name $NAME \
   --min-size $MIN \
   --max-size $MAX
   • Instance Group in GCE?
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 6
   

   View Slide

7. CA
   1.
   kubectl edit deployment cluster-autoscaler
   2.
   cluster-autoscaler --nodes $MIN:$MAX:$NAME
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 7
   

   View Slide

8. 2౓खؒ
   • ࡞ۀ΋Εͦ͠͏
   • Scripting?
   • ͜Ε͚ͩͷͨΊʹ…?
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 8
   

   View Slide

9. ASGͱCAͷઃఆ
   • Ͳ͕ͬͪਖ਼ͳͷ?
   • ASGͷ৔߹
   • CAͷmin >= ASGͷmin
   • CAͷmax <= ASGͷmax
   Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 9
   

   View Slide

10. How
    • ASGͷઃఆΛCA͕ࣗಈతʹಡΜͰ͘ΕΕ͹Α͘ͳ͍?
    Shouldn't CA automatically sync up with AWS about ASGs it is
    told to manage?
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 10
    

    View Slide

11. What
    • CAͷ৽ػೳ
    1. Dynamic Reconfiguration
    2. Node Group Automatic Discovery !
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 11
    

    View Slide

12. Dynamic Reconfiguration
    • CAͷઃఆΛConfigMapʹॻ͘
    • ConfigMapΛॻ͖׵͑ΔͱCA͕ࣗಈϦϩʔυ(--configmap)
    • PR: github.com/kubernetes/contrib/pull/2226
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 12
    

    View Slide

13. Dynamic Reconfigurationͷར༻ํ๏
    kubectl create configmap cluster-autoscaler -f ca-config.yaml
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 13
    

    View Slide

14. kind: ConfigMap
    apiVersion: v1
    metadata:
    name: cluster-autoscaler
    namespace: kube-system
    data:
    settings: |-
    {
    "nodeGroups": [
    {
    "minSize": 1,
    "maxSize": 2,
    "name": "myasg"
    }
    ]
    }
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 14
    

    View Slide

15. ./cluster-autoscaler --configmap=cluster-autoscaler ...
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 15
    

    View Slide

16. όʔδϣϯ
    • CA 0.5.2͔Β
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 16
    

    View Slide

17. Dynamic ReconfigurationͷͭΒ͍ͱ͜Ζ
    • kubectl edit deployment ca͕kubectl edit configmap
    caʹมΘ͚ͬͨͩͰ͸
    • ͔֬ʹएׯ៉ྷʹεΫϦϓτԽ͸͠΍͘͢ͳΔ͚Ͳ
    • BEFORE: εΫϦϓτ͸deploymentͷ໊લͱશઃఆɺcaί
    ϚϯυϥΠϯͷॻ͖׵͑ํΛ஌Βͳ͍ͱ͍͚ͳ͍
    • AFTER: configmapͷ໊લͱઃఆ஋͚ͩ஌͍ͬͯΕ͹͍͍
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 17
    

    View Slide

18. ͦ͜Ͱ
    Node Group Automatic Discovery
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 18
    

    View Slide

19. Automatic discovery in nutshell
    ҰݴͰ͍͏ͱɺ
    CA͕AWS APIΛ࢖ͬͯASGͷઃఆΛࣗಈಉظͯ͘͠ΕΔػೳ
    cluster-autoscaler: Re: AWS Autoscaler autodiscover ASG
    names and sizes
    github.com/kubernetes/autoscaler/pull/11
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 19
    

    View Slide

20. How it works
    ͋ͳͨ:
    • ASGʹλάΛ͚ͭΔ
    • CAʹλάΛڭ͑Δ
    CA:
    • λάͷ͍ͭͨASGΛݕ஌͢Δ
    !
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 20
    

    View Slide

21. How operating CA improved?
    BEFORE:
    cluster-autoscaler --nodes 1:10:myasg-123456
    & ASGઃఆมߋͷͨͼʹ͜ͷҾ਺ΛରԠͤ͞Δӡ༻
    AFTER:
    cluster-autoscaler --node-group-automatic-discovery asg:tag:key1=val1,key2=val2
    & ӡ༻ͷखؒͳ͠ !
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 21
    

    View Slide

22. ࠓͷͱ͜ΖAWSͷΈ
    • ΠϯλϑΣʔε͸GCE΍Azureʹ΋ରԠͰ͖ΔΑ͏ʹ੾ͬͨ
    ͭ΋Γ
    The automatic discovery feature works only with the AWS
    provider as of today. However, the common interface is there
    for you to easily add GCE/Azure supports !
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 22
    

    View Slide

23. GCEରԠΠϝʔδ
    AWS(Supported):
    --cloudprovider=aws \
    --node-group-automatic-discovery asg:tag:key1=val1,key2=val2
    GCE(Unsupported/Imaginary interface):
    --cloudprovider=gce \
    --node-group-automatic-discovery gce:label:key1=val1,key2=val2
    An instance group could be "labeled" in GCE, right?
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 23
    

    View Slide

24. GCE, AzureରԠٻΉ!
    Would anyone mind contributing supports for GCE and Azure
    to the automatic discovery feature?!
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 24
    

    View Slide

25. Supported since CA 0.6
    • Automatic Discovery͸CA 0.6͔Β
    • kube-aws v0.9.8-rc.1͔Β
    • CA 0.6͕σϑΥϧτ
    • Automatic DiscoveryΛΑ͠ͳʹઃఆͯ͘͠ΕΔ
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 25
    

    View Slide

26. kube-aws: Configuring CA
    autoscaling:
    clusterAutoscaler:
    enabled: true
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 26
    

    View Slide

27. kube-aws: Configuring CA per node pool
    worker:
    nodePools:
    # CA is disabled by default
    - name: static
    # CA is enabled only on this node pool
    - name: dynamic
    autoscaling:
    clusterAutoscaler:
    enabled: true
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 27
    

    View Slide

28. ·ͱΊ/Recap
    • CAͷӡ༻ࣗಈԽ/ָʹΦʔτεέʔϦϯά͕࢖͑Δ
    • ͍·ͷͱ͜ΖAWS޲͚ͷํ͕ॆ࣮
    • Automate a boring operational task for CA
    • Dynamic Reconfiguration: supported since CA 0.5.2
    • Automatic Discovery: supported on AWS since CA 0.6
    • kube-aws supports CA & Automatic Discovery
    Kubernetes Meetup Tokyo #6 "Cluster Autoscaler: Dynamic & Automatic Reconfiguration © 2017 mumoshu 28
    

    View Slide