Auto Scaling Kubernetes Clusters on OpenStack

Auto Scaling Kubernetes Clusters on OpenStack

One of the key features of Kubernetes is to make up and down scaling of your application workloads as easy as possible. In this talk you will learn how to setup and use the horizontal pod autoscaler to dynamically up and down scale pods in complex setups. We are also excited to show you how we built a node autoscaler for Kubernetes on OpenStack, which allows to add worker nodes to running clusters, before the horizontal pod autoscaler runs out of resources. We will also give you an overview of the current state of the vertical autoscaler in Kubernetes. When configured, it will set the resource requests for CPU and memory automatically based on usage and thus allow proper scheduling onto nodes so that the appropriate resource amount is available for each pod.

Ded87c77266697ee6981c2277bb97633?s=128

Bastian Hofmann

November 13, 2018
Tweet

Transcript

  1. @BastianHofmann Auto Scaling Kubernetes Clusters On OpenStack Bastian Hofmann Simon

    Pearce
  2. None
  3. Container orchestration platform

  4. Easy Scaling of applications

  5. Why do we want to scale?

  6. Handle increased load

  7. Only pay what you actually need

  8. Safe the environment

  9. How can applications be scaled?

  10. Horizontal Scaling

  11. Increase or decrease the amount of instances and nodes

  12. Vertical Scaling

  13. Increase or decrease CPU and memory usage/capacity of one instance

    or node
  14. Bare metal approach

  15. Order a new server

  16. Put the server into the datacenter

  17. Install the Operating System

  18. Provision the server with necessary dependencies

  19. Deploy services to the new server

  20. Reconfigure the load balancer

  21. A lot of steps

  22. This is way too slow

  23. Cloud provider approach

  24. Create a new VM from an OS image

  25. Provision the server with necessary dependencies

  26. Deploy services to the new server

  27. Reconfigure the load balancer

  28. Still a lot of steps

  29. AutoScaling Groups

  30. Proprietary APIs for every Cloud Provider

  31. Kubernetes makes this easier

  32. Standardized APIs

  33. How does Kubernetes work?

  34. • A container runs a docker image. • Only 1

    process can run inside of a container Container
  35. • A group of 1 or more containers • Shared

    network • Shared storage volumes Pod
  36. php-fpm Nginx Filebeat

  37. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: template: spec:

    containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
  38. Horizontal Scaling

  39. • Defines and manages how many instances of a pod

    should run Replica Set
  40. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: replicas: 3

    template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
  41. Vertical Scaling

  42. Container

  43. CPU and Memory requests and limits

  44. kind: Deployment ... containers: - name: hello-world image: nginxdemos/hello:0.2 resources:

    requests: cpu: 100m memory: 256Mi limits: cpu: 100m memory: 256Mi ...
  45. "Requests" are used by Kubernetes for scheduling pods on nodes

  46. "Limits" limit the container to not use more CPU and

    memory
  47. You can change these values manually

  48. And automatically

  49. Don't get up at night

  50. Focus on what is important

  51. Let's show this live

  52. Preparations for the demos

  53. We need a cluster

  54. Do-it-yourself vs. Managed Kubernetes

  55. Setting up and maintaining Kubernetes is hard

  56. Managed Kubernetes

  57. Google GKE

  58. None
  59. SysEleven MetaKube

  60. Easy upgrades

  61. Easy scaling

  62. Load Balancing

  63. Distributed Persistent Storage

  64. Backups

  65. Premium support

  66. Monitoring

  67. You can focus on what is important

  68. None
  69. None
  70. None
  71. None
  72. What if a we need to scale a pod?

  73. Manual horizontal scaling

  74. Demo

  75. Create a Deployment

  76. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: replicas: 1

    template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
  77. $ kubectl apply -f deployment.yaml

  78. $ kubectl get pods NAME READY STATUS RESTARTS AGE hello-world-fc5fd8f57-dmfjt

    1/1 Running 0 26h
  79. Create a LoadBalancer

  80. $ kubectl expose deployment hello-world--name=hello-world- svc --port=80 --target-port=80 --type=LoadBalancer

  81. $ kubectl get service hello-world NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)

    AGE hello-world-svc LoadBalancer 10.10.10.102 195.192.xxx.xxx 80:31750/TCP 37s
  82. Scale the deployment

  83. $ kubectl scale deployment/hello-world --replicas 15

  84. $ kubectl get pods NAME READY STATUS RESTARTS AGE hello-world-fc5fd8f57-dmfjt

    1/1 Running 0 26h hello-world-fc5fd8f57-db42a 1/1 Running 0 26h hello-world-fc5fd8f57-u9htw 1/1 Running 0 26h hello-world-fc5fd8f57-btw4h 1/1 Running 0 26h hello-world-fc5fd8f57-t4qn5 1/1 Running 0 26h hello-world-fc5fd8f57-nw5yj 1/1 Running 0 26h hello-world-fc5fd8f57-ny53n 1/1 Running 0 26h hello-world-fc5fd8f57-klxw5 1/1 Running 0 26h hello-world-fc5fd8f57-k0t5s 1/1 Running 0 26h hello-world-fc5fd8f57-653na 1/1 Running 0 26h hello-world-fc5fd8f57-xfis4 1/1 Running 0 26h hello-world-fc5fd8f57-klds7 1/1 Running 0 26h hello-world-fc5fd8f57-babre 1/1 Running 0 26h hello-world-fc5fd8f57-aj5et 1/1 Running 0 26h hello-world-fc5fd8f57-q5aha 1/1 Running 0 26h hello-world-fc5fd8f57-au5a5 1/1 Running 0 26h
  85. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: replicas: 15

    template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
  86. What if we need to scale the amount of pods

    automatically?
  87. Horizontal Pod Auto Scaling

  88. Demo

  89. Cluster needs metrics-server

  90. Create Horizontal Pod Autoscaler

  91. $ kubectl top pods NAME CPU(cores) MEMORY(bytes) hello-world-fc5fd8f57-dmfjt 0m 1Mi

    hello-world-fc5fd8f57-ntasr 0m 1Mi
  92. $ kubectl autoscale deployment hello-world --min=1 --max=6 --cpu-percent=5

  93. $ kubectl get horizontalpodautoscaler hello-app NAME REFERENCE TARGETS MINPODS MAXPODS

    REPLICAS hello-app Deployment/hello-world <unknown>/5% 1 6 2
  94. Increase load to containers by sending lots of requests to

    it
  95. $ ab -c 900 -n 15000 http://195.192.129.xyz/

  96. How does the Horizontal Pod Autoscaler work?

  97. See what the autoscaler is doing

  98. $ kubectl describe horizontalpodautoscaler hello-world ... Events: Type Reason Age

    From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 15m (x222 over 71m) horizontal-pod-autoscaler New size: 6; reason: Current number of replicas above Spec.MaxReplicas
  99. What if a process needs more resources?

  100. Manual vertical scaling

  101. Demo

  102. Change pod resource requests and limits

  103. $ kubectl edit deployment hello-world

  104. kind: Deployment ... containers: - name: hello-world image: nginxdemos/hello:0.2 resources:

    requests: cpu: 100m memory: 256Mi limits: cpu: 100m memory: 256Mi ...
  105. Pods are re-scheduled on the cluster if necessary

  106. If there are not enough resources, Pods remain pending

  107. $ kubectl get pods NAME READY STATUS RESTARTS AGE hello-world-fc5fd8f57-dmfjt

    0/1 Pending 0 26h hello-world-fc5fd8f57-db42a 0/1 Pending 0 26h hello-world-fc5fd8f57-u9htw 0/1 Pending 0 26h hello-world-fc5fd8f57-btw4h 0/1 Pending 0 26h hello-world-fc5fd8f57-t4qn5 1/1 Running 0 26h hello-world-fc5fd8f57-nw5yj 1/1 Running 0 26h hello-world-fc5fd8f57-ny53n 1/1 Running 0 26h hello-world-fc5fd8f57-klxw5 1/1 Running 0 26h hello-world-fc5fd8f57-k0t5s 1/1 Running 0 26h hello-world-fc5fd8f57-653na 1/1 Running 0 26h hello-world-fc5fd8f57-xfis4 1/1 Running 0 26h hello-world-fc5fd8f57-klds7 1/1 Running 0 26h hello-world-fc5fd8f57-babre 1/1 Running 0 26h hello-world-fc5fd8f57-aj5et 1/1 Running 0 26h hello-world-fc5fd8f57-q5aha 1/1 Running 0 26h hello-world-fc5fd8f57-au5a5 1/1 Running 0 26h
  108. $ kubectl descript pod hello-world-fc5fd8f57-dmfjt Events: Type Reason Age From

    Message ---- ------ ---- ---- ------- Warning FailedScheduling 2s (x5 over 10s) default- scheduler 0/3 nodes are available: 3 Insufficient cpu.
  109. What if we need more nodes to schedule additional pods?

  110. Node Scaling

  111. Manually adding more VMS to the cluster

  112. Cloud provider dependent

  113. SysEleven MetaKube

  114. None
  115. Cluster Management API https:/ /github.com/kubernetes-sigs/cluster-api

  116. Kubermatic Machine Controller https:/ /github.com/kubermatic/machine-controller

  117. MachineDeployment MachineSet Machine Machine MachineController VM VM Node Node Rolling

    Updates Ensure Replica Count Create Listen
  118. What happens if you have to add nodes outside of

    working hours?
  119. Node Auto Scaling

  120. SysEleven MetaKube

  121. Cluster Auto Scaler https:/ /github.com/kubernetes/autoscaler

  122. Cluster Management API https:/ /github.com/kubernetes-sigs/cluster-api

  123. Demo

  124. kubectl get nodes NAME STATUS ROLES AGE VERSION kubermatic-fhgbvx65xg-7flj7 Ready

    <none> 7d1h v1.12.2 kubermatic-fhgbvx65xg-hmgd4 Ready <none> 7d2h v1.12.2 kubermatic-fhgbvx65xg-q287t Ready <none> 7d2h v1.12.2
  125. kubectl get machines -n kube-system NAME AGE machine-kubermatic-fhgbvx65xg-7flj7 7d machine-kubermatic-fhgbvx65xg-hmgd4

    7d machine-kubermatic-fhgbvx65xg-q287t 7d
  126. apiVersion: cluster.k8s.io/v1alpha1 kind: MachineDeployment metadata: annotations: cluster-autoscaler/minsize: 1 cluster-autoscaler/maxsize: 15

    name: scalable-machine-deployment namespace: kube-system spec: replicas: 1 ...
  127. ... cloudProvider: openstack cloudProviderSpec: availabilityZone: dbl flavor: m1-small floatingIpPool: ext-net

    identityEndpoint: "https://cloud.sys11.net/v3" image: "Ubuntu 18.04" network: kubermatic-c123 region: dbl securityGroups: - kubermatic-c123 operatingSystem: ubuntu operatingSystemSpec:
  128. ... sshPublicKeys: - "..." ... versions: kubelet: "v1.12.2"

  129. kubectl get machines -n kube-system NAME AGE machine-kubermatic-fhgbvx65xg-7flj7 7d machine-kubermatic-fhgbvx65xg-hmgd4

    7d machine-kubermatic-fhgbvx65xg-q287t 7d scalable-machine-deployment-hueab94ghq-abiur 3m scalable-machine-deployment-hueab94ghq-4uhva 3m scalable-machine-deployment-hueab94ghq-vhues 3m
  130. Summary

  131. With auto scaling you can ...

  132. Safe valuable resources

  133. Safe the environment

  134. Kubernetes makes auto scaling a lot easier

  135. And you don't have to get up at night

  136. b.hofmann@syseleven.de https:/ /twitter.com/BastianHofmann Join us in the SysEleven Lounge metakube@syseleven.de

    http:/ /speakerdeck.com/u/bastianhofmann s.pearce@syseleven.de