Auto Scaling Kubernetes Clusters on OpenStack

One of the key features of Kubernetes is to make up and down scaling of your application workloads as easy as possible. In this talk you will learn how to setup and use the horizontal pod autoscaler to dynamically up and down scale pods in complex setups. We are also excited to show you how we built a node autoscaler for Kubernetes on OpenStack, which allows to add worker nodes to running clusters, before the horizontal pod autoscaler runs out of resources. We will also give you an overview of the current state of the vertical autoscaler in Kubernetes. When configured, it will set the resource requests for CPU and memory automatically based on usage and thus allow proper scheduling onto nodes so that the appropriate resource amount is available for each pod.

Transcript

1. @BastianHofmann Auto Scaling Kubernetes Clusters On OpenStack Bastian Hofmann Simon Pearce
3. Container orchestration platform
4. Easy Scaling of applications
5. Why do we want to scale?
6. Handle increased load
7. Only pay what you actually need
8. Safe the environment
9. How can applications be scaled?
10. Horizontal Scaling
11. Increase or decrease the amount of instances and nodes
12. Vertical Scaling
13. Increase or decrease CPU and memory usage/capacity of one instance or node
14. Bare metal approach
15. Order a new server
16. Put the server into the datacenter
17. Install the Operating System
18. Provision the server with necessary dependencies
19. Deploy services to the new server
20. Reconfigure the load balancer
21. A lot of steps
22. This is way too slow
23. Cloud provider approach
24. Create a new VM from an OS image
25. Provision the server with necessary dependencies
26. Deploy services to the new server
27. Reconfigure the load balancer
28. Still a lot of steps
29. AutoScaling Groups
30. Proprietary APIs for every Cloud Provider
31. Kubernetes makes this easier
32. Standardized APIs
33. How does Kubernetes work?
34. • A container runs a docker image. • Only 1 process can run inside of a container Container
35. • A group of 1 or more containers • Shared network • Shared storage volumes Pod
36. php-fpm Nginx Filebeat
37. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
38. Horizontal Scaling
39. • Defines and manages how many instances of a pod should run Replica Set
40. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: replicas: 3 template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
41. Vertical Scaling
42. Container
43. CPU and Memory requests and limits
44. kind: Deployment ... containers: - name: hello-world image: nginxdemos/hello:0.2 resources: requests: cpu: 100m memory: 256Mi limits: cpu: 100m memory: 256Mi ...
45. "Requests" are used by Kubernetes for scheduling pods on nodes
46. "Limits" limit the container to not use more CPU and memory
47. You can change these values manually
48. And automatically
49. Don't get up at night
50. Focus on what is important
51. Let's show this live
52. Preparations for the demos
53. We need a cluster
54. Do-it-yourself vs. Managed Kubernetes
55. Setting up and maintaining Kubernetes is hard
56. Managed Kubernetes
57. Google GKE
59. SysEleven MetaKube
60. Easy upgrades
61. Easy scaling
62. Load Balancing
63. Distributed Persistent Storage
64. Backups
65. Premium support
66. Monitoring
67. You can focus on what is important
72. What if a we need to scale a pod?
73. Manual horizontal scaling
74. Demo
75. Create a Deployment
76. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: replicas: 1 template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
77. $ kubectl apply -f deployment.yaml
78. $ kubectl get pods NAME READY STATUS RESTARTS AGE hello-world-fc5fd8f57-dmfjt 1/1 Running 0 26h
79. Create a LoadBalancer
80. $ kubectl expose deployment hello-world--name=hello-world- svc --port=80 --target-port=80 --type=LoadBalancer
81. $ kubectl get service hello-world NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-world-svc LoadBalancer 10.10.10.102 195.192.xxx.xxx 80:31750/TCP 37s
82. Scale the deployment
83. $ kubectl scale deployment/hello-world --replicas 15
84. $ kubectl get pods NAME READY STATUS RESTARTS AGE hello-world-fc5fd8f57-dmfjt 1/1 Running 0 26h hello-world-fc5fd8f57-db42a 1/1 Running 0 26h hello-world-fc5fd8f57-u9htw 1/1 Running 0 26h hello-world-fc5fd8f57-btw4h 1/1 Running 0 26h hello-world-fc5fd8f57-t4qn5 1/1 Running 0 26h hello-world-fc5fd8f57-nw5yj 1/1 Running 0 26h hello-world-fc5fd8f57-ny53n 1/1 Running 0 26h hello-world-fc5fd8f57-klxw5 1/1 Running 0 26h hello-world-fc5fd8f57-k0t5s 1/1 Running 0 26h hello-world-fc5fd8f57-653na 1/1 Running 0 26h hello-world-fc5fd8f57-xfis4 1/1 Running 0 26h hello-world-fc5fd8f57-klds7 1/1 Running 0 26h hello-world-fc5fd8f57-babre 1/1 Running 0 26h hello-world-fc5fd8f57-aj5et 1/1 Running 0 26h hello-world-fc5fd8f57-q5aha 1/1 Running 0 26h hello-world-fc5fd8f57-au5a5 1/1 Running 0 26h
85. kind: Deployment apiVersion: extensions/v1beta1 metadata: name: hello-world spec: replicas: 15 template: spec: containers: - name: hello-world image: nginxdemos/hello:0.2 ports: - containerPort: 80
86. What if we need to scale the amount of pods automatically?
87. Horizontal Pod Auto Scaling
88. Demo
89. Cluster needs metrics-server
90. Create Horizontal Pod Autoscaler
91. $ kubectl top pods NAME CPU(cores) MEMORY(bytes) hello-world-fc5fd8f57-dmfjt 0m 1Mi hello-world-fc5fd8f57-ntasr 0m 1Mi
92. $ kubectl autoscale deployment hello-world --min=1 --max=6 --cpu-percent=5
93. $ kubectl get horizontalpodautoscaler hello-app NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS hello-app Deployment/hello-world <unknown>/5% 1 6 2
94. Increase load to containers by sending lots of requests to it
95. $ ab -c 900 -n 15000 http://195.192.129.xyz/
96. How does the Horizontal Pod Autoscaler work?
97. See what the autoscaler is doing
98. $ kubectl describe horizontalpodautoscaler hello-world ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 15m (x222 over 71m) horizontal-pod-autoscaler New size: 6; reason: Current number of replicas above Spec.MaxReplicas
99. What if a process needs more resources?
100. Manual vertical scaling
101. Demo
102. Change pod resource requests and limits
103. $ kubectl edit deployment hello-world
104. kind: Deployment ... containers: - name: hello-world image: nginxdemos/hello:0.2 resources: requests: cpu: 100m memory: 256Mi limits: cpu: 100m memory: 256Mi ...
105. Pods are re-scheduled on the cluster if necessary
106. If there are not enough resources, Pods remain pending
107. $ kubectl get pods NAME READY STATUS RESTARTS AGE hello-world-fc5fd8f57-dmfjt 0/1 Pending 0 26h hello-world-fc5fd8f57-db42a 0/1 Pending 0 26h hello-world-fc5fd8f57-u9htw 0/1 Pending 0 26h hello-world-fc5fd8f57-btw4h 0/1 Pending 0 26h hello-world-fc5fd8f57-t4qn5 1/1 Running 0 26h hello-world-fc5fd8f57-nw5yj 1/1 Running 0 26h hello-world-fc5fd8f57-ny53n 1/1 Running 0 26h hello-world-fc5fd8f57-klxw5 1/1 Running 0 26h hello-world-fc5fd8f57-k0t5s 1/1 Running 0 26h hello-world-fc5fd8f57-653na 1/1 Running 0 26h hello-world-fc5fd8f57-xfis4 1/1 Running 0 26h hello-world-fc5fd8f57-klds7 1/1 Running 0 26h hello-world-fc5fd8f57-babre 1/1 Running 0 26h hello-world-fc5fd8f57-aj5et 1/1 Running 0 26h hello-world-fc5fd8f57-q5aha 1/1 Running 0 26h hello-world-fc5fd8f57-au5a5 1/1 Running 0 26h
108. $ kubectl descript pod hello-world-fc5fd8f57-dmfjt Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2s (x5 over 10s) default- scheduler 0/3 nodes are available: 3 Insufficient cpu.
109. What if we need more nodes to schedule additional pods?
110. Node Scaling
111. Manually adding more VMS to the cluster
112. Cloud provider dependent
113. SysEleven MetaKube
115. Cluster Management API https:/ /github.com/kubernetes-sigs/cluster-api
116. Kubermatic Machine Controller https:/ /github.com/kubermatic/machine-controller
117. MachineDeployment MachineSet Machine Machine MachineController VM VM Node Node Rolling Updates Ensure Replica Count Create Listen
118. What happens if you have to add nodes outside of working hours?
119. Node Auto Scaling
120. SysEleven MetaKube
121. Cluster Auto Scaler https:/ /github.com/kubernetes/autoscaler
122. Cluster Management API https:/ /github.com/kubernetes-sigs/cluster-api
123. Demo
124. kubectl get nodes NAME STATUS ROLES AGE VERSION kubermatic-fhgbvx65xg-7flj7 Ready <none> 7d1h v1.12.2 kubermatic-fhgbvx65xg-hmgd4 Ready <none> 7d2h v1.12.2 kubermatic-fhgbvx65xg-q287t Ready <none> 7d2h v1.12.2
125. kubectl get machines -n kube-system NAME AGE machine-kubermatic-fhgbvx65xg-7flj7 7d machine-kubermatic-fhgbvx65xg-hmgd4 7d machine-kubermatic-fhgbvx65xg-q287t 7d
126. apiVersion: cluster.k8s.io/v1alpha1 kind: MachineDeployment metadata: annotations: cluster-autoscaler/minsize: 1 cluster-autoscaler/maxsize: 15 name: scalable-machine-deployment namespace: kube-system spec: replicas: 1 ...
127. ... cloudProvider: openstack cloudProviderSpec: availabilityZone: dbl flavor: m1-small floatingIpPool: ext-net identityEndpoint: "https://cloud.sys11.net/v3" image: "Ubuntu 18.04" network: kubermatic-c123 region: dbl securityGroups: - kubermatic-c123 operatingSystem: ubuntu operatingSystemSpec:
128. ... sshPublicKeys: - "..." ... versions: kubelet: "v1.12.2"
129. kubectl get machines -n kube-system NAME AGE machine-kubermatic-fhgbvx65xg-7flj7 7d machine-kubermatic-fhgbvx65xg-hmgd4 7d machine-kubermatic-fhgbvx65xg-q287t 7d scalable-machine-deployment-hueab94ghq-abiur 3m scalable-machine-deployment-hueab94ghq-4uhva 3m scalable-machine-deployment-hueab94ghq-vhues 3m
130. Summary
131. With auto scaling you can ...
132. Safe valuable resources
133. Safe the environment
134. Kubernetes makes auto scaling a lot easier
135. And you don't have to get up at night
136. b.hofmann@syseleven.de https:/ /twitter.com/BastianHofmann Join us in the SysEleven Lounge metakube@syseleven.de http:/ /speakerdeck.com/u/bastianhofmann s.pearce@syseleven.de