Cost Optimization with Cluster Autoscaler

Slide 1

Slide 1 text

Cost Optimization with Cluster Autoscaler Takeshi Kondo / @chaspy Lightning Talks

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Target Who knows Kubernetes

Slide 4

Slide 4 text

tl;dr Cluster Autoscaler is useful

Slide 5

Slide 5 text

We are deploying (almost) application on Kubernetes

Slide 6

Slide 6 text

All application is deployed for each PR https://quipper.hatenablog.com/entry/future-with-kubernetes

Slide 7

Slide 7 text

Appendix: namespaces and num of pods/deployment in staging

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

We should do scale-in/out by manually

Slide 10

Slide 10 text

Problem

Slide 11

Slide 11 text

How to solve the problem? • Cluster Autoscaler • Horizontal Pod Autoscaler • Vertical Pod Autoscaler

Slide 12

Slide 12 text

How to solve the problem? • Cluster Autoscaler • Horizontal Pod Autoscaler • Vertical Pod Autoscaler Tried and reverted because “JaJa Uma” (unmanageable) Not trying

Slide 13

Slide 13 text

Agenda • Introduction / Background • Cluster Autoscaler • How to scale-in/out • Check the code • And more topic • (Production) Cluster Autoscaler works when releasing • (Production) PodDisruptionBudget prevents to scale-in • (Production) Making zero downtime when scaling-in • (Staging) Remove stale PullRequests namespaces • Achievement • Conclusion

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Cluster Autoscaler *1 • Scale-up • When any unschedulable pods exist • Scale-in (Need All below conditions) • If no scale-up is needed, • The sum of cpu and memory requests of all pods running on this node is smaller than 50% of the node's allocatable • All pods running on the node can be moved to other nodes • For example, PodDisruptionBudget prevents. See for details *2 • It doesn't have scale-down disabled annotation • "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" *1 https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md *2 https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can prevent-ca-from-removing-a-node

Slide 16

Slide 16 text

Check the code: How to scale-in? https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/scale_down.go#L568 Default: 0.5

Slide 17

Slide 17 text

Check the code: How to scale-in? https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/simulator/cluster.go#L158 Get cpu and memory Save the bigger metrics

Slide 18

Slide 18 text

Check the code: How to scale-in? https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/simulator/cluster.go#L61

Slide 19

Slide 19 text

Check the code: How to scale-in? https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/vendor/github.com/aws/aws-sdk-go/service/autoscaling/api.go#L5100

Slide 20

Slide 20 text

Check the code: How to scale-in? https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_TerminateInstanceInAutoScalingGroup.html

Slide 21

Slide 21 text

Slide 22

Slide 22 text

And more topic • (Production) Cluster Autoscaler works when releasing • (Production) PodDisruptionBudget prevents to scale-in • (Production) Making zero downtime when scaling-in • (Staging) Remove stale PullRequests namespaces

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Cluster Autoscaler works when releasing • When rolling update, the count of "max surge" pods is created so there is temporarily Unschedulable pod. Num of running pods Num of desired capacity of ASG

Slide 25

Slide 25 text

Slide 26

Slide 26 text

PodDisruptionBudget prevented to scale-in • All deployment has “safe-draining” label • PDB keeps maxUnavailable is 1 In all deployment

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Making zero downtime when scaling-in https://speakerdeck.com/chaspy/rolling-update-kubernetes-deployment-with-zero-downtime

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Remove stale PullRequests namespaces

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Achievement (Staging) Instance Class Node Count (Aug.) Node Count (Sep.) Metrics Usage / Capacity (Aug.) Usage / Capacity (Sep.) Japan/ default r5.2xlarge 20.56 17.65 Memory 0.53 0.59 Global/ default r5.2xlarge 12 12 Memory 0.51 0.56 $443.840000 (monthly) * (20.56 - 17.65) = $1291.5744

Slide 33

Slide 33 text

Achievement (Production) Instance Class Node Count (Aug.) Node Count (Sep.) Metrics Usage / Capacity (Aug.) Usage / Capacity (Sep.) Japan/ default m5.xlarge 6.38 8.26 Memory 0.68 0.58 Japan/ api m5.2xlarge 17.1 13.42 CPU 0.17 0.14 Global/ default m5.xlarge 6 6 Memory 0.51 0.51 Global/ api m5.2xlarge 6 6 CPU 0.15 0.17 $181.040000 (monthly) * (6.38 - 8.26) = $340.3552 $362.080000 (monthly) * (17.1 - 13.42) = $1332.4544

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Conclusion • Cluster Autoscaler saves the cost of both operation and infrastructure • (In Staging) With deleting stale namespaces • (In Production) With making no downtime when scaling-in • Pod level (Horizontal / Vertical) autoscaling is not yet introduced • Mean we SRE should increase pods/nodes when high loading • Read Code

Slide 37

Slide 37 text

Special Thanks • @yuya-takeyama / SRE • Thanks to review PR • @rbmrclo / SRE • Thanks to review PR • @hiroki-iwasaki / People & Culture • Thanks to organize the “Lightning Talks”

Slide 38

Slide 38 text

Thank You! chaspy chaspy_ Site Reliability Engineer at Quipper Takeshi Kondo SRE Lounge Terraform-jp