3 Ways to get Capacity Utilization Wrong

Right-sizing your environment is one of the most stressful decisions to make when moving to the cloud. If you under-provision resources, systems are at risk of going down and you lose money. If you over-provision, you’re wasting money that could be used elsewhere. In this presentation, we’ll share with you 3 ways we’ve learned how to get capacity utilization wrong and how we eventually got it right.

1. CPU measurement alone won’t give you the full view of your infrastructure utilization
2. You can’t measure the utilization for a metric if you don’t know how high it can go
3. If you only rely on request count and don’t include queue length, you will miss an early warning

Transcript

1. 3 Ways to get Capacity Utilization Wrong Jason Simpson
2. www.netuitive.com 2 Capacity Utilization Metric • A metric used to measure the rate at which potential output levels are being met or used
3. www.netuitive.com 3 Capacity Utilization Rate • Usually expressed as a percentage, it is computed by dividing the total capacity with the portion being utilized 100 50 0 75 units / 100 units capacity = .75 (75%) Utilization
4. www.netuitive.com 4 So that is easy… • Now just monitor the CPU metric to understand the capacity utilization
5. www.netuitive.com 5 So how did I get it wrong? • Most times a single metric will not give the full picture for capacity
6. www.netuitive.com 6 Need to find more metrics… • What key metrics make up the capacity utilization CPU, Memory, I/O, Network Speed, Queue Length, etc.
7. www.netuitive.com 7 Multiple metrics and still not right… • Understanding the high water mark for metrics is essential when calculating capacity
8. www.netuitive.com 8 How do we find total capacity… • Some times it is as simple as pulling attributes (or metadata) about the metric and other times the metric will need to be observed 4 GB RAM, 45 TPM
9. www.netuitive.com 9 Don’t toss in the towel… • Applying best practices and subject matter expertise are required
10. www.netuitive.com 10 What is this computed metric… • Creating computed metrics is required in most capacity utilization measurements (X IOPS/sec / 300 total IOPS/sec ) = Y (%) Utilization
11. www.netuitive.com 11 Takes some trial and error… • Unfortunately for most complex systems it will take some tuning to get correct You tried your best and you failed miserably. The lesson is, never try
12. www.netuitive.com 12 Capacity falls into two buckets… • Macro trend – expected capacity • Micro trends – unexpected capacity
13. www.netuitive.com 13 Planning verse real time… • Capacity planning is for macro trends and real time monitoring addresses micro trends Performance and capacity meet
14. www.netuitive.com 14 Applying best practices… • Create capacity utilization metrics from industry practices (Z Run Queue / X CPU#)*100 = Y (%) Utilization
15. www.netuitive.com 15 Don’t fall into old habits… • Over provisioning infrastructure, applications, and services
16. www.netuitive.com 16 Leveraging tools… • Automation is a goal for DevOps
17. www.netuitive.com 17 Getting automation correct… • Tying capacity utilization to automation helps to ensure systems have been provisioned correctly My system auto-provisioned 36 instances but should I leave them running ?
18. www.netuitive.com 18 Icing on the cake… • Behavior learning technologies coupled with metric correlation can assist with building accurate capacity utilization metrics
19. www.netuitive.com 19 Putting it all together… • Example capacity utilization for Amazon Web Services (AWS) Infrastructure as a Service (IaaS)
20. www.netuitive.com 20 Recap top 3 items… 1. Many times a single metric will not give the full picture for capacity 2. Understanding the high water mark for the metrics is essential when calculating capacity 3. Applying best practices and subject matter expertise are required