From Zero To Capacity Planning 2016 Edition!

@Randommood INES 
 Sombra

Globally distributed and Highly available

NOT AN INFRA person

INSTRUMENT MONITOR & ALERT PLAN & PREDICT Why care? About Capacity planning ✨ ✨

Capacity planning 
 101

Defining Capacity planning Measuring, planning, & managing system growth Determines what your system needs & when From the observation of actual traffic. Use current performance as baseline for predictions Must happen regardless of what you might optimize in the future

a Fastly POP

I Rule! Evaluates weekly global POPs performance & makes projections Weekly plublishes capacity performance report Plans for our physical capacity & transit capacity Meet Catharine

Planning Our Capacity Contextual metrics - Network Capacity (Gb) 
 - Ordered Network Capability (Gb) 
 - Planned Network Capacity (Gb)
 - RPS Capacity (k) 
 - Network peak (Gb) 
 - RPS peak (k) 
 - Site CPU Peak (%) 
 - Network Utilization (%) Over 30%: flagged, Over 70%: Red status

Fastly Insights Our ability to correctly plan for capacity is critical to our bottom line Capacity doesn’t just involve hardware; software & transfer optimizations matter People affect capacity

allspaW’s Admiration society

ARE WE RIGHT NOW? We have to be this fast & reliable 
 X per second & Y% Uptime MEASURE HOW/RELIABLE WE ARE HARDWARE SOFTWARE ARCHITECTURE CHANGE / ADD / REMOVE FIGURE OUT HOW TO STAY FAST/RELIABLE ENOUGH Yes! No! Allspaw's Wisdom From The Art of Capacity Planning

System’s Ceiling: critical level of a resource that cannot be crossed without failure. Find yours Another form of Capacity Planning: Controlled load testing Predictions = ceilings + historical data Allspaw's Wisdom

Allspaw's Wisdom System architecture can affect your ability to add capacity Identify & track your application’s metrics Tying metrics to user behavior is helpful If you don’t have ways to measure your current capacity you can’t plan

& Putting things in practice Findings

Unexpected Challenges The goal when adding capacity is no service disruption Localhost is the goddamn devil Gap from metric/graph to insight can be huge Slowness is the nemesis of distributed system

more Insights Capacity tied to murky organizational structure is both good & bad (but mostly bad) Mind your system dependencies: practice defensive system design & architecture New SLAs can be tricky CAPACITY PLANNING ALERTING MONITORING

more Insights Possible to have plenty of capacity and a slow site nonetheless Projections & curve fitting are guesses Keep track of API calls & their rates Always gonna be spikes & hiccups. Take the bad with the good & plan for it

TL;DR Is a process not a one time event Pushes you to better understand your system, its capacity & its boundaries - that is good! Proactivity is best Capacity planning Request lifecycle gets tricky System boundaries, dependencies & SLAs must be discussed Your system’s capacity may bound other systems capacity Distributed systems

github.com/Randommood/ZerotoCapacityPlanning Special Thanks to: Catharine Strauss, Alan Kasindorf, Matt Whiteley, Caitie McCaffrey, Thom Mahoney, Mike O’Neill, Devon O’Dell, Katherine Daniels, Nathan Taylor, Bruce Spang, and Greg Bako Thank you !