Future-proofing Production Systems.

How does your system perform under load? What’re the bottlenecks, and how does it fail at its limits?
How do you stay ahead as your system evolves and its workload grows?

In this talk, we’ll explore strategies to prepare systems for flux and scale. From Facebook’s Kraken that provides shadow traffic, to the custom load simulator we built at Samsara, we’ll discuss how to go about understanding your systems as they run today, and planning for how they will tomorrow.

Transcript

1. Future-proofing Production Systems @kavya719
2. kavya
3. analyzing of the performance of systems
4. performance capacity • What’s the additional load the system can support, 
 without degrading response time? • What’re the system utilization bottlenecks? • What’s the impact of a change on response time,
 maximum throughput? • How many additional servers to support 10x load? • Is the system over-provisioned?
5. more robust, performant, scalable …use prod to make prod better.
6. more robust, performant, scalable …use prod to make prod better. A/ B testing, canaries and ramped deploys. chaos engineering. stressing the system. empirically determine performance characteristics, bottlenecks.
7. OrgSim etc. a standard load simulator Little’s law Kraken a fancy load “simulator”
 utilization law, the USL stepping back the sweet middle ground
8. Kraken
9. • Facebook’s load “simulator”.
 In use in production since ~2013.
 • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken
10. • Facebook’s load “simulator”.
 In use in production since ~2013.
 • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken maximum throughput, subject to a response time constraint.
11. • Facebook’s load “simulator”.
 In use in production since ~2013.
 • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken maximum throughput, subject to a response time constraint.
12. • Facebook’s load “simulator”.
 In use in production since ~2013.
 • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken maximum throughput, subject to a response time constraint.
13. the model stateless servers that serve requests without using sticky sessions/ server affinity. load can be controlled by re-routing requests for example, this does not apply to a global message queue. downstream services respond to upstream service load shifts for example, a web server querying a database.
14. load generation need a representative workload…use live traffic! traffic shifting:
 increase the fraction of traffic to a region, cluster, server, by
 adjusting the weights that control load balancing, monitoring need reliable metrics that track the health of the system. p99 response time HTTP error rate user experience ] CPU utilization connections, queue length safety ]
15. response time threshold capacity …is this good or is there a bottleneck? let’s run it!
16. interlude: performance modeling assume no upstream saturation, so service time constant i.e.
 response time ∝ queueing delay. model a web server as a queueing system. queueing delay + service time response time = Step I: single server capacity
17. utilization = throughput * service time (Utilization Law) queueing delay increases 
 (non-linearly); so, response time. throughput increases utilization increases throughput response time “busyness”
18. utilization = throughput * service time (Utilization Law) throughput “busyness” queueing delay increases 
 (non-linearly); so, response time. throughput increases utilization increases
19. Iff linear scaling, cluster of N servers’ capacity = single server capacity * N Step II: cluster capacity theoretical cluster capacity throughput concurrency • contention penalty
 due to queueing for shared resources • consistency penalty
 due to increase in service time Universal Scalability Law (USL): target cluster capacity should account for this. … but systems don’t scale linearly.
20. Facebook sets target cluster capacity = 93% of theoretical. …is this good or is there a bottleneck?
21. cluster capacity is ~90% of theoretical, so there’s a bottleneck to fix! Facebook sets target cluster capacity = 93% of theoretical.
22. bottlenecks uncovered
 • cache bottleneck • network saturation • poor load balancing • misconfiguration Also, insufficient capacity 
 i.e. no bottlenecks per-se, but organic growth. …so, can we have it too?
23. OrgSim etc.
24. load generation Run a configurable number of virtual clients. A virtual client sends/receives in a loop.
 
 Use synthetic workloads.
 OrgSim’s load profile is based on historical data. monitoring external to the load simulator system. We use Datadog alerts on metric thresholds.
25. gotchas • synthetic workloads may not be representative of actual.
26. number of virtual clients (N) = 1, …, 100 response time concurrency (N) wrong shape for response time curve! should be (from the USL) concurrency (N) response time … load simulator hit a bottleneck!
27. gotchas • synthetic workloads may not be representative of actual. • load simulator may hit a bottleneck! concurrency = throughput * response time Little’s Law: number of virtual clients actually running
28. stepping back
29. Case for performance testing in production
 empiricism is queen. …performance testing or modeling? yes. Case for performance modeling expectations better than no expectations.
30. @kavya719 speakerdeck.com/kavya719/future-proofing-production-systems Special thanks to Eben Freeman for reading drafts of this. Kraken
 https://research.fb.com/publications/kraken-leveraging-live-traffic-tests-to-identify- and-resolve-resource-utilization-bottlenecks-in-large-scale-web-services/ Performance modeling
 Performance Modeling and Design of Computer Systems, Mor Harchol-Balter How to Quantify Scalability, Neil Gunther:
 http://www.perfdynamics.com/Manifesto/USLscalability.html
32. throughput latency non-linear responses to load throughput concurrency non-linear scaling
33. throughput latency non-linear responses to load throughput concurrency non-linear scaling microservices: systems are complex continuous deploys:
 systems are in flux
34. load generation need a representative workload. …use live traffic. traffic shifting profile (read, write requests) arrival pattern including traffic bursts capture and replay
35. edge weight cluster weight server weight adjust weights that control load balancing, to increase the fraction of traffic to a cluster, region, server. traffic shifting
36. monitoring need reliable metrics that track the health of the system. p99 response time HTTP error rate user experience ] CPU utilization
 memory utilization connections, queue length safety ]
37. let’s run it! kraken ]
38. the cloud (AWS) industry site devices web dashboard user’s browser samsara
39. the cloud (AWS) industry site hubs data processors storage devices frontend servers web dashboard user’s browser websocket
 sticky sessions samsara