Future-proofing Production Systems.

69c2f55e7b157c112c0d988ddba7484d?s=47 kavya
October 20, 2017

Future-proofing Production Systems.

How does your system perform under load? What’re the bottlenecks, and how does it fail at its limits?
How do you stay ahead as your system evolves and its workload grows?

In this talk, we’ll explore strategies to prepare systems for flux and scale. From Facebook’s Kraken that provides shadow traffic, to the custom load simulator we built at Samsara, we’ll discuss how to go about understanding your systems as they run today, and planning for how they will tomorrow.

69c2f55e7b157c112c0d988ddba7484d?s=128

kavya

October 20, 2017
Tweet

Transcript

  1. Future-proofing Production Systems @kavya719

  2. kavya

  3. analyzing of the performance of systems

  4. performance capacity • What’s the additional load the system can

    support, 
 without degrading response time? • What’re the system utilization bottlenecks? • What’s the impact of a change on response time,
 maximum throughput? • How many additional servers to support 10x load? • Is the system over-provisioned?
  5. more robust, performant, scalable …use prod to make prod better.

  6. more robust, performant, scalable …use prod to make prod better.

    A/ B testing, canaries and ramped deploys. chaos engineering. stressing the system. empirically determine performance characteristics, bottlenecks.
  7. OrgSim etc. a standard load simulator Little’s law Kraken a

    fancy load “simulator”
 utilization law, the USL stepping back the sweet middle ground
  8. Kraken

  9. • Facebook’s load “simulator”.
 In use in production since ~2013.


    • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken
  10. • Facebook’s load “simulator”.
 In use in production since ~2013.


    • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken maximum throughput, subject to a response time constraint.
  11. • Facebook’s load “simulator”.
 In use in production since ~2013.


    • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken maximum throughput, subject to a response time constraint.
  12. • Facebook’s load “simulator”.
 In use in production since ~2013.


    • Used to determine a system’s capacity. • And to identify and resolve utilization bottlenecks.
 • Allowed them to increase Facebook’s capacity 
 by over 20% using the same hardware! kraken maximum throughput, subject to a response time constraint.
  13. the model stateless servers that serve requests without using sticky

    sessions/ server affinity. load can be controlled by re-routing requests for example, this does not apply to a global message queue. downstream services respond to upstream service load shifts for example, a web server querying a database.
  14. load generation need a representative workload…use live traffic! traffic shifting:


    increase the fraction of traffic to a region, cluster, server, by
 adjusting the weights that control load balancing, monitoring need reliable metrics that track the health of the system. p99 response time HTTP error rate user experience ] CPU utilization connections, queue length safety ]
  15. response time threshold capacity …is this good or is there

    a bottleneck? let’s run it!
  16. interlude: performance modeling assume no upstream saturation, so service time

    constant i.e.
 response time ∝ queueing delay. model a web server as a queueing system. queueing delay + service time response time = Step I: single server capacity
  17. utilization = throughput * service time (Utilization Law) queueing delay

    increases 
 (non-linearly); so, response time. throughput increases utilization increases throughput response time “busyness”
  18. utilization = throughput * service time (Utilization Law) throughput “busyness”

    queueing delay increases 
 (non-linearly); so, response time. throughput increases utilization increases
  19. Iff linear scaling, cluster of N servers’ capacity = single

    server capacity * N Step II: cluster capacity theoretical cluster capacity throughput concurrency • contention penalty
 due to queueing for shared resources • consistency penalty
 due to increase in service time Universal Scalability Law (USL): target cluster capacity should account for this. … but systems don’t scale linearly.
  20. Facebook sets target cluster capacity = 93% of theoretical. …is

    this good or is there a bottleneck?
  21. cluster capacity is ~90% of theoretical, so there’s a bottleneck

    to fix! Facebook sets target cluster capacity = 93% of theoretical.
  22. bottlenecks uncovered
 • cache bottleneck • network saturation • poor

    load balancing • misconfiguration Also, insufficient capacity 
 i.e. no bottlenecks per-se, but organic growth. …so, can we have it too?
  23. OrgSim etc.

  24. load generation Run a configurable number of virtual clients. A

    virtual client sends/receives in a loop.
 
 Use synthetic workloads.
 OrgSim’s load profile is based on historical data. monitoring external to the load simulator system. We use Datadog alerts on metric thresholds.
  25. gotchas • synthetic workloads may not be representative of actual.

  26. number of virtual clients (N) = 1, …, 100 response

    time concurrency (N) wrong shape for response time curve! should be (from the USL) concurrency (N) response time … load simulator hit a bottleneck!
  27. gotchas • synthetic workloads may not be representative of actual.

    • load simulator may hit a bottleneck! concurrency = throughput * response time Little’s Law: number of virtual clients actually running
  28. stepping back

  29. Case for performance testing in production
 empiricism is queen. …performance

    testing or modeling? yes. Case for performance modeling expectations better than no expectations.
  30. @kavya719 speakerdeck.com/kavya719/future-proofing-production-systems Special thanks to Eben Freeman for reading drafts

    of this. Kraken
 https://research.fb.com/publications/kraken-leveraging-live-traffic-tests-to-identify- and-resolve-resource-utilization-bottlenecks-in-large-scale-web-services/ Performance modeling
 Performance Modeling and Design of Computer Systems, Mor Harchol-Balter How to Quantify Scalability, Neil Gunther:
 http://www.perfdynamics.com/Manifesto/USLscalability.html
  31. None
  32. throughput latency non-linear responses to load throughput concurrency non-linear scaling

  33. throughput latency non-linear responses to load throughput concurrency non-linear scaling

    microservices: systems are complex continuous deploys:
 systems are in flux
  34. load generation need a representative workload. …use live traffic. traffic

    shifting profile (read, write requests) arrival pattern including traffic bursts capture and replay
  35. edge weight cluster weight server weight adjust weights that control

    load balancing, to increase the fraction of traffic to a cluster, region, server. traffic shifting
  36. monitoring need reliable metrics that track the health of the

    system. p99 response time HTTP error rate user experience ] CPU utilization
 memory utilization connections, queue length safety ]
  37. let’s run it! kraken ]

  38. the cloud (AWS) industry site devices web dashboard user’s browser

    samsara
  39. the cloud (AWS) industry site hubs data processors storage devices

    frontend servers web dashboard user’s browser websocket
 sticky sessions samsara