Performance Testing & Profiling: A Virtuous Cycle by Dan Crosta

Donald Knuth famously said that we should avoid optimization 97% of the time and focus on the "critical 3%". How can we identify that 3%? How can we best focus our optimization efforts, and avoid the "root of all evil" that is premature optimization? This talk introduces key types of performance testing, and demonstrates how they can be paired with profiling techniques in a cycle of improvement.

Transcript

1. Dan Crosta | @lazlofruvous Performance Testing & Profiling: A Virtuous Cycle Dan Crosta | Magnetic http://late.am/ | @lazlofruvous
2. Dan Crosta | @lazlofruvous Overview • Performance testing web apps • Profiling with the standard library • Instrumentation • The Virtuous Cycle
3. Dan Crosta | @lazlofruvous Performance Testing Basics • Generate requests against your app • Record and replay production • Measure response time & error rate
4. Dan Crosta | @lazlofruvous Types of Performance Testing • Stress Test • Load Test ! (I won’t talk about) • Spike Test • Soak Test
5. Dan Crosta | @lazlofruvous Stress Test • Generate excessive load • Lots of requests • Slow/difficult requests • Adversarial testing • “How much?” • Identify breaking point • Especially if you control synthetic load
6. Dan Crosta | @lazlofruvous Load Test • Generate specific, constant load • Expected conditions • Exaggerated conditions • “What If?” • Capacity planning
7. Dan Crosta | @lazlofruvous Best Practices • Isolate testing from external influences • Use dedicated load testing environment • “Scaled down” copies of all components
8. Dan Crosta | @lazlofruvous Best Practices • Generate load consistently • Random considered harmful • Automate, automate, automate
9. Dan Crosta | @lazlofruvous Profiling • Batteries included • cProfile, pstats • Documentation not really included • Goofy, horrible API • Avoid run(), runctx()
10. Dan Crosta | @lazlofruvous Profiling import cProfile ! profiler = cProfile.Profile()
 profiler.enable() ! # do stuff here ... ! profiler.disable() profiler.dump_stats("myprogram.prof")
11. Dan Crosta | @lazlofruvous Profiling import pstats ! stats = pstats.Stats("myprogram.prof") stats.sort_stats("calls") stats.print_stats() ! # prints a lot of stuff ...
12. Dan Crosta | @lazlofruvous Profiling Live Demo!
13. Dan Crosta | @lazlofruvous Profiling in Practice • “Why is it slow?” • Good for identifying un-optimized code • Tight loops, recursion, lots of function calls • These are candidates for optimization • Good for identifying bottlenecks • Distinguish between slow external resource and slow application code
14. Dan Crosta | @lazlofruvous Other Profiles • line_profiler • http://pythonhosted.org/line_profiler/ • yappi • https://code.google.com/p/yappi/
15. Dan Crosta | @lazlofruvous Instrumentation • Use statsd to collect time-series metrics • Lightweight, low-overhead, always-on profiling • Two key instruments: • Counters let you know how many things happened • Timers let you know how long they each took
16. Dan Crosta | @lazlofruvous Instrumentation from statsd import statsd ! @statsd.timed("login.response_time") def handle_login(username, password): if authenticated(username, password): statsd.increment("login.valid", 1) return redirect("/homepage") else: saatsd.increment("login.invalid", 1) return redirect("/login")
17. Dan Crosta | @lazlofruvous Instrumentation: Counter
18. Dan Crosta | @lazlofruvous Instrumentation: Timer
19. Dan Crosta | @lazlofruvous Instrumentation • Learn what’s normal for your app • Bonus: alert when things are not normal • “Does the real world match expectations?”
20. Dan Crosta | @lazlofruvous The Virtuous Cycle Instrumentation
 & Alerting Performance Testing Profiling Performance Optimization
21. Dan Crosta | @lazlofruvous Thanks! Dan Crosta | Magnetic http://late.am/ | @lazlofruvous