Load Testing Rails Apps with Apache Bench, Siege, and JMeter

Your app works great in development and in production with dozens of users. Now, the marketing team wants to send out a blast: can the app handle hundreds? Thousands? How many users *can* it handle? These are tricky questions to parse, let alone answer, but any answer is going to involve some form of load testing.

Transcript

1. Load Testing Rails Apps with Apache Bench, Siege, and JMeter Steve Grossi
2. What is load testing?
3. Types of Load Testing • Generic Load Testing • Performance Benchmarking • Stress Testing • Surprise Load Testing! (a.k.a. DDOS attacks)
4. Generic Load Testing • What?
 Steadily increase load on a system up until the maximum it can handle • Why?
 To reveal potential problems in a system which may appear only when the system is under pressure (e.g. memory leaks) • Answers the Question
 "How does my app's behavior change with a lot of people using it?"
5. Performance Benchmarking • What?
 Testing a system under certain reproducible conditions • Why?
 To establish a baseline which can be tested against regularly to ensure a system's performance remains constant, or validate improvements as a result of change • Answers the Question
 "How is my app performing, and how does that compare with the past?"
6. Stress Testing • What?
 Increasing the load on a system up to and beyond what it can handle • Why?
 To test how a system fails when overwhelmed ensure that it fails and recovers gracefully • Answers the Question
 "What happens when my app gets swamped?"
7. Can you do load testing in dev? • Too many variables • Likely not similar enough to production • Unhelpful feedback loop at volume Technically, yeah, but you shouldn’t. Use a staging site.
8. Let’s do some testing!
9. Apache Bench $ ab -n 100 -c 10 -l http://beta.stevegrossi.com/on Concurrency Level: 10 Time taken for tests: 11.582 seconds Complete requests: 100 Failed requests: 0 Total transferred: 1691256 bytes HTML transferred: 1618056 bytes Requests per second: 8.63 [#/sec] (mean) Time per request: 1158.245 [ms] (mean) Time per request: 115.824 [ms] (mean, across all concurrent requests) Transfer rate: 142.60 [Kbytes/sec] received
10. Apache Bench Connection Times (ms) min mean [+/-sd] median max Connect: 52 100 85.4 58 393 Processing: 365 1014 1165.7 721 8057 Waiting: 245 915 1144.1 642 7887 Total: 420 1114 1167.5 790 8112
11. Apache Bench Percentage of the requests served within a certain time (ms) 50% 790 66% 980 75% 1066 80% 1198 90% 1448 95% 2065 98% 8111 99% 8112 100% 8112 (longest request)
12. Dealing with Authentication HTTP Authentication $ ab -n 10 -c 5 -A username:password http://beta.stevegrossi.com/on Session-based Authentication $ ab -n 100 -c 10 -C '_stevegrossi_session=BAh7CUkiD3Nlc...' http://beta.stevegrossi.com/meta
13. What Do We Do With This Data? • Change something and compare • Benchmark regularly (even automate it) to detect regressions • New Relic: see how your app behaved under load from the inside • Stress Testing: turn up the heat!
14. Siege $ siege -r 10 -c 10 http://beta.stevegrossi.com/on Transactions: 100 hits Availability: 100.00 % Elapsed time: 18.02 secs Data transferred: 0.50 MB Response time: 0.90 secs Transaction rate: 5.55 trans/sec Throughput: 0.03 MB/sec Concurrency: 5.01 Successful transactions: 100 Failed transactions: 0 Longest transaction: 2.31 Shortest transaction: 0.30
15. Dealing with Authentication HTTP Authentication via .siegerc login = username:password Session-based Authentication
 (get the cookie from the web inspector) $ siege -r 10 -c 10 -H 'Cookie: _stevegrossi_session=BAh7CUkiD3Nlc...' http://beta.stevegrossi.com/meta
16. Advantages of Siege $ siege -c 25 -t 5M -d 5 -f urls.txt • Multiple URLs (1 per line in urls.txt) • Random delay These help simulate real-world traffic to your app.
17. Siege + New Relic
18. Siege + New Relic
19. Siege + New Relic
20. Apache JMeter • With Apache Bench and Siege, we’re still not really simulating users • Not downloading assets (CSS, JS, images) • No continuity between requests • Reporting kind of meh • JMeter: all the things!
21. Apache JMeter First install Java $ brew install jmeter $ jmeter
22. ruby-jmeter • GUIs are as gross as they sound—let’s use a Ruby DSL! • ruby-jmeter to the rescue!
 https://github.com/flood-io/ruby-jmeter • Nested blocks and configuration via methods • Run via a Ruby file: $ touch jmeter.rb $ chmod +x jmeter.rb $ ./jmeter.rb
23. jmeter.rb #!/usr/bin/env ruby require ‘ruby-jmeter' test do defaults domain: 'beta.stevegrossi.com' cookies clear_each_iteration: true # threads(options) do ... view_results_tree summary_report end.run(gui: true)
24. jmeter.rb threads count: 10, loops: 1 do extract name: 'authenticity_token', regex: 'meta name=“csrf-token" content="(.+?)"' think_time 3000, 2000 # transaction 'Log In and View All Books' do end
25. jmeter.rb transaction 'Log In and View All Books' do visit '/meta/log-in' do assert contains: 'Sign In', scope: 'main' end submit '/meta/', { always_encode: true, fill_in: { 'utf8' => '✓', 'authenticity_token' => '${authenticity_token}', 'email' => 'test@example.com', 'password' => 'p455w0rd', 'commit' => 'Sign In', } } do assert contains: 'Dashboard', scope: 'main' end end
26. $ ./jmeter.rb
27. Report: Results Tree
28. Report: Summary Report
29. What Do We Do With This Data? (reprise) • Change something and compare • Benchmark regularly (even automate it) to detect regressions • New Relic: see how your app behaved under load from the inside • Stress Testing: turn up the heat!
30. Third Party Tools • Loader.io • Blitz.io • Flood.io • More every day… Automation! Tracking!
31. Recap • Load testing reveals things about your app you can’t learn in development, or by yourself. • Know what your app can handle. • Support decisions & refactorings with data. • Avoid surprises. • Be happy.
32. Thank You. Steve Grossi @stevegrossi work.stevegrossi.com