Building a toolkit to detect performance regressions in Ruby on Rails core

My talk from RailsConf 2015.

1. Kir Shatrov, RailsConf 2015 Building a toolkit to detect performance regressions in Ruby on Rails core
2. @kirs @kirshatrov !"
4. Catching Performance Regressions in Rails
5. What’s a performance regression?
6. What’s a performance regression in Rails?
7. Page load 250ms → 4s
8. Case Study
9. 4.2rc1
11. 2 times slower
12. 4.2rc3
13. Faster framework = Faster app
14. Examples from Rails commits
15. Metrics
16. Timing & Allocations
17. Faster Methods = Faster App
18. start = Time.now.to_f sleep 5 ends = Time.now puts "it took #{ends - start}"
19. Allocations = GC work Less allocations = less GC work
20. before = GC.stat[:total_allocated_object] 10_000.times do { "key" => "value" } end after = GC.stat[:total_allocated_object] puts "allocated: #{after - before}"
21. Basic primitives
22. Examples from Rails commits String
33. Examples from Rails commits Hash
40. Optimize only “hot” code
41. Track the changes
42. The Idea to build a service
43. Track it with benchmarks
44. Any existing benchmarks?
46. Setup Discourse Populate records Start Unicorn with RAILS_ENV=production Make requests with Apache Bench to the list of endpoints Print timings and memory usage
47. Components to benchmark activerecord activemodel actionview actioncontroller activesupport
48. 3.2 – 4.0 – 4.1 – 4.2
53. ActiveRecord Finders require_relative 'support/activerecord_base.rb' require_relative 'support/benchmark_rails.rb' User.create!(name: 'kir', email: 'shatrov@me.com') m = Benchmark.rails(100, "activerecord/#{db_adapter}/finders") do User.find(1) end puts m.to_json
54. Disable GC is necessary Warm up the code Make N iterations Calculate the mean timing and object allocations
55. ActiveRecord::Base#create require_relative 'support/activerecord_base.rb' require_relative 'support/benchmark_rails.rb' fields = { name: "Kir", notes: "Note", created_at: Date.today } Benchmark.rails(1000, "activerecord/#{db_adapter}/create") do Exhibit.create(fields) end
56. Correct benchmarks
57. Full app benchmark
58. Full app benchmark before
59. stackprof by Aman Gupta github.com/tmm1/stackprof
60. 4.2 4.1
61. config/environments/production.rb
62. Full app benchmark after
63. ✅ Benchmarks
66. The Service
67. What do we want? See benchmark for every commit See benchmark for every PR Report to PR author (“activerecord became 2% slower”) See charts
68. railsperf.evilmartians.io The prototype of the service. Built in January
69. rubybench.org by Sam Saffron and Guo Xiang Tan “Alan”
71. 3.6GHz Intel® Xeon® E3-1270 v3 Quad-Core 4 x 8GB Micron ECC Samsung SSD 845DC EVO - 240 GB Sponsored hardware
72. ruby-bench + rails
73. github.com/ruby/ruby webhook web app docker server benchmark suite
75. github.com/ruby-bench web app docker files benchmark suite
76. github.com/ruby-bench web app docker files benchmark suite
77. github.com/ruby-bench web app docker files benchmark suite
78. github.com/ruby-bench web app docker files benchmark suite
79. Results
80. Running builds for Rails
81. 250k builds for Ruby
82. MRI performance regression caught
83. What’s next?
84. What’s next Pull Request benchmarks Weekly reports JRuby support
85. Overview What is a performance regression How to track them How to solve them How to write benchmarks Automate tracking!
86. How to track your app performance? Study how do Ruby objects work and how to treat them Write a Discourse-like benchmark and run it on Travis Subscribe to Rails Weekly
87. Ruby Under a Microscope
88. http://bit.ly/writing-fast-ruby
89. How to track your app performance? Study how do Ruby objects work and how to treat them Write a Discourse-like benchmark and run it on Travis Subscribe to Rails Weekly
90. Setup the app Populate records Start Unicorn with RAILS_ENV=production Make requests with Apache Bench to the list of endpoints Print timings and memory usage
91. How to track your app performance? Study how do Ruby objects work and how to treat them Write a Discourse-like benchmark and run it on Travis Subscribe to Rails Weekly
92. bit.ly/railsperf-gh github.com/rubybench bit.ly/railsperf-static
93. Thanks! @kirs @kirshatrov @evilmartians evilmartians.com