Tests should be fast: why and how?

Speed is power, fast is good. But how about tests? Here I explain why speed of tests is important and how to improve it.

Transcript

1. Tests should be fast: why and how? Event: Tokyo Rubyist Meetup Date: 2019-09-04 Speaker: OKURA Masafumi
2. Tests should be fast.
3. Any opposition?✋
4. Because: Fast is good.
5. That’s all, thank you!
7. Introduction: Why should tests be faster?
8. Short answer: It’s a low hanging fruit.
9. Long answer: Making tests faster is easier than making production code faster.
10. To make production code faster, we: • Measure performance with some specific tools such as “rack- mini-profiler” or “ruby-prof”. • Reduce N+1 queries with “bullet”. • Try to make Ruby code faster but it’s not a bottleneck. • Sometimes spend lots of time for small improvements.
11. To make tests faster, we: • Profile tests with “—profile” option (if we use RSpec). • Look into slow tests and find the causes (described later). • Improve slow tests and win! • (If we struggle to find a bottleneck, “test-prof” gem helps us)!
12. But you say: What’s bad with slow tests?
13. Short answer: Fast is good.
14. Long answer: Tests are critical in our feedback cycle, so faster tests mean faster feedback.
15. Imagine: software development without tests.
16. Software development without tests • We often miss errors/bugs before shipping. • We cannot do any refactoring because it causes bugs. • Fixing a bug causes more bugs including regressions. • These mean we develop software without any feedback.
17. Tests are virtually the only way to get feedback often enough.
18. Next: How can we make tests faster?
19. Before getting started, one thing to say:
21. Capybara is really slow, so are feature tests and system tests.
22. However: they are all good and cannot be removed.
23. Removing capybara is not an option here.
24. Strategy to make tests faster 1. Measure and profile using “profile” option and “test-prof” gem. 2. Make slow tests faster taking care of setup (before hook) and database interactions (typically “let” in RSpec). 3. Removing things your tests don’t need (optimizations and compilations).
25. Strategy to make tests faster 1. Measure and profile using “profile” option and “test- prof” gem. 2. Make slow tests faster taking care of setup (before hook) and database interactions (typically “let” in RSpec). 3. Removing things your tests don’t need (optimizations and compilations).
26. Measure and profile using “profile” option and “test-prof” gem
27. “profile” option
28. Profile option is: • Available natively for RSpec and as a plugin for Minitest • Used to show top 10 (default) slowest test cases. • The easiest way to know where’s slow in your test suite.
29. “test-prof” gem
30. “test-prof” is: • A collection of tools to profile your test suite. • Integrated with well-known general Ruby profiling tools such as “ruby-prof” and “stackprof”. • Used by many companies including GitLab and Dev.to.
31. Example: Measuring time on before/let • Set RD_PROF=1 and run RSpec as usual • Get the output like below:
32. Strategy to make tests faster 1. Measure and profile using “profile” option and “test-prof” gem. 2. Make slow tests faster taking care of setup (before hook) and database interactions (typically “let” in RSpec). 3. Removing things your tests don’t need (optimizations and compilations).
33. Make slow tests faster taking care of setup (before hook) and database interactions (typically “let” in RSpec)
34. Question: How many minutes of the test suite are spent on before and let?
35. Answer: Most of the time. (In my case, 5:50 out of 7:35 are spent on before and let)
36. Measured with “test-prof” gem.
37. From here we go refactoring.
38. Notes on reducing before • Too much DRYness is the enemy here. Duplicate code from before block to it block and cut redundant parts. • Don’t mimic login through web interaction. Set cookie directly. • Use tags effectively. Load something only with certain tags.
39. Notes on reducing let • Maybe you should use instance variables instead of let.
40. Notes on reducing let • Maybe you should use instance variables instead of let.
41. Notes on reducing let • If we use more than 5 lets in one example, maybe too many lets (one of the exceptions is search). • Prefer build over create, prefer build_stubbed over build. • Use let! only when required.
42. Strategy to make tests faster 1. Measure and profile using “profile” option and “test-prof” gem. 2. Make slow tests faster taking care of setup (before hook) and database interactions (typically “let” in RSpec). 3. Removing things your tests don’t need (optimizations and compilations).
43. Removing things your tests don’t need (optimizations and compilations)
44. Real story:
45. spec/support/stripe.rb require 'fake_stripe' RSpec.configure do |config| config.before(:each) do FakeStripe.stub_stripe end end
46. How many minutes does it take? (Hint: minute, not second)
47. 9 minutes.
48. Image optimization:
49. Several minutes (I forgot exact number)
50. Compiling with Webpacker
51. Compile once # config/webpacker.yml test: <<: *default compile: false # spec/rails_helper.rb RSpec.configure do |config| config.before(:suite) do Webpacker.compile end end
52. A few minutes.
53. Remove these for fast tests • Redundant `before each` call (or use some technique to load code only when required). • Redundant asset optimizations. • Webpacker compilation.
54. Recap
55. Strategy to make tests faster 1. Measure and profile using “profile” option and “test-prof” gem. 2. Make slow tests faster taking care of setup (before hook) and database interactions (typically “let” in RSpec). 3. Removing things your tests don’t need (optimizations and compilations).
56. One more thing:
57. Who am I?
58. pp self • Name: OKURA Masafumi • Software developer for several years using Ruby and Rails • Vimmer (and a staff of VimConf 2019!) • Freelance now, seeking international jobs!
59. References • https://relishapp.com/rspec/rspec-core/docs/configuration/ profile-examples • https://github.com/palkan/test-prof