Running Just the Test Cases You Need

When you're writing software, fast feedback is key. The less you have to wait for your tests to run, the sooner you'll know whether or not your code is correct.

Ruby's two main test frameworks (minitest and RSpec) support several different techniques for testing only what you need for what you're currently working on, and nothing more. In this talk, we'll go through several of these practices for both frameworks, each more automated and awesome than the last.

Open Source Bridge 2017

Transcript

1. Testing Just
 What You Need @undees Open Source Bridge June 22, 2017
3. Public domain: https://commons.wikimedia.org/wiki/File:Vancouver_Tornado_-_RTX_loop.gif
4. Testing Just
 What You Need
5. The problem
6. Tests are slow
7. Test output is huge
8. It’s tough to find errors
9. The real problem
10. Feedback cycle is too slow
11. The solution:
12. Run just what you need
14. Let’s look at MiniTest first
15. Use the files, Luke!
16. test └── unit ├── coffee_test.rb └── test_helper.rb
 
 
 
 

17. test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ ruby -I.:./lib:./test test/unit/coffee_test.rb
18. test └── unit ├── coffee_test.rb └── test_helper.rb
 $ export RUBYOPT=-I.:./lib:./test 
 $ ruby -I.:./lib:./test test/unit/coffee_test.rb
19. test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ export RUBYOPT=-I.:./lib:./test 
 $ ruby test/unit/coffee_test.rb
20. test └── unit ├── coffee_test.rb └── test_helper.rb
 
 
 $ ruby test/unit/coffee_test.rb
21. test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ ruby test/unit/*_test.rb
22. test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ ruby test/**/*_test.rb

23. • Organize tests by directory • Wildcards (globs) are your friend
24. Run tests by name
25. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end # ... end
26. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end # Using macros from `shoulda` context 'coffee' do should 'cost $1' {} end end
27. Exact match
28. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 
 $ ruby test/unit/coffee_test.rb -n test_coffee_should_be_awesome
29. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 
 $ ruby test/unit/coffee_test.rb -n test_coffee_should_be_awesome
30. What about
 generated test names?
31. Shoulda
32. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'coffee should cost $1'
33. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'coffee should cost $1'
34. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'coffee should cost $1' HAHA NOPE
35. 
 class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'test_: coffee should cost $1. '
 ^
36. MiniTest::Spec
37. 
 describe Coffee do it 'costs $1' {} end
 $ ruby test/unit/coffee_spec.rb -n 'test_0001_costs $1'
38. Regular expression
39. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} context 'with milk' do should 'cost $1.25' {} end end end
 $ ruby test/unit/coffee_test.rb -n '/coffee/'
40. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} context 'with milk' do should 'cost $1.25' {} end end end
 $ ruby test/unit/coffee_test.rb -n '/coffee/'
41. class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} context 'with milk' do should 'cost $1.25' {} end end end
 $ ruby test/unit/coffee_test.rb -n '/milk/'
42. • Pass the -n option to run all matching tests • Use a regular expression inside single quotes:
 '/test case to run/' • Be mindful of names generated by Shoulda,
 MiniTest::Spec, etc.
43. Integrate with your editor
45. Whoa
46. You can go so far
 with what we’ve covered so far
47. But now…
48. RSpec
49. You can do all the stuff we just did!
50. spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 

51. spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 $ rspec spec/unit/coffee_spec.rb
52. spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 $ rspec spec/unit/*_spec.rb
53. spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 $ rspec spec/**/*_spec.rb
54. spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 $ rspec
55. • You can still separate specs by directory • RSpec can run a directory for you • Wildcards are still your friend • Run specs from your text editor
56. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 
 $ rspec -e 'Coffee costs $1'
57. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 
 $ rspec -e 'Coffee costs $1'
58. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 
 $ rspec -e 'milk'
59. • Pass the -e option to run all matching specs • No need for regex-style slashes; just use:
 'test case to run'
60. Failing line number is right there in the output!
61. $ rspec
 
 A cup of coffee costs $1 with milk probably tastes a little milder costs $1.25 (FAILED - 1)
 3 examples, 1 failure Failed examples: rspec ./spec/unit/coffee_spec.rb:13
 # A cup of coffee with milk costs $1.25
62. $ rspec
 
 A cup of coffee costs $1 with milk probably tastes a little milder costs $1.25 (FAILED - 1)
 3 examples, 1 failure Failed examples: rspec ./spec/unit/coffee_spec.rb:13
 # A cup of coffee with milk costs $1.25
63. Paste and run
64. $ rspec ./spec/unit/coffee_spec.rb:13
 
 A cup of coffee with milk costs $1.25 (FAILED - 1) 
 
 1 example, 1 failure
 
 Failed examples: rspec ./spec/unit/coffee_spec.rb:15
 # A cup of coffee with milk costs $1.25
65. $ rspec ./spec/unit/coffee_spec.rb:13
 
 A cup of coffee with milk costs $1.25 (FAILED - 1) 
 
 1 example, 1 failure
 
 Failed examples: rspec ./spec/unit/coffee_spec.rb:15
 # A cup of coffee with milk costs $1.25
66. $ rspec ./spec/unit/coffee_spec.rb:13
 
 A cup of coffee with milk costs $1.25 (FAILED - 1) 
 
 1 example, 1 failure
 
 Failed examples: rspec ./spec/unit/coffee_spec.rb:15
 # A cup of coffee with milk costs $1.25
67. • Run the spec on a specific line number:
 some_spec.rb:42 • Paste and run from test failure output
68. RSpec offers even more shortcuts
 for the studiously lazy
69. Focus on specific specs
70. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end
71. $ rspec A cup of coffee costs $1 with milk probably tastes a little milder costs $1.25 3 examples, 0 failures
72. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end
73. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' fit 'probably tastes a little milder' end
 end
74. $ rspec . 1 example, 0 failures
75. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end
76. RSpec.describe 'Coffee' do it 'costs $1' fcontext 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end
77. RSpec.fdescribe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end
78. A little configuration
 (automatic if you use rspec --init)
79. # spec_helper.rb RSpec.configure do |config| config.filter_run_when_matching :focus end
80. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' fit 'probably tastes a little milder' end
 end
81. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder', focus: true end
 end
82. Tag your examples!
83. RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end
84. RSpec.describe 'Coffee' do it 'costs $1', :payment context 'with milk' do it 'costs $1.25', :payment it 'probably tastes a little milder' end
 end
85. RSpec.describe 'Coffee' do it 'costs $1', :payment context 'with milk' do it 'costs $1.25', :payment it 'probably tastes a little milder' end
 end
86. # Run just the payment specs $ rspec -t payment
 
 # Run just the non-payment specs $ rspec -t ~payment
87. # Run just the specs related to one bug $ rspec -t bug_id:123

88. Run just what failed
89. $ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures
90. $ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures
91. $ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures
92. $ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures
93. $ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures
94. $ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures
95. $ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures
96. $ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures
97. …or just one failure at a time
98. $ rspec --next-failure
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) 1 example, 1 failure
99. $ rspec --next-failure
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) 1 example, 1 failure
100. How to set this up:
101. # spec_helper.rb RSpec.configure do |config| config.example_status_persistence_file_path =
 'spec/examples.txt' end
103. Reproduce errors, minimally
104. $ rspec .........* 10 examples, 0 failures, 1 pending
105. RSpec.describe Ledger, :db do
 describe '#record' do context 'when the expense lacks a payee' do it 'rejects the expense as invalid' end end end
106. $ rspec FF........* 11 examples, 2 failures, 1 pending
107. Two failures?!?
108. $ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ...
109. $ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ...
110. $ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ...
111. $ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ...
112. $ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ...
113. $ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds.
114. $ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds.
115. $ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds.
116. $ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds.
117. $ rspec --bisect 
 # ... The minimal reproduction command is: rspec ./spec/integration/app/ ledger_spec.rb[1:1:1:1,1:1:2:1] ./spec/unit/app/ api_spec.rb[1:2:2:2]
118. Recap:
119. • Copy/paste to run the spec at a given line • Focus on specific specs:
 fdescribe / fit • Run specs with specific tags:
 -t tag:value • Run just what failed:
 --only-failures, --next-failure • Bisect to track down errors
120. Back to MiniTest
121. You can get some of these benefits with add-ons
122. Run tests by line (with minitest-line gem)
123. 1: class CoffeeTest < Minitest::Test 2: def test_coffee_should_be_awesome; end 3: 4: context 'coffee' do 5: should 'cost $1' {} 6: 7: context 'with milk' do 8: should 'cost $1.25' {} 9: end 10: end 11: end
 
 

124. 1: class CoffeeTest < Minitest::Test 2: def test_coffee_should_be_awesome; end 3: 4: context 'coffee' do 5: should 'cost $1' {} 6: 7: context 'with milk' do 8: should 'cost $1.25' {} 9: end 10: end 11: end
 
 

125. 1: class CoffeeTest < Minitest::Test 2: def test_coffee_should_be_awesome; end 3: 4: context 'coffee' do 5: should 'cost $1' {} 6: 7: context 'with milk' do 8: should 'cost $1.25' {} 9: end 10: end 11: end
 
 
 $ ruby test/unit/coffee_test.rb -l 8
126. Helps especially with failing tests
127. $ ruby test/unit/coffee_test.rb 
 3 tests, 3 assertions, 1 failures, 0 errors, 0 skips Focus on failing tests: ruby test/unit/coffee_test.rb -l 20
128. $ ruby test/unit/coffee_test.rb 
 3 tests, 3 assertions, 1 failures, 0 errors, 0 skips Focus on failing tests: ruby test/unit/coffee_test.rb -l 20
129. Focus on specific tests (with minitest-focus gem)
130. require 'minitest/focus'
 
 class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do
 focus should 'cost $1' {} end
 end
 $ ruby test/unit/coffee_test.rb
131. require 'minitest/focus'
 
 class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do
 focus should 'cost $1' {} end
 end
 $ ruby test/unit/coffee_test.rb
132. Reproduce errors minimally (with minitest-bisect gem)
133. $ minitest_bisect test/**/*_test.rb
134. • You can do a lot with MiniTest plugins • You get a lot for free with RSpec
137. http://pragprog.com/titles/rspec3
 
 Coupon code: Open_Source_Bridge_2017
138. Closing thoughts
139. Don’t worry about getting the perfect automation setup
140. Even a ball of duct tape can save you time
141. Make the time to try this stuff out
142. Automate what you can
143. Faster feedback = qualitative improvement
145. This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.