Testing Just
 What You Need @undees Open Source Bridge June 22, 2017 

No content

Public domain: https://commons.wikimedia.org/wiki/File:Vancouver_Tornado_-_RTX_loop.gif 

Testing Just
 What You Need 

The problem 

Tests are slow 

Test output is huge 

It’s tough to find errors 

The real problem 

Feedback cycle is too slow 

The solution: 

Run just what you need 

No content

Let’s look at MiniTest first 

Use the files, Luke! 

test └── unit ├── coffee_test.rb └── test_helper.rb
 
 
 
 
 

test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ ruby -I.:./lib:./test test/unit/coffee_test.rb 

test └── unit ├── coffee_test.rb └── test_helper.rb
 $ export RUBYOPT=-I.:./lib:./test 
 $ ruby -I.:./lib:./test test/unit/coffee_test.rb 

test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ export RUBYOPT=-I.:./lib:./test 
 $ ruby test/unit/coffee_test.rb 

test └── unit ├── coffee_test.rb └── test_helper.rb
 
 
 $ ruby test/unit/coffee_test.rb 

test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ ruby test/unit/*_test.rb 

test └── unit ├── coffee_test.rb └── test_helper.rb
 
 $ ruby test/**/*_test.rb
 

• Organize tests by directory • Wildcards (globs) are your friend 

Run tests by name 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end # ... end 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end # Using macros from `shoulda` context 'coffee' do should 'cost $1' {} end end 

Exact match 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 
 $ ruby test/unit/coffee_test.rb -n test_coffee_should_be_awesome 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 
 $ ruby test/unit/coffee_test.rb -n test_coffee_should_be_awesome 

What about
 generated test names? 

Shoulda 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'coffee should cost $1' 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'coffee should cost $1' 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'coffee should cost $1' HAHA NOPE 


 class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} end end
 $ ruby test/unit/coffee_test.rb -n 'test_: coffee should cost $1. '
 ^ 

MiniTest::Spec 


 describe Coffee do it 'costs $1' {} end
 $ ruby test/unit/coffee_spec.rb -n 'test_0001_costs $1' 

Regular expression 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} context 'with milk' do should 'cost $1.25' {} end end end
 $ ruby test/unit/coffee_test.rb -n '/coffee/' 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} context 'with milk' do should 'cost $1.25' {} end end end
 $ ruby test/unit/coffee_test.rb -n '/coffee/' 

class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do should 'cost $1' {} context 'with milk' do should 'cost $1.25' {} end end end
 $ ruby test/unit/coffee_test.rb -n '/milk/' 

• Pass the -n option to run all matching tests • Use a regular expression inside single quotes:
 '/test case to run/' • Be mindful of names generated by Shoulda,
 MiniTest::Spec, etc. 

Integrate with your editor 

No content

Whoa 

You can go so far
 with what we’ve covered so far 

But now… 

RSpec 

You can do all the stuff we just did! 

spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 

spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 $ rspec spec/unit/coffee_spec.rb 

spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 $ rspec spec/unit/*_spec.rb 

spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 
 $ rspec spec/**/*_spec.rb 

spec ├── spec_helper.rb └── unit └── coffee_spec.rb
 $ rspec 

• You can still separate specs by directory • RSpec can run a directory for you • Wildcards are still your friend • Run specs from your text editor 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 
 $ rspec -e 'Coffee costs $1' 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 
 $ rspec -e 'Coffee costs $1' 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 
 $ rspec -e 'milk' 

• Pass the -e option to run all matching specs • No need for regex-style slashes; just use:
 'test case to run' 

Failing line number is right there in the output! 

$ rspec
 
 A cup of coffee costs $1 with milk probably tastes a little milder costs $1.25 (FAILED - 1)
 3 examples, 1 failure Failed examples: rspec ./spec/unit/coffee_spec.rb:13
 # A cup of coffee with milk costs $1.25 

$ rspec
 
 A cup of coffee costs $1 with milk probably tastes a little milder costs $1.25 (FAILED - 1)
 3 examples, 1 failure Failed examples: rspec ./spec/unit/coffee_spec.rb:13
 # A cup of coffee with milk costs $1.25 

Paste and run 

$ rspec ./spec/unit/coffee_spec.rb:13
 
 A cup of coffee with milk costs $1.25 (FAILED - 1) 
 
 1 example, 1 failure
 
 Failed examples: rspec ./spec/unit/coffee_spec.rb:15
 # A cup of coffee with milk costs $1.25 

$ rspec ./spec/unit/coffee_spec.rb:13
 
 A cup of coffee with milk costs $1.25 (FAILED - 1) 
 
 1 example, 1 failure
 
 Failed examples: rspec ./spec/unit/coffee_spec.rb:15
 # A cup of coffee with milk costs $1.25 

$ rspec ./spec/unit/coffee_spec.rb:13
 
 A cup of coffee with milk costs $1.25 (FAILED - 1) 
 
 1 example, 1 failure
 
 Failed examples: rspec ./spec/unit/coffee_spec.rb:15
 # A cup of coffee with milk costs $1.25 

• Run the spec on a specific line number:
 some_spec.rb:42 • Paste and run from test failure output 

RSpec offers even more shortcuts
 for the studiously lazy 

Focus on specific specs 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 

$ rspec A cup of coffee costs $1 with milk probably tastes a little milder costs $1.25 3 examples, 0 failures 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' fit 'probably tastes a little milder' end
 end 

$ rspec . 1 example, 0 failures 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 

RSpec.describe 'Coffee' do it 'costs $1' fcontext 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 

RSpec.fdescribe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 

A little configuration
 (automatic if you use rspec --init) 

# spec_helper.rb RSpec.configure do |config| config.filter_run_when_matching :focus end 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' fit 'probably tastes a little milder' end
 end 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder', focus: true end
 end 

Tag your examples! 

RSpec.describe 'Coffee' do it 'costs $1' context 'with milk' do it 'costs $1.25' it 'probably tastes a little milder' end
 end 

RSpec.describe 'Coffee' do it 'costs $1', :payment context 'with milk' do it 'costs $1.25', :payment it 'probably tastes a little milder' end
 end 

RSpec.describe 'Coffee' do it 'costs $1', :payment context 'with milk' do it 'costs $1.25', :payment it 'probably tastes a little milder' end
 end 

# Run just the payment specs $ rspec -t payment
 
 # Run just the non-payment specs $ rspec -t ~payment 

# Run just the specs related to one bug $ rspec -t bug_id:123
 

Run just what failed 

$ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures 

$ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures 

$ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures 

$ rspec 
 
 
 A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2) probably tastes a little milder
 
 3 examples, 2 failures 

$ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures 

$ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures 

$ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures 

$ rspec --only-failures
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) with milk costs $1.25 (FAILED - 2)
 
 2 examples, 2 failures 

…or just one failure at a time 

$ rspec --next-failure
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) 1 example, 1 failure 

$ rspec --next-failure
 Run options: include {:last_run_status=>"failed"} A cup of coffee costs $1 (FAILED - 1) 1 example, 1 failure 

How to set this up: 

# spec_helper.rb RSpec.configure do |config| config.example_status_persistence_file_path =
 'spec/examples.txt' end 

No content

Reproduce errors, minimally 

$ rspec .........* 10 examples, 0 failures, 1 pending 

RSpec.describe Ledger, :db do
 describe '#record' do context 'when the expense lacks a payee' do it 'rejects the expense as invalid' end end end 

$ rspec FF........* 11 examples, 2 failures, 1 pending 

Two failures?!? 

$ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ... 

$ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ... 

$ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ... 

$ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ... 

$ rspec --bisect
 Bisect started using options: "" Running suite to find failures... (0.56287 seconds) Starting bisect with 2 failing examples
 and 9 non-failing examples. Checking that failure(s) are order-dependent...
 failure appears to be order-dependent 
 # ... 

$ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds. 

$ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds. 

$ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds. 

$ rspec --bisect 
 # ... Round 1: bisecting over non-failing examples 1-9 . ignoring examples 1-5 (0.45315 seconds) Round 2: bisecting over non-failing examples 6-9 .. ignoring examples 8-9 (0.94028 seconds) Round 3: bisecting over non-failing examples 6-7 . ignoring example 6 (0.5078 seconds) Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 2.31 seconds. 

$ rspec --bisect 
 # ... The minimal reproduction command is: rspec ./spec/integration/app/ ledger_spec.rb[1:1:1:1,1:1:2:1] ./spec/unit/app/ api_spec.rb[1:2:2:2] 

Recap: 

• Copy/paste to run the spec at a given line • Focus on specific specs:
 fdescribe / fit • Run specs with specific tags:
 -t tag:value • Run just what failed:
 --only-failures, --next-failure • Bisect to track down errors 

Back to MiniTest 

You can get some of these benefits with add-ons 

Run tests by line (with minitest-line gem) 

1: class CoffeeTest < Minitest::Test 2: def test_coffee_should_be_awesome; end 3: 4: context 'coffee' do 5: should 'cost $1' {} 6: 7: context 'with milk' do 8: should 'cost $1.25' {} 9: end 10: end 11: end
 
 
 

1: class CoffeeTest < Minitest::Test 2: def test_coffee_should_be_awesome; end 3: 4: context 'coffee' do 5: should 'cost $1' {} 6: 7: context 'with milk' do 8: should 'cost $1.25' {} 9: end 10: end 11: end
 
 
 

1: class CoffeeTest < Minitest::Test 2: def test_coffee_should_be_awesome; end 3: 4: context 'coffee' do 5: should 'cost $1' {} 6: 7: context 'with milk' do 8: should 'cost $1.25' {} 9: end 10: end 11: end
 
 
 $ ruby test/unit/coffee_test.rb -l 8 

Helps especially with failing tests 

$ ruby test/unit/coffee_test.rb 
 3 tests, 3 assertions, 1 failures, 0 errors, 0 skips Focus on failing tests: ruby test/unit/coffee_test.rb -l 20 

$ ruby test/unit/coffee_test.rb 
 3 tests, 3 assertions, 1 failures, 0 errors, 0 skips Focus on failing tests: ruby test/unit/coffee_test.rb -l 20 

Focus on specific tests (with minitest-focus gem) 

require 'minitest/focus'
 
 class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do
 focus should 'cost $1' {} end
 end
 $ ruby test/unit/coffee_test.rb 

require 'minitest/focus'
 
 class CoffeeTest < Minitest::Test def test_coffee_should_be_awesome; end context 'coffee' do
 focus should 'cost $1' {} end
 end
 $ ruby test/unit/coffee_test.rb 

Reproduce errors minimally (with minitest-bisect gem) 

$ minitest_bisect test/**/*_test.rb 

• You can do a lot with MiniTest plugins • You get a lot for free with RSpec 

No content

No content

http://pragprog.com/titles/rspec3
 
 Coupon code: Open_Source_Bridge_2017 

Closing thoughts 

Don’t worry about getting the perfect automation setup 

Even a ball of duct tape can save you time 

Make the time to try this stuff out 

Automate what you can 

Faster feedback = qualitative improvement 

No content

This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.