Little Leaks Sink Your Tests

"The tests pass on my machine." "Wait, it was working a minute ago." "Oh, that test is flaky sometimes." Unpredictable tests are toxic for our productivity. They undermine confidence in our code. They encourage us to wallpaper over the immediate problem, rather than fixing the underlying cause.

In this presentation, we'll talk about a chief cause of flaky tests: leaky global state.

Transcript

1. Little
 LEAKS SINK
 Your Tests Seymour R. Goff
 https://en.wikipedia.org/wiki/Loose_lips_sink_ships
 Public Domain @undees #osb16
2. ambitious
3. world-changing
4. visionary
5. These are words
 that describe other talks at this conference
6. McCall Newland / https://flic.kr/p/8DSZob / CC-BY
7. Can’t spell
 Open Source Bridge
 without U
11. Amy Carter / https://youtu.be/eBrZsVeA28A / CC BY
14. Patrick So / https://youtu.be/GD9EQpPONdE / CC BY
17. “ The tests pass on my machine. ಠ_ಠ
18. “ Wait, it was working a minute ago! ಠ_ಠ
19. “ Oh, that test is flaky sometimes. ಠ_ಠ
20. “ What’s wrong with the continuous integration server? ಠ_ಠ
21. ONE TEST
 AFFECTS
 THE OTHERS
22. GLOBAL STATE LEAKS
23. EXAMPLES?
24. 1. DATABASES 2. TEMP FILES 1. 2.
25. TEMPORARY FILES
26. RSpec.describe 'Dear Diary' do it 'starts empty' do expect(Dir['diary/*.txt']).to be_empty end it 'lets me add deep thoughts' do FileUtils.touch('diary/deep_thoughts.txt') expect(Dir['diary/*.txt']).not_to be_empty end end
27. RSpec.describe 'Dear Diary' do it 'starts empty' do expect(Dir['diary/*.txt']).to be_empty end it 'lets me add deep thoughts' do FileUtils.touch('diary/deep_thoughts.txt') expect(Dir['diary/*.txt']).not_to be_empty end end
28. RSpec.describe 'Dear Diary' do it 'starts empty' do expect(Dir['diary/*.txt']).to be_empty end it 'lets me add deep thoughts' do FileUtils.touch('diary/deep_thoughts.txt') expect(Dir['diary/*.txt']).not_to be_empty end end
29. $ rspec before_spec.rb .. Finished in 0.00942 seconds (files took 1.03 seconds to load) 2 examples, 0 failures
30. $ rspec before_spec.rb .F Failures: 1) Dear Diary starts empty Failure/Error:
 expect(Dir['diary/*.txt']).to be_empty
31. $ rspec before_spec.rb .F Failures: 1) Dear Diary starts empty Failure/Error:
 expect(Dir['diary/*.txt']).to be_empty
32. LEFTOVER FILES!
33. RSpec.describe 'Dear Diary' do before do FileUtils.rm Dir['diary/*.txt'] end # ... end
34. RSpec.describe 'Dear Diary' do before do FileUtils.rm Dir['diary/*.txt'] end # ... end
35. DATABASES
36. RSpec.describe 'Cloud tracker' do context 'on a clear day' do it 'reports that there are no clouds' do expect(DB[:clouds].count).to eq(0) end end # ... end
37. RSpec.describe 'Cloud tracker' do context 'on a clear day' do it 'reports that there are no clouds' do expect(DB[:clouds].count).to eq(0) end end # ... end
38. RSpec.describe 'Cloud tracker' do # ... context 'with a cirrus cloud' do before do DB[:clouds].insert(type: 'cirrus',
 spotted_at: Time.now) end it 'reports the cloud' do expect(DB[:clouds].map(:type)).to eq(['cirrus']) end end end
39. RSpec.describe 'Cloud tracker' do # ... context 'with a cirrus cloud' do before do DB[:clouds].insert(type: 'cirrus',
 spotted_at: Time.now) end it 'reports the cloud' do expect(DB[:clouds].map(:type)).to eq(['cirrus']) end end end
40. RSpec.describe 'Cloud tracker' do # ... context 'with a cirrus cloud' do before do DB[:clouds].insert(type: 'cirrus',
 spotted_at: Time.now) end it 'reports the cloud' do expect(DB[:clouds].map(:type)).to eq(['cirrus']) end end end
41. RSpec.describe 'Cloud tracker' do context 'on a clear day' do # ... end context 'with a cirrus cloud' do # ... end end
42. WE NEED TO START FRESH, RIGHT?
43. RSpec.configure do |config| config.before(:suite) do Sequel.extension :migration Sequel::Migrator.run(DB, 'migrations') DB[:clouds].truncate end end
44. RSpec.configure do |config| config.before(:suite) do Sequel.extension :migration Sequel::Migrator.run(DB, 'migrations') DB[:clouds].truncate end end
45. RSpec.configure do |config| config.before(:suite) do Sequel.extension :migration Sequel::Migrator.run(DB, 'migrations') DB[:clouds].truncate end end
46. RSpec.configure do |config| config.before(:suite) do Sequel.extension :migration Sequel::Migrator.run(DB, 'migrations') DB[:clouds].truncate end end
47. $ rspec before_spec.rb .. Finished in 0.01159 seconds (files took 0.65775 seconds to load) 2 examples, 0 failures
48. $ rspec before_spec.rb .F Failures: 1) Cloud tracker on a clear day reports that there are no clouds Failure/Error:
 expect(DB[:clouds].count).to eq(0) expected: 0 got: 1
49. $ rspec before_spec.rb .F Failures: 1) Cloud tracker on a clear day reports that there are no clouds Failure/Error:
 expect(DB[:clouds].count).to eq(0) expected: 0 got: 1
50. LEFTOVER DATA!
51. RSpec.configure do |config| # ...
 config.around(:example) do |example| DB.transaction(rollback: :always) do
 example.run
 end end end
52. RSpec.configure do |config| # ...
 config.around(:example) do |example| DB.transaction(rollback: :always) do
 example.run
 end end end
53. RSpec.configure do |config| # ...
 config.around(:example) do |example| DB.transaction(rollback: :always) do
 example.run
 end end end
54. DATABASE CLEANER https://github.com/DatabaseCleaner/database_cleaner
55. RSpec.configure do |config| config.before(:suite) do # ... DatabaseCleaner.clean_with :truncation DatabaseCleaner.strategy = :transaction end config.around(:example) do |example| DatabaseCleaner.cleaning { example.run } end end
56. RSpec.configure do |config| config.before(:suite) do # ... DatabaseCleaner.clean_with :truncation DatabaseCleaner.strategy = :transaction end config.around(:example) do |example| DatabaseCleaner.cleaning { example.run } end end
57. RSpec.configure do |config| config.before(:suite) do # ... DatabaseCleaner.clean_with :truncation DatabaseCleaner.strategy = :transaction end config.around(:example) do |example| DatabaseCleaner.cleaning { example.run } end end
58. MORE SUBTLE LEAKS?
59. THE BINARY SEARCH
60. $ ruby a.rb b.rb c.rb d.rb e.rb # => FAIL
61. $ ruby a.rb b.rb c.rb d.rb e.rb # => FAIL
62. $ ruby a.rb b.rb e.rb # => FAIL
63. $ ruby a.rb e.rb # => PASS
64. $ ruby b.rb e.rb # => FAIL
65. WHAT NOW?
66. Alterego
 https://en.wikipedia.org/wiki/Deerstalker
 CC-BY-SA
67. 1.ಠ_ಠ 2.puts() 3.pry
68. AARON PATTERSON
 ON PUTS DEBUGGING http://tenderlovemaking.com/2016/02/05/i-am-a-puts- debuggerer.html
69. NOT COVERED TODAY (BUT WATCH OUT FOR THESE) ➤Global variables / singletons ➤Class variables ➤Mutating constants
70. JASON CLARK
 “DON’T LET YOUR TESTS FLAKE OUT” https://youtu.be/GAAG-Nr7Bf4
71. “ …let us prepare to grapple with the ineffable itself, and see if we may not eff it after all. -Douglas Adams, Dirk Gently’s Holistic Detective Agency
72. MOCK OBJECTS (sometimes)
73. class FootTest < ActiveSupport::TestCase test 'quadruped detection' do Foot.expects(:count).returns(4) assert Foot.belongs_to_quadruped? end test 'biped default' do assert_equal 2, Foot.count end end
74. class FootTest < ActiveSupport::TestCase test 'quadruped detection' do Foot.expects(:count).returns(4) assert Foot.belongs_to_quadruped? end test 'biped default' do assert_equal 2, Foot.count end end
75. class FootTest < ActiveSupport::TestCase test 'quadruped detection' do Foot.expects(:count).returns(4) assert Foot.belongs_to_quadruped? end test 'biped default' do assert_equal 2, Foot.count end end
76. class FootTest < ActiveSupport::TestCase test 'quadruped detection' do Foot.expects(:count).returns(4) assert Foot.belongs_to_quadruped? end test 'biped default' do assert_equal 2, Foot.count end end
77. $ bundle exec rake test test/models/before_test.rb # Running: .. Finished in 0.012140s, 164.7389 runs/s, 164.7389 assertions/s. 2 runs, 2 assertions, 0 failures, 0 errors, 0 skips
78. $ bundle exec rake test test/models/before_test.rb # Running: .E Finished in 0.011270s, 177.4576 runs/s, 88.7288 assertions/s. 1) Error: FootTest#test_biped_default: Mocha::ExpectationError: unexpected invocation: 
 Foot(...).count() unsatisfied expectations: - expected exactly once, invoked twice:
 Foot(...).count(any_parameters)
79. $ bundle exec rake test test/models/before_test.rb # Running: .E Finished in 0.011270s, 177.4576 runs/s, 88.7288 assertions/s. 1) Error: FootTest#test_biped_default: Mocha::ExpectationError: unexpected invocation: 
 Foot(...).count() unsatisfied expectations: - expected exactly once, invoked twice:
 Foot(...).count(any_parameters)
80. MOCKS NOT
 CLEANED UP!
81. class FootTest < ActiveSupport::TestCase include Mocha::Hooks teardown do Foot.unstub(:count) end 
 # ... end
82. class FootTest < ActiveSupport::TestCase include Mocha::Hooks teardown do Foot.unstub(:count) end 
 # ... end
83. class FootTest < ActiveSupport::TestCase include Mocha::Hooks teardown do Foot.unstub(:count) end 
 # ... end
84. class FootTest < ActiveSupport::TestCase include Mocha::Hooks teardown do mocha_teardown end 
 # ... end
85. FAKE SERVICES
86. RSpec.describe 'The information superhighway' do before do FakeWeb.register_uri( :get, 'http://google.com', body: 'Welcome online!') end it 'gets me online' do response = Net::HTTP.get('google.com', '/') expect(response).to eq('Welcome online!') end end
87. RSpec.describe 'The information superhighway' do before do FakeWeb.register_uri( :get, 'http://google.com', body: 'Welcome online!') end it 'gets me online' do response = Net::HTTP.get('google.com', '/') expect(response).to eq('Welcome online!') end end
88. RSpec.describe 'The information superhighway' do before do FakeWeb.register_uri( :get, 'http://google.com', body: 'Welcome online!') end it 'gets me online' do response = Net::HTTP.get('google.com', '/') expect(response).to eq('Welcome online!') end end
89. RSpec.describe 'The information superhighway' do before do FakeWeb.register_uri( :get, 'http://google.com', body: 'Welcome online!') end it 'gets me online' do response = Net::HTTP.get('google.com', '/') expect(response).to eq('Welcome online!') end end
90. RSpec.describe 'The information superhighway' do before do FakeWeb.register_uri( :get, 'http://google.com', body: 'Welcome online!') end it 'gets me online' do response = Net::HTTP.get('google.com', '/') expect(response).to eq('Welcome online!') end end
91. RSpec.describe 'My awesome browser' do it 'handles redirects' do response = Net::HTTP.get('google.com', '/') expect(response).to include('Moved') end end
92. RSpec.describe 'My awesome browser' do it 'handles redirects' do response = Net::HTTP.get('google.com', '/') expect(response).to include('Moved') end end
93. RSpec.describe 'My awesome browser' do it 'handles redirects' do response = Net::HTTP.get('google.com', '/') expect(response).to include('Moved') end end
94. $ rspec before_spec.rb .. Finished in 1.01 seconds (files took 0.21161 seconds to load) 2 examples, 0 failures
95. $ rspec before_spec.rb .F Failures: 1) My awesome browser handles redirects Failure/Error: expect(response).to include('Moved') expected "Welcome online!" to include "Moved"
96. $ rspec before_spec.rb .F Failures: 1) My awesome browser handles redirects Failure/Error: expect(response).to include('Moved') expected "Welcome online!" to include "Moved"
97. STILL USING THE
 FAKE SERVICE!
98. RSpec.describe 'The information superhighway' do # ...
 after do FakeWeb.clean_registry end # ... end
99. RSpec.describe 'The information superhighway' do # ...
 after do FakeWeb.clean_registry end # ... end
100. CLASSES ಠ_ಠ
101. class FavoriteThing class Raindrops < FavoriteThing; end class Whiskers < FavoriteThing; end ALL = [ Raindrops.new, Whiskers.new ] def better_than_a_bee_sting? true end end
102. class FavoriteThing class Raindrops < FavoriteThing; end class Whiskers < FavoriteThing; end ALL = [ Raindrops.new, Whiskers.new ] def better_than_a_bee_sting? true end end
103. class FavoriteThing class Raindrops < FavoriteThing; end class Whiskers < FavoriteThing; end ALL = [ Raindrops.new, Whiskers.new ] def better_than_a_bee_sting? true end end
104. class FavoriteThing class Raindrops < FavoriteThing; end class Whiskers < FavoriteThing; end ALL = [ Raindrops.new, Whiskers.new ] def better_than_a_bee_sting? true end end
105. class FavoriteThing class Raindrops < FavoriteThing; end class Whiskers < FavoriteThing; end ALL = [ Raindrops.new, Whiskers.new ] def better_than_a_bee_sting? true end end
106. RSpec.describe FavoriteThing do it 'always ends in S for some reason' do ObjectSpace.each_object(FavoriteThing) do
 |thing|
 last_letter = thing.class.to_s[-1] expect(last_letter).to eq('s') end end 
 # ... end
107. RSpec.describe FavoriteThing do it 'always ends in S for some reason' do ObjectSpace.each_object(FavoriteThing) do
 |thing|
 last_letter = thing.class.to_s[-1] expect(last_letter).to eq('s') end end 
 # ... end
108. RSpec.describe FavoriteThing do it 'always ends in S for some reason' do ObjectSpace.each_object(FavoriteThing) do
 |thing|
 last_letter = thing.class.to_s[-1] expect(last_letter).to eq('s') end end 
 # ... end
109. RSpec.describe FavoriteThing do it 'always ends in S for some reason' do ObjectSpace.each_object(FavoriteThing) do
 |thing|
 last_letter = thing.class.to_s[-1] expect(last_letter).to eq('s') end end 
 # ... end
110. RSpec.describe FavoriteThing do it 'always ends in S for some reason' do ObjectSpace.each_object(FavoriteThing) do
 |thing|
 last_letter = thing.class.to_s[-1] expect(last_letter).to eq('s') end end 
 # ... end
111. ಠ_ಠ
112. RSpec.describe FavoriteThing do # ... it 'helps me get over bee stings' do Wine = Class.new(FavoriteThing) expect(Wine.new).to be_better_than_a_bee_sting end end
113. RSpec.describe FavoriteThing do # ... it 'helps me get over bee stings' do Wine = Class.new(FavoriteThing) expect(Wine.new).to be_better_than_a_bee_sting end end
114. RSpec.describe FavoriteThing do # ... it 'helps me get over bee stings' do Wine = Class.new(FavoriteThing) expect(Wine.new).to be_better_than_a_bee_sting end end
115. $ rspec before_spec.rb .. Finished in 0.00803 seconds (files took 0.22942 seconds to load) 2 examples, 0 failures
116. $ rspec before_spec.rb .F Failures: 1) FavoriteThing always ends in S for some reason Failure/Error:
 expect(last_letter).to eq('s') expected: "s" got: "e"
117. $ rspec before_spec.rb .F Failures: 1) FavoriteThing always ends in S for some reason Failure/Error:
 expect(last_letter).to eq('s') expected: "s" got: "e"
118. after do GC.start end
119. after do # HAHA srsly no, just delete the test GC.start end
120. RSpec.describe FavoriteThing do it 'always ends in S for some reason' do ObjectSpace.each_object(FavoriteThing) do
 |thing|
 last_letter = thing.class.to_s[-1] expect(last_letter).to eq('s') end end 
 # ... end
121. MONKEY- PATCHES
122. guerrilla patching gorilla monkey-patching duck punching runtime changes to classes
123. module DoItLive class << self def after_delay(delay, &block) sleep delay block.call end end end
124. module DoItLive class << self def after_delay(delay, &block) sleep delay block.call end end end
125. module DoItLive class << self def after_delay(delay, &block) sleep delay block.call end end end
126. module DoItLive class << self def after_delay(delay, &block) sleep delay block.call end end end
127. DoItLive.after_delay(7) do puts "**** it, we'll do it live!" end
128. DoItLive.after_delay(7) do puts "**** it, we'll do it live!" end # 7-second delay to work the bleep button, # then prints the message
129. RSpec.describe DoItLive do it 'waits before acting' do expect(DoItLive).to receive(:sleep).with(5) DoItLive.after_delay(5) { 'WHOA' } end # ... end
130. RSpec.describe DoItLive do it 'waits before acting' do expect(DoItLive).to receive(:sleep).with(5) DoItLive.after_delay(5) { 'WHOA' } end # ... end
131. alias_method :old_after_delay, :after_delay def after_delay(delay, &block)
 # no sleep! block.call end
132. alias_method :old_after_delay, :after_delay def after_delay(delay, &block)
 # no sleep! block.call end
133. alias_method :old_after_delay, :after_delay def after_delay(delay, &block)
 # no sleep! block.call end
134. def become_impatient! DoItLive.module_eval do class << self alias_method :old_after_delay, :after_delay def after_delay(delay, &block)
 # no sleep! block.call end end end end
135. alias_method :after_delay, :old_after_delay undef :old_after_delay
136. alias_method :after_delay, :old_after_delay undef :old_after_delay
137. alias_method :after_delay, :old_after_delay undef :old_after_delay
138. def chillax! DoItLive.module_eval do class << self alias_method :after_delay, :old_after_delay undef :old_after_delay end end end
139. RSpec.describe DoItLive do
 # ... it 'preserves the return value' do become_impatient! result = DoItLive.after_delay(5) { 'ALL DONE' } expect(result).to eq('ALL DONE')
 chillax! end end
140. RSpec.describe DoItLive do
 # ... it 'preserves the return value' do become_impatient! result = DoItLive.after_delay(5) { 'ALL DONE' } expect(result).to eq('ALL DONE')
 chillax! end end
141. RSpec.describe DoItLive do
 # ... it 'preserves the return value' do become_impatient! result = DoItLive.after_delay(5) { 'ALL DONE' } expect(result).to eq('ALL DONE')
 chillax! end end
142. RSpec.describe DoItLive do
 # ... it 'preserves the return value' do become_impatient! result = DoItLive.after_delay(5) { 'ALL DONE' } expect(result).to eq('ALL DONE')
 # HAHA KIDDING OF COURSE WE FORGOT TO CHILLAX end end
143. $ rspec before_spec.rb .. Finished in 0.00848 seconds (files took 0.1619 seconds to load) 2 examples, 0 failures
144. $ rspec before_spec.rb .F Failures: 1) DoItLive waits before acting Failure/Error: expect(DoItLive).to receive(:sleep).with(5) (DoItLive).sleep(5) expected: 1 time with arguments: (5) received: 0 times
145. $ rspec before_spec.rb .F Failures: 1) DoItLive waits before acting Failure/Error: expect(DoItLive).to receive(:sleep).with(5) (DoItLive).sleep(5) expected: 1 time with arguments: (5) received: 0 times
146. FORGOTTEN
 MONKEY-PATCH
 CLEANUP
147. RSpec.describe DoItLive do # ... context 'unit spec', :monkey_patches do it 'preserves the return value' do result = DoItLive.after_delay(1) { 'ALL DONE' } expect(result).to eq('ALL DONE') end end end
148. RSpec.describe DoItLive do # ... context 'unit spec', :monkey_patches do it 'preserves the return value' do result = DoItLive.after_delay(1) { 'ALL DONE' } expect(result).to eq('ALL DONE') end end end
149. RSpec.describe DoItLive do # ... context 'unit spec', :monkey_patches do it 'preserves the return value' do result = DoItLive.after_delay(1) { 'ALL DONE' } expect(result).to eq('ALL DONE') end end end
150. RSpec.configure do |config| config.before(:example, :monkey_patches) do MonkeyPatches.become_impatient! end config.after(:example, :monkey_patches) do MonkeyPatches.chillax! end end
151. RSpec.configure do |config| config.before(:example, :monkey_patches) do MonkeyPatches.become_impatient! end config.after(:example, :monkey_patches) do MonkeyPatches.chillax! end end
152. RSpec.configure do |config| config.before(:example, :monkey_patches) do MonkeyPatches.become_impatient! end config.after(:example, :monkey_patches) do MonkeyPatches.chillax! end end
153. RSpec.configure do |config| config.before(:example, :monkey_patches) do MonkeyPatches.become_impatient! end config.after(:example, :monkey_patches) do MonkeyPatches.chillax! end end
154. WHAT CAN YA DO?
155. RANDOMIZE YOUR
 TEST ORDER
156. BREAK OUT YOUR DETECTIVE SKILLS
157. INVENTORY YOUR
 TEST HELPERS
158. USE HOOKS TO DO YOUR WORK FOR YOU
159. MAKE ERRORS
 INEXPRESSIBLE
160. STAY ON TARGET
161. SLIDES http://opensourcebridge.org/sessions/1851
162. This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.