Don't Let Your Tests Flake Out

The build’s red with a test failure. You re-run the tests and suddenly all is well. What’s going on?

While Ruby makes it easy to start testing your code, it’s also easy to write tests that fail in subtle, unexpected ways. Luckily, flaky tests share common causes, and there are ways to flush them out.

From order dependencies to static state, time comparisons to threading bugs, we’ll see what’s making your test suite unstable and how to get it rock solid again.

Transcript

1. Don't Let Your Tests Flake Out Jason Clark @jasonrclark Tuesday, June 24, 14
2. TROUBLES Tuesday, June 24, 14
3. TROUBLES TECHNIQUES Tuesday, June 24, 14
4. TROUBLES TECHNIQUES TOOLS Tuesday, June 24, 14
5. 3 seattlerb/minitest https://github.com/seattlerb/minitest Tuesday, June 24, 14
6. TROUBLES 4 Tuesday, June 24, 14
7. State 5 Tuesday, June 24, 14
8. Globals 6 $evil = true Tuesday, June 24, 14
9. 7 include Singleton Globals Tuesday, June 24, 14
10. Globals 8 def test_isnt_evil refute $evil end Tuesday, June 24, 14
11. Globals 9 def test_isnt_evil refute $evil end def test_if_we_were_evil $evil = true assert $evil end Tuesday, June 24, 14
12. Globals 10 def test_isnt_evil refute $evil end def test_if_we_were_evil $evil = true assert $evil end false Tuesday, June 24, 14
13. Globals 11 def test_isnt_evil refute $evil end def test_if_we_were_evil $evil = true assert $evil end true Tuesday, June 24, 14
14. Globals 12 def test_if_we_were_evil $evil = true assert $evil end def test_isnt_evil refute $evil end true Tuesday, June 24, 14
15. Globals 13 def test_if_we_were_evil $evil = true assert $evil end def test_isnt_evil refute $evil end true! Tuesday, June 24, 14
16. 14 https://flic.kr/p/97nB5F Tuesday, June 24, 14
17. 15 SPECIAL = 1 "Constants" Tuesday, June 24, 14
18. 16 SPECIAL = [1,2,3] "Constants" Tuesday, June 24, 14
19. 17 SPECIAL = [1,2,3] SPECIAL << "Ha" "Constants" Tuesday, June 24, 14
20. 18 https://flic.kr/p/5vKLwQ Tuesday, June 24, 14
21. 19 class ClassyTest < Minitest::Test @@classy = true def test_classy assert @@classy end end Class Variables Tuesday, June 24, 14
22. 20 Class Variables class ClassyTest < Minitest::Test # ... def test_not_classy @@classy = false refute @@classy end end Tuesday, June 24, 14
23. 21 https://flic.kr/p/4EZada Tuesday, June 24, 14
24. 22 AS = ActiveSupport AS::Notifications.subscribe("event") do testing.is_awesome? end Events and Callbacks Tuesday, June 24, 14
25. 23 https://flic.kr/p/4Mu5NX Tuesday, June 24, 14
26. 24 class Object def boo puts "Boo" end end Metaprogramming Tuesday, June 24, 14
27. 24 class Object def boo puts "Boo" end end class Object remove_method(:boo) end Metaprogramming Tuesday, June 24, 14
28. 24 class Object def boo puts "Boo" end end class Object remove_method(:boo) end class Object alias :hoo :boo end Metaprogramming Tuesday, June 24, 14
29. 25 require 'code' require Tuesday, June 24, 14
30. 26 https://flic.kr/p/4YwYM8 Tuesday, June 24, 14
31. 27 Threading class ThreadsAreCoolTest < Minitest::Test def test_coolness Thread.new do loop { $cool = true } end assert $cool end end Tuesday, June 24, 14
32. 28 https://flic.kr/p/B72L5 Tuesday, June 24, 14
33. 29 Databases Tuesday, June 24, 14
34. 29 Queues Databases Tuesday, June 24, 14
35. 29 Files Queues Databases Tuesday, June 24, 14
36. 29 Files Queues Databases Services Tuesday, June 24, 14
37. Comparison 30 Tuesday, June 24, 14
38. Floats 31 def test_28 assert_equal 28, (0.28 * 100) end Tuesday, June 24, 14
39. Floats 32 1) Failure: test_28(Test_28) [/Users/jclark/test_28.rb]: <28> expected but was <28.000000000000004>. Tuesday, June 24, 14
40. Floats 33 def test_28 assert_in_delta(28, 0.28 * 100, 0.00001) end Tuesday, June 24, 14
41. Time Roundtripping 34 def test_time time = Time.now.to_f round_trip = Time.at(time).to_f assert_equal(time, round_trip) end Tuesday, June 24, 14
42. 35 https://flic.kr/p/62n3eY Tuesday, June 24, 14
43. 36 https://twitter.com/joshsusser/status/257725572275376128 Tuesday, June 24, 14
44. https://flic.kr/p/bKnFx https://flic.kr/p/65uiJH Tuesday, June 24, 14
45. 38 https://flic.kr/p/btvCeB Tuesday, June 24, 14
46. TECHNIQUE 39 Tuesday, June 24, 14
47. Avoid 40 Tuesday, June 24, 14
48. Instances are Friends 41 def test_newly_created subject = Testing.new assert subject.fresh? end Tuesday, June 24, 14
49. Reset Consistently 42 def setup_and_teardown_agent(opts = {}, &block) define_method(:setup) do setup_agent(opts, &block) end define_method(:teardown) do teardown_agent end end Tuesday, June 24, 14
50. Reset Consistently 43 class AgentTest < Minitest::Test setup_and_teardown_agent # ... end Tuesday, June 24, 14
51. https://github.com/travisjeffery/timecop travisjeffery/timecop 44 describe "some set of tests to mock" do before do Timecop.freeze(Time.local(1990)) end after do Timecop.return end it "should do blah blah blah" {} end Tuesday, June 24, 14
52. 45 rails/rails https://github.com/rails/rails Hat tip to http://brandonhilkert.com/blog/rails-4-1-travel-to-test-helper/ travel_to Time.new(1999, 12, 31, 11, 59, 59) do # Test like it's 1999! end Tuesday, June 24, 14
53. Time 46 def freeze_time(now=Time.now) Time.stubs(:now).returns(now) now end def advance_time(seconds) freeze_time(Time.now + seconds) end Tuesday, June 24, 14
54. Metaprogramming 47 class TransmogifierTest < Minitest::Test def test_modifies_class clazz = Class.new Transmogifier.go!(clazz) assert clazz.new.tentacles? end end Tuesday, June 24, 14
55. Metaprogramming 48 https://github.com/thoughtbot/appraisal Tuesday, June 24, 14
56. Metaprogramming 48 https://github.com/thoughtbot/appraisal https://github.com/sinisterchipmunk/testbeds Tuesday, June 24, 14
57. Metaprogramming 48 https://github.com/newrelic/rpm/blob/master/test/multiverse/README.md https://github.com/thoughtbot/appraisal https://github.com/sinisterchipmunk/testbeds Tuesday, June 24, 14
58. Locate 49 Tuesday, June 24, 14
59. 50 Tuesday, June 24, 14
60. 51 puts "JRC: GOT HERE" Always Be Logging Tuesday, June 24, 14
61. 52 caller.join("\n\t") Where Am I? Tuesday, June 24, 14
62. 53 TESTOPTS="--seed=33023" bundle exec rake ruby test/awesome/code_test.rb --seed 33023 Seeds of !Change Tuesday, June 24, 14
63. 54 Tuesday, June 24, 14
64. 55 class OrderedTest < Minitest::Test i_suck_and_my_tests_are_order_dependent! def test_feel_bad_but_carry_on # ... end end Method Order Tuesday, June 24, 14
65. 56 class SingleFailingTest < Minitest::Test def test_i_sometimes_fail flaky_call() end end Loop Tuesday, June 24, 14
66. 57 class SingleFailingTest < Minitest::Test def test_i_sometimes_fail 100.times do setup flaky_call() teardown end end end Loop Tuesday, June 24, 14
67. 58 while true do bundle exec rake test || break done Loop Tuesday, June 24, 14
68. Differences on the Build Box 59 OS Hardware Performance Paths, files, directories Gem versions Tuesday, June 24, 14
69. Threading 60 class MultiThreadedWorker def go! Thread.new do work() end end end Tuesday, June 24, 14
70. Threading 61 class MultiThreadedWorker def go! Thread.new do # TODO: ONLY FOR DIAGNOSISING!!!!!!! sleep(0.1) work() end end end Tuesday, June 24, 14
71. Threading 62 class MultiThreadedWorker def go! 50.times do Thread.new do work() end end end end Tuesday, June 24, 14
72. Threading 63 class MultiThreadedWorker attr_reader :thread def go! @thread = Thread.new do # ... end end end Tuesday, June 24, 14
73. Threading 64 class MultiThreadedWorkerTest < Minitest::Test def test_worker worker = MultiThreadedWorker.new worker.go! worker.thread.join end end Tuesday, June 24, 14
74. TOOLS 65 Tuesday, June 24, 14
75. 66 jasonrclark/flake_test https://github.com/jasonrclark/flake_test Tuesday, June 24, 14
76. 67 gnarg/test_bisect https://github.com/gnarg/test_bisect Tuesday, June 24, 14
77. gnarg/test_bisect https://github.com/gnarg/test_bisect 68 # Gemfile gem 'test_bisect' # Rakefile require 'test_bisect' TestBisect::BisectTask.new Tuesday, June 24, 14
78. gnarg/test_bisect https://github.com/gnarg/test_bisect 69 class Test_01 < Test::Unit::TestCase def test_truth refute $globals_are_awesome end end Tuesday, June 24, 14
79. 70 Tuesday, June 24, 14
80. gnarg/test_bisect https://github.com/gnarg/test_bisect 71 class Test_06 < Test::Unit::TestCase def test_truth refute $globals_are_awesome end end Tuesday, June 24, 14
81. 72 Tuesday, June 24, 14
82. 73 Tuesday, June 24, 14
83. 74 Tuesday, June 24, 14
84. gnarg/test_bisect https://github.com/gnarg/test_bisect 75 class Test_05 < Test::Unit::TestCase def test_filthy_lies $globals_are_awesome = true assert $globals_are_awesome end end Tuesday, June 24, 14
85. gnarg/test_bisect https://github.com/gnarg/test_bisect 76 Test::Unit :( Tuesday, June 24, 14
86. 77 jasonrclark/hometown https://github.com/jasonrclark/hometown Tuesday, June 24, 14
87. jasonrclark/hometown https://github.com/jasonrclark/hometown 78 # Gemfile gem 'hometown' # test_helper.rb Hometown.watch(::Thread) Tuesday, June 24, 14
88. jasonrclark/hometown https://github.com/jasonrclark/hometown 79 class NoticeThreadTest < Minitest::Test def setup Thread.new { work! } end def test_notice_thread Thread.list.each do |thread| trace = Hometown.for(thread) p trace unless trace.nil? end end end Tuesday, June 24, 14
89. jasonrclark/hometown https://github.com/jasonrclark/hometown 80 #<Hometown::Trace:0x007fbbfb63edb8 @traced_class=Thread, @backtrace=[ "...loose_thread_test.rb:5:in `test_loose_...'", "...minitest-5.3.4/lib/minitest/test.rb:106", "...minitest-5.3.4/lib/minitest/test.rb:204", "...minitest-5.3.4/lib/minitest/test.rb:103" # ... > Tuesday, June 24, 14
90. jasonrclark/minitest-stately https://github.com/jasonrclark/minitest-stately 81 Tuesday, June 24, 14
91. jasonrclark/minitest-stately https://github.com/jasonrclark/minitest-stately 82 # Gemfile gem 'minitest-stately' # test_helper.rb # Watch for freshly started threads Minitest::Stately.watch("thread count") do Thread.list.count end Tuesday, June 24, 14
92. 83 Tuesday, June 24, 14
93. jasonrclark/minitest-stately https://github.com/jasonrclark/minitest-stately 84 Minitest::Stately.run do $silly_global_state = nil end Tuesday, June 24, 14
94. jasonrclark/minitest-stately https://github.com/jasonrclark/minitest-stately 85 Minitest::Stately.fail_if do $silly_global_state == "seriously" end Tuesday, June 24, 14
95. jasonrclark/minitest-stately https://github.com/jasonrclark/minitest-stately 86 Minitest 5.x :( Tuesday, June 24, 14
96. TROUBLES 87 Tuesday, June 24, 14
97. TECHNIQUE 88 Tuesday, June 24, 14
98. TOOLS 89 Tuesday, June 24, 14
99. 90 Jason Clark @jasonrclark Tuesday, June 24, 14