$30 off During Our Annual Pro Sale. View Details »

Don't Let Your Tests Flake Out

Don't Let Your Tests Flake Out

The build’s red with a test failure. You re-run the tests and suddenly all is well. What’s going on?

While Ruby makes it easy to start testing your code, it’s also easy to write tests that fail in subtle, unexpected ways. Luckily, flaky tests share common causes, and there are ways to flush them out.

From order dependencies to static state, time comparisons to threading bugs, we’ll see what’s making your test suite unstable and how to get it rock solid again.

Jason R Clark

June 24, 2014
Tweet

More Decks by Jason R Clark

Other Decks in Technology

Transcript

  1. Don't Let Your
    Tests Flake Out
    Jason Clark
    @jasonrclark
    Tuesday, June 24, 14

    View Slide

  2. TROUBLES
    Tuesday, June 24, 14

    View Slide

  3. TROUBLES
    TECHNIQUES
    Tuesday, June 24, 14

    View Slide

  4. TROUBLES
    TECHNIQUES
    TOOLS
    Tuesday, June 24, 14

    View Slide

  5. 3
    seattlerb/minitest
    https://github.com/seattlerb/minitest
    Tuesday, June 24, 14

    View Slide

  6. TROUBLES
    4
    Tuesday, June 24, 14

    View Slide

  7. State
    5
    Tuesday, June 24, 14

    View Slide

  8. Globals
    6
    $evil = true
    Tuesday, June 24, 14

    View Slide

  9. 7
    include Singleton
    Globals
    Tuesday, June 24, 14

    View Slide

  10. Globals
    8
    def test_isnt_evil
    refute $evil
    end
    Tuesday, June 24, 14

    View Slide

  11. Globals
    9
    def test_isnt_evil
    refute $evil
    end
    def test_if_we_were_evil
    $evil = true
    assert $evil
    end
    Tuesday, June 24, 14

    View Slide

  12. Globals
    10
    def test_isnt_evil
    refute $evil
    end
    def test_if_we_were_evil
    $evil = true
    assert $evil
    end
    false
    Tuesday, June 24, 14

    View Slide

  13. Globals
    11
    def test_isnt_evil
    refute $evil
    end
    def test_if_we_were_evil
    $evil = true
    assert $evil
    end
    true
    Tuesday, June 24, 14

    View Slide

  14. Globals
    12
    def test_if_we_were_evil
    $evil = true
    assert $evil
    end
    def test_isnt_evil
    refute $evil
    end
    true
    Tuesday, June 24, 14

    View Slide

  15. Globals
    13
    def test_if_we_were_evil
    $evil = true
    assert $evil
    end
    def test_isnt_evil
    refute $evil
    end
    true!
    Tuesday, June 24, 14

    View Slide

  16. 14
    https://flic.kr/p/97nB5F
    Tuesday, June 24, 14

    View Slide

  17. 15
    SPECIAL = 1
    "Constants"
    Tuesday, June 24, 14

    View Slide

  18. 16
    SPECIAL = [1,2,3]
    "Constants"
    Tuesday, June 24, 14

    View Slide

  19. 17
    SPECIAL = [1,2,3]
    SPECIAL << "Ha"
    "Constants"
    Tuesday, June 24, 14

    View Slide

  20. 18
    https://flic.kr/p/5vKLwQ
    Tuesday, June 24, 14

    View Slide

  21. 19
    class ClassyTest < Minitest::Test
    @@classy = true
    def test_classy
    assert @@classy
    end
    end
    Class Variables
    Tuesday, June 24, 14

    View Slide

  22. 20
    Class Variables
    class ClassyTest < Minitest::Test
    # ...
    def test_not_classy
    @@classy = false
    refute @@classy
    end
    end
    Tuesday, June 24, 14

    View Slide

  23. 21
    https://flic.kr/p/4EZada
    Tuesday, June 24, 14

    View Slide

  24. 22
    AS = ActiveSupport
    AS::Notifications.subscribe("event") do
    testing.is_awesome?
    end
    Events and Callbacks
    Tuesday, June 24, 14

    View Slide

  25. 23
    https://flic.kr/p/4Mu5NX
    Tuesday, June 24, 14

    View Slide

  26. 24
    class Object
    def boo
    puts "Boo"
    end
    end
    Metaprogramming
    Tuesday, June 24, 14

    View Slide

  27. 24
    class Object
    def boo
    puts "Boo"
    end
    end
    class Object
    remove_method(:boo)
    end
    Metaprogramming
    Tuesday, June 24, 14

    View Slide

  28. 24
    class Object
    def boo
    puts "Boo"
    end
    end
    class Object
    remove_method(:boo)
    end
    class Object
    alias :hoo :boo
    end
    Metaprogramming
    Tuesday, June 24, 14

    View Slide

  29. 25
    require 'code'
    require
    Tuesday, June 24, 14

    View Slide

  30. 26
    https://flic.kr/p/4YwYM8
    Tuesday, June 24, 14

    View Slide

  31. 27
    Threading
    class ThreadsAreCoolTest < Minitest::Test
    def test_coolness
    Thread.new do
    loop { $cool = true }
    end
    assert $cool
    end
    end
    Tuesday, June 24, 14

    View Slide

  32. 28
    https://flic.kr/p/B72L5
    Tuesday, June 24, 14

    View Slide

  33. 29
    Databases
    Tuesday, June 24, 14

    View Slide

  34. 29
    Queues
    Databases
    Tuesday, June 24, 14

    View Slide

  35. 29
    Files
    Queues
    Databases
    Tuesday, June 24, 14

    View Slide

  36. 29
    Files
    Queues
    Databases
    Services
    Tuesday, June 24, 14

    View Slide

  37. Comparison
    30
    Tuesday, June 24, 14

    View Slide

  38. Floats
    31
    def test_28
    assert_equal 28, (0.28 * 100)
    end
    Tuesday, June 24, 14

    View Slide

  39. Floats
    32
    1) Failure:
    test_28(Test_28) [/Users/jclark/test_28.rb]:
    <28> expected but was
    <28.000000000000004>.
    Tuesday, June 24, 14

    View Slide

  40. Floats
    33
    def test_28
    assert_in_delta(28, 0.28 * 100, 0.00001)
    end
    Tuesday, June 24, 14

    View Slide

  41. Time Roundtripping
    34
    def test_time
    time = Time.now.to_f
    round_trip = Time.at(time).to_f
    assert_equal(time, round_trip)
    end
    Tuesday, June 24, 14

    View Slide

  42. 35
    https://flic.kr/p/62n3eY
    Tuesday, June 24, 14

    View Slide

  43. 36
    https://twitter.com/joshsusser/status/257725572275376128
    Tuesday, June 24, 14

    View Slide

  44. https://flic.kr/p/bKnFx
    https://flic.kr/p/65uiJH
    Tuesday, June 24, 14

    View Slide

  45. 38
    https://flic.kr/p/btvCeB
    Tuesday, June 24, 14

    View Slide

  46. TECHNIQUE
    39
    Tuesday, June 24, 14

    View Slide

  47. Avoid
    40
    Tuesday, June 24, 14

    View Slide

  48. Instances are Friends
    41
    def test_newly_created
    subject = Testing.new
    assert subject.fresh?
    end
    Tuesday, June 24, 14

    View Slide

  49. Reset Consistently
    42
    def setup_and_teardown_agent(opts = {}, &block)
    define_method(:setup) do
    setup_agent(opts, &block)
    end
    define_method(:teardown) do
    teardown_agent
    end
    end
    Tuesday, June 24, 14

    View Slide

  50. Reset Consistently
    43
    class AgentTest < Minitest::Test
    setup_and_teardown_agent
    # ...
    end
    Tuesday, June 24, 14

    View Slide

  51. https://github.com/travisjeffery/timecop
    travisjeffery/timecop
    44
    describe "some set of tests to mock" do
    before do
    Timecop.freeze(Time.local(1990))
    end
    after do
    Timecop.return
    end
    it "should do blah blah blah" {}
    end
    Tuesday, June 24, 14

    View Slide

  52. 45
    rails/rails
    https://github.com/rails/rails
    Hat tip to http://brandonhilkert.com/blog/rails-4-1-travel-to-test-helper/
    travel_to Time.new(1999, 12, 31, 11, 59, 59) do
    # Test like it's 1999!
    end
    Tuesday, June 24, 14

    View Slide

  53. Time
    46
    def freeze_time(now=Time.now)
    Time.stubs(:now).returns(now)
    now
    end
    def advance_time(seconds)
    freeze_time(Time.now + seconds)
    end
    Tuesday, June 24, 14

    View Slide

  54. Metaprogramming
    47
    class TransmogifierTest < Minitest::Test
    def test_modifies_class
    clazz = Class.new
    Transmogifier.go!(clazz)
    assert clazz.new.tentacles?
    end
    end
    Tuesday, June 24, 14

    View Slide

  55. Metaprogramming
    48
    https://github.com/thoughtbot/appraisal
    Tuesday, June 24, 14

    View Slide

  56. Metaprogramming
    48
    https://github.com/thoughtbot/appraisal
    https://github.com/sinisterchipmunk/testbeds
    Tuesday, June 24, 14

    View Slide

  57. Metaprogramming
    48
    https://github.com/newrelic/rpm/blob/master/test/multiverse/README.md
    https://github.com/thoughtbot/appraisal
    https://github.com/sinisterchipmunk/testbeds
    Tuesday, June 24, 14

    View Slide

  58. Locate
    49
    Tuesday, June 24, 14

    View Slide

  59. 50
    Tuesday, June 24, 14

    View Slide

  60. 51
    puts "JRC: GOT HERE"
    Always Be Logging
    Tuesday, June 24, 14

    View Slide

  61. 52
    caller.join("\n\t")
    Where Am I?
    Tuesday, June 24, 14

    View Slide

  62. 53
    TESTOPTS="--seed=33023" bundle exec rake
    ruby test/awesome/code_test.rb --seed 33023
    Seeds of !Change
    Tuesday, June 24, 14

    View Slide

  63. 54
    Tuesday, June 24, 14

    View Slide

  64. 55
    class OrderedTest < Minitest::Test
    i_suck_and_my_tests_are_order_dependent!
    def test_feel_bad_but_carry_on
    # ...
    end
    end
    Method Order
    Tuesday, June 24, 14

    View Slide

  65. 56
    class SingleFailingTest < Minitest::Test
    def test_i_sometimes_fail
    flaky_call()
    end
    end
    Loop
    Tuesday, June 24, 14

    View Slide

  66. 57
    class SingleFailingTest < Minitest::Test
    def test_i_sometimes_fail
    100.times do
    setup
    flaky_call()
    teardown
    end
    end
    end
    Loop
    Tuesday, June 24, 14

    View Slide

  67. 58
    while true
    do
    bundle exec rake test || break
    done
    Loop
    Tuesday, June 24, 14

    View Slide

  68. Differences on the
    Build Box
    59
    OS
    Hardware
    Performance
    Paths, files, directories
    Gem versions
    Tuesday, June 24, 14

    View Slide

  69. Threading
    60
    class MultiThreadedWorker
    def go!
    Thread.new do
    work()
    end
    end
    end
    Tuesday, June 24, 14

    View Slide

  70. Threading
    61
    class MultiThreadedWorker
    def go!
    Thread.new do
    # TODO: ONLY FOR DIAGNOSISING!!!!!!!
    sleep(0.1)
    work()
    end
    end
    end
    Tuesday, June 24, 14

    View Slide

  71. Threading
    62
    class MultiThreadedWorker
    def go!
    50.times do
    Thread.new do
    work()
    end
    end
    end
    end
    Tuesday, June 24, 14

    View Slide

  72. Threading
    63
    class MultiThreadedWorker
    attr_reader :thread
    def go!
    @thread = Thread.new do
    # ...
    end
    end
    end
    Tuesday, June 24, 14

    View Slide

  73. Threading
    64
    class MultiThreadedWorkerTest < Minitest::Test
    def test_worker
    worker = MultiThreadedWorker.new
    worker.go!
    worker.thread.join
    end
    end
    Tuesday, June 24, 14

    View Slide

  74. TOOLS
    65
    Tuesday, June 24, 14

    View Slide

  75. 66
    jasonrclark/flake_test
    https://github.com/jasonrclark/flake_test
    Tuesday, June 24, 14

    View Slide

  76. 67
    gnarg/test_bisect
    https://github.com/gnarg/test_bisect
    Tuesday, June 24, 14

    View Slide

  77. gnarg/test_bisect
    https://github.com/gnarg/test_bisect
    68
    # Gemfile
    gem 'test_bisect'
    # Rakefile
    require 'test_bisect'
    TestBisect::BisectTask.new
    Tuesday, June 24, 14

    View Slide

  78. gnarg/test_bisect
    https://github.com/gnarg/test_bisect
    69
    class Test_01 < Test::Unit::TestCase
    def test_truth
    refute $globals_are_awesome
    end
    end
    Tuesday, June 24, 14

    View Slide

  79. 70
    Tuesday, June 24, 14

    View Slide

  80. gnarg/test_bisect
    https://github.com/gnarg/test_bisect
    71
    class Test_06 < Test::Unit::TestCase
    def test_truth
    refute $globals_are_awesome
    end
    end
    Tuesday, June 24, 14

    View Slide

  81. 72
    Tuesday, June 24, 14

    View Slide

  82. 73
    Tuesday, June 24, 14

    View Slide

  83. 74
    Tuesday, June 24, 14

    View Slide

  84. gnarg/test_bisect
    https://github.com/gnarg/test_bisect
    75
    class Test_05 < Test::Unit::TestCase
    def test_filthy_lies
    $globals_are_awesome = true
    assert $globals_are_awesome
    end
    end
    Tuesday, June 24, 14

    View Slide

  85. gnarg/test_bisect
    https://github.com/gnarg/test_bisect
    76
    Test::Unit :(
    Tuesday, June 24, 14

    View Slide

  86. 77
    jasonrclark/hometown
    https://github.com/jasonrclark/hometown
    Tuesday, June 24, 14

    View Slide

  87. jasonrclark/hometown
    https://github.com/jasonrclark/hometown
    78
    # Gemfile
    gem 'hometown'
    # test_helper.rb
    Hometown.watch(::Thread)
    Tuesday, June 24, 14

    View Slide

  88. jasonrclark/hometown
    https://github.com/jasonrclark/hometown
    79
    class NoticeThreadTest < Minitest::Test
    def setup
    Thread.new { work! }
    end
    def test_notice_thread
    Thread.list.each do |thread|
    trace = Hometown.for(thread)
    p trace unless trace.nil?
    end
    end
    end
    Tuesday, June 24, 14

    View Slide

  89. jasonrclark/hometown
    https://github.com/jasonrclark/hometown
    80
    #@traced_class=Thread,
    @backtrace=[
    "...loose_thread_test.rb:5:in `test_loose_...'",
    "...minitest-5.3.4/lib/minitest/test.rb:106",
    "...minitest-5.3.4/lib/minitest/test.rb:204",
    "...minitest-5.3.4/lib/minitest/test.rb:103"
    # ... >
    Tuesday, June 24, 14

    View Slide

  90. jasonrclark/minitest-stately
    https://github.com/jasonrclark/minitest-stately
    81
    Tuesday, June 24, 14

    View Slide

  91. jasonrclark/minitest-stately
    https://github.com/jasonrclark/minitest-stately
    82
    # Gemfile
    gem 'minitest-stately'
    # test_helper.rb
    # Watch for freshly started threads
    Minitest::Stately.watch("thread count") do
    Thread.list.count
    end
    Tuesday, June 24, 14

    View Slide

  92. 83
    Tuesday, June 24, 14

    View Slide

  93. jasonrclark/minitest-stately
    https://github.com/jasonrclark/minitest-stately
    84
    Minitest::Stately.run do
    $silly_global_state = nil
    end
    Tuesday, June 24, 14

    View Slide

  94. jasonrclark/minitest-stately
    https://github.com/jasonrclark/minitest-stately
    85
    Minitest::Stately.fail_if do
    $silly_global_state == "seriously"
    end
    Tuesday, June 24, 14

    View Slide

  95. jasonrclark/minitest-stately
    https://github.com/jasonrclark/minitest-stately
    86
    Minitest 5.x :(
    Tuesday, June 24, 14

    View Slide

  96. TROUBLES
    87
    Tuesday, June 24, 14

    View Slide

  97. TECHNIQUE
    88
    Tuesday, June 24, 14

    View Slide

  98. TOOLS
    89
    Tuesday, June 24, 14

    View Slide

  99. 90
    Jason Clark
    @jasonrclark
    Tuesday, June 24, 14

    View Slide