Deterministic Solutions to Intermittent Failures

Deterministic Solutions to Intermittent Failures

Presented at RubyConf 2017 in New Orleans.


Tim Mertens

November 16, 2017



  2. T i m M e r t e n s

    HELLO, My Name Is @rockfx01
  3. None
  4. RSpec A testing framework for ruby

  5. Source Code Examples

  6. “ Do not observe the build status. You will disrupt

    its quantum state! “
  7. The Myth of Flaky Tests

  8. The Myth of Flaky Tests DECONSTRUCTED

  9. Tests Are Software Too • Test code does exactly what

    you tell it to do • “Flaky” implies an unsolvable problem • “Non-Deterministic” behavior can be accounted for • Any failure can be resolved once you know the root cause
  10. Real Defects • Ignored failures may be real defects


  12. Continuous Integration They call me “CI” for short A process

    or system by which new code is continuously validated against an existing test suite.
  13. None
  14. None
  15. None
  16. Parallelized Builds Builds which spread the work of executing tests

    across 2 or more workers (e.g. containers, nodes)
  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None

  29. None
  30. Debugging Reproducible Failures

  31. Common Reproducible Failures • Stale Branches • Business Dates and

    Times • Mocked Time vs System Time • Missing Preconditions • Real Bugs
  32. None
  33. Test Group A subset of tests from the test suite

    which run on a specific node in a parallelized build.
  34. RSpec Test Group $ rspec # OR $ rspec ./spec/some_spec.rb

 $ rspec . --tag focus No Group - Runs All Tests Metadata Tags Specific Files
  35. Test Seed A value, usually an integer, which determines the

    order in which tests are executed.
  36. RSpec Test Seed $ rspec
 Finished in 1 minutes

    17.7 seconds
 99 examples, 0 failures, 4 pending
 Randomized with seed 13391 Test Seed
  37. Re-running Test Group With Seed $ rspec --seed 12345 --fail-fast

    # OR $ rspec ./spec/some_spec.rb \
 ./spec/other_spec.rb \
 --seed 12345 --fail-fast
  38. None
  39. Test Bisect Repeatedly dividing a set of tests in half

    until you find the minimal set of tests which cause another test to fail.
  40. Bisecting Test Group with Seed $ rspec --seed 12345 --bisect

    # OR $ rspec ./spec/some_spec.rb \
 ./spec/other_spec.rb \
 --seed 12345 --bisect
  41. None
  42. None
  43. None
  44. None
  45. Test Pollution When the side effects of one or more

    tests in a test group cause one or more other tests to fail.
  46. Debugging Test Pollution Failures

  47. Data Pollution • Data is persisted across test examples or

    test suite executions ◦ Database Records ◦ Caches (e.g. Redis)
  48. Defensive Testing • Tests should clean up after themselves, but…

    • Don’t expect pristine starting conditions
  49. • Don’t expect tables to be empty Defensive Testing #

 expect(User.count ).to eq 1
 # Do:
 expect { }.to change { User.count }.by(1)
  50. • Don’t expect global scopes to only return test records

    # Don’t:
 expect( match_array [user1, user2]
 # Do:
 expect( include(user1, user2)
 expect( include(user3) Defensive Testing
  51. Class/Singleton Caching • Reset cache mutations after tests that modify

    them • Avoid mutating caches in tests
  52. Class/Singleton Caching # Don’t:
 described_class.add("foo") # mutates the singleton

    be true
 # Do:
 subject =
 expect(subject.contains?("foo")).to be true
  53. Mutated Constants • Don’t Overwrite constants # Don’t:
 before {

    SOME_CONST = "my test value” }
 # Do:
 stub_const("MyClass", "my test value")
 allow(MyClass).to receive(:foo).and_return("foo")
 fake_class = class_double(MyClass, foo: "foo")
 stub_const("MyClass", fake_class)
  54. Mutated Constants # Don't:
 before do
 MyClass.define_method(:foo) { "foo" }

 # Do:
 instance =
 allow(instance).to receive(:foo).and_return(“foo")
  55. Mutated (Test) Constants describe Foo do
 # Don’t:
 BAR =

 it { expect( eq BAR }
 # Do:
 let(:bar) { "some_value" }
 it { expect( eq bar }
  56. Real Bugs! • Always ensure you understand the reason for

    the test failure and ensure your production code is not at fault
  57. None
  58. Running Tests in a Loop describe MyClass do
 100.times do

    describe "#some_method" do
 it "does something" do
 # ...
  59. $ rspec ./spec/some_spec.rb --fail-fast Running Tests in a Loop

  60. None
  61. Non-Deterministic Failure F a i l u r e s

    t h a t o c c u r a t seemingly random frequencies due to non-deterministic behavior of the code under test.
  62. Debugging Non-Deterministic Failures

  63. Unordered Queries • Don’t assume queries return results in specific

    order • Unordered queries in Postgresql ◦ Postgresql returns results in non-deterministic order if query is not explicitly sorted # Don’t:
 expect(results).to eq [record_1, record_2]
 # Do:
 expect(results).to contain_exactly record_1, record_2
 expect(results).to match_array [record_1, record_2]
  64. Frozen Time • Creating records in frozen time ◦ All

    records have the same created_at time ◦ Queries ordered by created_at will return results in non-deterministic order • Prefer Timecop#travel over Timecop#freeze • Only freeze time when precise time is needed
  65. Randomized Test Data • Faker or other data generation or

    sampling methods return unexpected or unsupported data ◦ Non-alpha names (“D’Angelo”, “Doe-Smith”, “Mc Donald”) ◦ Invalid phone numbers, zip codes, unsupported states, etc. • Output relevant randomized data in the test error message to make troubleshooting easier
  66. None
  67. Debugging “Unreproducible” Failures

  68. Date and Time • Tests only fail on weekends/holidays? •

    Tests only fail at certain time of day? • Timecop to the date/time when the tests ran in CI Avant timecop-rspec gem:
  69. UTC vs Local Date/Time • `` uses system time zone

    • `Date.current` uses application time zone
  70. UTC vs Local Date/Time ENV["TZ"] = "UTC" = "America/Chicago"

 early_morning_utc = Time.utc(2017,11,10,2) do
 # This will fail:
 expect(Date.current).to eq
  71. SQL Date Comparisons • Database queries comparing Dates to Time

    ◦ Never pass Time objects to sequel queries against Date columns MyModel.where(‘start_date <= ?’,
 #=> SELECT “my_models”.*
 FROM “my_models”
 WHERE (start_date <= ‘2017-11-03 06:29:45’)
  72. Timeouts and Asynchronous Javascript • CI performance is often worse

    than your local machine • Page load performance can vary widely based on application configuration and test ordering • Increase timeouts for CI as needed • Don’t use browser tests for performance testing
  73. Timeouts and Asynchronous Javascript • Wait for pages to finish

    loading before interacting with them ◦ SitePrism load_validations:
  74. Environmental Differences • Compare CI configuration and setup to local

    ◦ Environment Variables ◦ Test setup or execution inconsistencies • Database ◦ Seeds ◦ Migrations missing from schema or structure files
  75. Environmental Differences • Library versions or inconsistencies • Operating System

    differences • Use Docker for consistency
  76. Strategies for Unreproducible Failures • SSH into CI and try

    to reproduce • Use common sense ◦ What are the probable causes of the failure? • Check gem github repos for related issues or changes • Learn to use pry, byebug • Incrementally narrow the scope of the defect
  77. Strategies for Unreproducible Failures • Know your test support code

    in and out • Look at failure trends over time • Add logging
  78. Describe “#the_end” do
 It “is just the beginning”

  79. Takeaways • Keep your builds green to avoid sadness •

    Tests are code too • Set realistic goals • Celebrate success!
  80. None
  81. Get In Touch Tim Mertens Github { tmertens } Twitter

    { @rockfx01 }
  82. None