Fight flaky specs with RSpec

Presented at November 2016 meetup of Rorosyd - Ruby or Rails User Group Sydney
https://www.meetup.com/Ruby-On-Rails-Oceania-Sydney/events/232860358/

Transcript

1. Fight flaky specs with RSpec by Qing Wu
2. Blake Education
3. A transition from PHP to anything would be an improvement — Stuart Coyle
4. TDD Is Dead — @DHH
6. Flaky specs » sometimes pass and sometimes don't » the reason is not obvious
7. Example » Repo with 3000+ specs » The build takes about 10 minutes » Build failed six times in a row
8. Wasted hours and frustration
9. Cost $$$ to the company
10. Less confidence in the codebase
11. RSpec to the rescue rspec --bisect (in rspec >= 3.3) bisect output
12. The command be rspec --seed 33313 --bisect
14. step 1 Running suite to find failures... (19 minutes 32 seconds)
15. step 2 Starting bisect with 28 failing examples and 3128 non-failing examples.
16. step 3 Checking that failure(s) are order-dependent... failure appears to be order-dependent
17. step 4 Round 1: bisecting over non-failing examples 1-3128 . ignoring examples 1-1564 (6 minutes 29 seconds) . . Round 11: ... Bisect complete! Reduced necessary non-failing examples from 3128 to 1 in 18 minutes 49 seconds.
18. Result The minimal reproduction command is: rspec './spec/model/some_model_spec.rb[1:1:1:1]' ...
19. Order dependent » pass when run individually » fail consistently when run in the same order
20. What is happening???
21. The fix
23. Two types of flaky specs 1. Order dependent 2. Race condition
24. How to use it
25. Just use it!
26. Questions