Quarantine: Get your confidence back

A technique for dealing with non-deterministic tests.

Transcript

1. @robb1e Software Engineer, Pivotal Labs Robbie Clutton Quarantine
2. @robb1e $ rake ..........F......... $ rake ................... $ rake ....F...........F.... $ rake ...................
3. @robb1e You say... Oh, it’s just that flakey test, run the build again
4. @robb1e 3 minutes ago: failed 12 minutes ago: passed 30 minutes ago: failed 45 minutes ago: failed 2 hours ago: passed
5. @robb1e You say... Run all build agents, one of them will go through
6. @robb1e
7. @robb1e Confidence, meet window
8. @robb1e @quarantine Scenario: Something flakey #.... describe “Flakey”, :quarantine do # ...
9. @robb1e #regular ci build #!/bin/bash cucumber --tags ~@quarantine rspec . --tags ~quarantine
10. @robb1e #quarantine ci build #!/bin/bash cucumber --tags @quarantine rspec . --tags quarantine
11. @robb1e Regular Quarantine Deploy Continuous Integration
12. @robb1e BUILD NANNY 1. Rewrite 2. Rethink 3. Delete
13. “You should not be afraid to delete tests that are no longer providing value, no matter whether you originally planned to keep them or not. We tend to treat tests as these holy creatures that live blameless, irreproachable lives once they have sprung into existence. Not so. The maintenance required to keep a test running weighs against its value in further development. Sometimes these lines cross, and the test simply becomes a burden on the project. Having the skill and experience to recognize a burdensome test is something we should be bringing to our clients, as well as the fortitude to rewrite it, rethink it, or delete it." - Adam Milligan
14. @robb1e Request Controller Model View Javascript Decorator Database
15. @robb1e $ rake ................... $ rake ................... $ rake ................... $ rake ...................
16. @robb1e 3 minutes ago: passed 12 minutes ago: passed 30 minutes ago: passed 45 minutes ago: passed 2 hours ago: passed
17. @robb1e Time to get the confidence back
18. @robb1e Q/A