Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Quarantine: Get your confidence back

Quarantine: Get your confidence back

A technique for dealing with non-deterministic tests.

Robbie Clutton

March 07, 2013
Tweet

More Decks by Robbie Clutton

Other Decks in Technology

Transcript

  1. @robb1e
    Software Engineer, Pivotal Labs
    Robbie Clutton
    Quarantine

    View full-size slide

  2. @robb1e
    $ rake
    ..........F.........
    $ rake
    ...................
    $ rake
    ....F...........F....
    $ rake
    ...................

    View full-size slide

  3. @robb1e
    You say...
    Oh, it’s just that
    flakey test, run the
    build again

    View full-size slide

  4. @robb1e
    3 minutes ago: failed
    12 minutes ago: passed
    30 minutes ago: failed
    45 minutes ago: failed
    2 hours ago: passed

    View full-size slide

  5. @robb1e
    You say...
    Run all build agents, one
    of them will go through

    View full-size slide

  6. @robb1e
    Confidence, meet window

    View full-size slide

  7. @robb1e
    @quarantine
    Scenario: Something flakey
    #....
    describe “Flakey”, :quarantine do
    # ...

    View full-size slide

  8. @robb1e
    #regular ci build
    #!/bin/bash
    cucumber --tags ~@quarantine
    rspec . --tags ~quarantine

    View full-size slide

  9. @robb1e
    #quarantine ci build
    #!/bin/bash
    cucumber --tags @quarantine
    rspec . --tags quarantine

    View full-size slide

  10. @robb1e
    Regular
    Quarantine
    Deploy
    Continuous Integration

    View full-size slide

  11. @robb1e
    BUILD NANNY
    1. Rewrite
    2. Rethink
    3. Delete

    View full-size slide

  12. “You should not be afraid to delete tests that are no
    longer providing value, no matter whether you originally
    planned to keep them or not. We tend to treat tests as
    these holy creatures that live blameless, irreproachable
    lives once they have sprung into existence. Not so. The
    maintenance required to keep a test running weighs
    against its value in further development. Sometimes
    these lines cross, and the test simply becomes a
    burden on the project. Having the skill and experience
    to recognize a burdensome test is something we should
    be bringing to our clients, as well as the fortitude to
    rewrite it, rethink it, or delete it."
    - Adam Milligan

    View full-size slide

  13. @robb1e
    Request
    Controller
    Model
    View Javascript
    Decorator
    Database

    View full-size slide

  14. @robb1e
    $ rake
    ...................
    $ rake
    ...................
    $ rake
    ...................
    $ rake
    ...................

    View full-size slide

  15. @robb1e
    3 minutes ago: passed
    12 minutes ago: passed
    30 minutes ago: passed
    45 minutes ago: passed
    2 hours ago: passed

    View full-size slide

  16. @robb1e
    Time to get the confidence back

    View full-size slide