Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Setting Your Tests Up to Fail

Setting Your Tests Up to Fail

It’s a fact of life that we often have to write automated UI tests for features that have defects, or that interact with 3rd party APIs that aren’t returning the right responses, or for items that we know aren’t working right. When the team has decided that the behavior isn’t going to be fixed, what’s an automation engineer to do? Let the tests fail? Not write them? Champion harder for the defects?

Jenny Bramble suggests writing your tests to pass.

By creating tests that pass on the current expected behavior (the defect), we are in a perfect position to tell when the defect is resolved, or the api is returning the correct information, or any of the other error cases we may be encountering. This prevents failure fatigue (from seeing a test ‘always fail), while still providing meaningful, actionable information out of our test suite.

She will discuss several cases that she’s experienced that this method has worked for as well as how to keep the rest of the team informed through TODOs, Jira stories, and documentation. And—of course!—what to do when your test finally fails.

You’ll walk away from this talk with a clear idea of when to design your tests to fail, how to use TODOs and other indicators to let the rest of the test team know what’s going on, as well as a frank discussion of automation as information—not as a bug detection system!

Jenny Bramble

April 15, 2021
Tweet

More Decks by Jenny Bramble

Other Decks in Technology

Transcript

  1. SETTING YOUR TESTS
    UP TO FAIL
    @jennydoesthings

    View full-size slide

  2. HI, I’M JENNY
    HUMAN INTERFACING IS MY
    FAVORITE THING
    MY PRONOUNS ARE SHE/HER
    @jennydoesthings

    View full-size slide

  3. A BIT OF HISTORY
    FIRST,

    View full-size slide

  4. PAUL GRIZZAFFI
    TWITTER.COM/PGRIZZAFFI
    @jennydoesthings

    View full-size slide

  5. I WANT TO WARN YOU
    AHEAD OF TIME

    View full-size slide

  6. LET’S TALK ABOUT
    DESIGNING TESTS TO FAIL
    NOW,

    View full-size slide

  7. TESTING VERIFIES
    EXPECTED BEHAVIORS
    POINT 1:

    View full-size slide

  8. I HAVE FOUND A
    LOT OF DEFECTS
    @jennydoesthings

    View full-size slide

  9. MANY DEFECTS
    ARE NEVER FIXED
    @jennydoesthings

    View full-size slide

  10. THIS MAKES THE
    BEHAVIOR EXPECTED
    @jennydoesthings

    View full-size slide

  11. THIS INCLUDES
    3RD PARTY APIS
    @jennydoesthings

    View full-size slide

  12. AND THEN THERE’S
    COWBOYS
    @jennydoesthings

    View full-size slide

  13. ….MAYBE
    @jennydoesthings

    View full-size slide

  14. AUTOMATION GIVES
    INFORMATION
    POINT 2:

    View full-size slide

  15. AUTOMATION HAS
    NEVER FOUND A DEFECT
    @jennydoesthings

    View full-size slide

  16. EVER.
    @jennydoesthings

    View full-size slide

  17. AUTOMATION HAS
    NEVER FOUND A DEFECT
    @jennydoesthings

    View full-size slide

  18. AUTOMATION PROVIDES
    INFORMATION
    @jennydoesthings

    View full-size slide

  19. HUMANS DETERMINE WHAT
    INFORMATION MEANS
    @jennydoesthings

    View full-size slide

  20. AUTOMATION PROVIDES INFORMATION
    ABOUT EXPECTED BEHAVIORS
    POINT 3:

    View full-size slide

  21. TESTS SHOULD PASS ON
    EXPECTED BEHAVIOR
    @jennydoesthings

    View full-size slide

  22. TESTS SHOULD PASS
    ON (SOME) DEFECTS
    @jennydoesthings

    View full-size slide

  23. KNOW THE STATE
    OF THE SYSTEM
    @jennydoesthings

    View full-size slide

  24. LIMITING FAILURE
    FATIGUE
    @jennydoesthings

    View full-size slide

  25. INCREASED CONFIDENCE
    IN OUR SUITE
    @jennydoesthings

    View full-size slide

  26. TREATING AUTOMATION
    AS INFORMATION
    @jennydoesthings

    View full-size slide

  27. I CALL THIS
    DESIGNING TESTS TO FAIL

    View full-size slide

  28. TESTS ARE
    WRITTEN TO PASS
    @jennydoesthings

    View full-size slide

  29. TESTS ARE
    PLANNED TO FAIL
    @jennydoesthings

    View full-size slide

  30. DOCUMENT THE
    BEHAVIOR
    @jennydoesthings

    View full-size slide

  31. HOW DO YOU
    DESIGN TESTS TO FAIL

    View full-size slide

  32. ALL HAIL
    DOCUMENTATION
    @jennydoesthings

    View full-size slide

  33. SERIOUSLY, DOCUMENT
    YOUR TESTS
    @jennydoesthings

    View full-size slide

  34. THE BEHAVIOR HAS
    CHANGED
    @jennydoesthings
    THE TEST IS
    FLAKY

    View full-size slide

  35. LET’S TALK ABOUT
    DOCUMENTATION
    @jennydoesthings

    View full-size slide

  36. DOCUMENTATION

    View full-size slide

  37. CREATE CONTEXT
    @jennydoesthings

    View full-size slide

  38. START WITH
    TICKETS
    @jennydoesthings

    View full-size slide

  39. Defect 1414
    @JENNYDOESTHINGS
    "THE TEST IS IN THE LOGIN GROUP AND CURRENTLY
    VERIFIES THAT THE NEW PASSWORD IS AT LEAST SIX
    CHARACTERS. WHEN THIS DEFECT IS RESOLVED, THE
    TEST WILL NEED TO BE UPDATED TO THE CORRECT
    VALUE OF EIGHT CHARACTERS."

    View full-size slide

  40. USE CODE
    COMMENTS
    @jennydoesthings

    View full-size slide

  41. Your Code
    @JENNYDOESTHINGS
    //TODO: REVISIT WHEN DEFECT 1414 IS RESOLVED. TEST
    WILL NEED TO BE UPDATED FROM 6 CHARS TO 8 CHARS.

    View full-size slide

  42. Xcode
    @JENNYDOESTHINGS

    View full-size slide

  43. Xcode
    @JENNYDOESTHINGS

    View full-size slide

  44. Xcode
    @JENNYDOESTHINGS

    View full-size slide

  45. Your Code
    @JENNYDOESTHINGS
    //TODO: REVISIT WHEN DEFECT 1414 IS RESOLVED. TEST
    WILL NEED TO BE UPDATED FROM 6 CHARS TO 8 CHARS.

    View full-size slide

  46. Android Studio
    @JENNYDOESTHINGS

    View full-size slide

  47. Android Studio
    @JENNYDOESTHINGS

    View full-size slide

  48. USE CODE
    COMMENTS
    @jennydoesthings

    View full-size slide

  49. REVISIT YOUR
    TODOS
    @jennydoesthings

    View full-size slide

  50. CONSIDER THE
    README
    @jennydoesthings

    View full-size slide

  51. RESOLVE FAILED
    TESTS QUICKLY
    @jennydoesthings

    View full-size slide

  52. @JENNYDOESTHINGS
    WHY DO WE DESIGN TESTS TO FAIL
    ▸ Know the state of the system
    ▸ Limit failure fatigue
    ▸ Increase confidence in our suite
    ▸ Treat automation as information

    View full-size slide

  53. @JENNYDOESTHINGS
    HOW DO WE DESIGN TESTS TO FAIL
    ▸ Write the test to pass on
    expected behavior
    ▸ Document your tests
    ▸ Expect your tests to fail
    ▸ Resolve failures quickly

    View full-size slide

  54. @JENNYDOESTHINGS
    FIND ME ONLINE!
    [email protected]
    ▸ http://twitter.com/jennydoesthings
    ▸ https://linkedin.com/in/
    jennybramble
    ▸ http://jennydoesthings.com
    ▸ Willowtree!

    View full-size slide

  55. IT’S STORY TIME!
    Me, probably
    @JENNYDOESTHINGS

    View full-size slide