Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Setting Your Tests Up to Fail

Setting Your Tests Up to Fail

It’s a fact of life that we often have to write automated UI tests for features that have defects, or that interact with 3rd party APIs that aren’t returning the right responses, or for items that we know aren’t working right. When the team has decided that the behavior isn’t going to be fixed, what’s an automation engineer to do? Let the tests fail? Not write them? Champion harder for the defects?

Jenny Bramble suggests writing your tests to pass.

By creating tests that pass on the current expected behavior (the defect), we are in a perfect position to tell when the defect is resolved, or the api is returning the correct information, or any of the other error cases we may be encountering. This prevents failure fatigue (from seeing a test ‘always fail), while still providing meaningful, actionable information out of our test suite.

She will discuss several cases that she’s experienced that this method has worked for as well as how to keep the rest of the team informed through TODOs, Jira stories, and documentation. And—of course!—what to do when your test finally fails.

You’ll walk away from this talk with a clear idea of when to design your tests to fail, how to use TODOs and other indicators to let the rest of the test team know what’s going on, as well as a frank discussion of automation as information—not as a bug detection system!

C15215a461aefca9440fdbc983a18272?s=128

Jenny Bramble

April 15, 2021
Tweet

Transcript

  1. SETTING YOUR TESTS UP TO FAIL @jennydoesthings

  2. HI, I’M JENNY HUMAN INTERFACING IS MY FAVORITE THING MY

    PRONOUNS ARE SHE/HER @jennydoesthings
  3. A BIT OF HISTORY FIRST,

  4. PAUL GRIZZAFFI TWITTER.COM/PGRIZZAFFI @jennydoesthings

  5. I WANT TO WARN YOU AHEAD OF TIME

  6. None
  7. YES.

  8. LET’S TALK ABOUT DESIGNING TESTS TO FAIL NOW,

  9. TESTING VERIFIES EXPECTED BEHAVIORS POINT 1:

  10. I HAVE FOUND A LOT OF DEFECTS @jennydoesthings

  11. MANY DEFECTS ARE NEVER FIXED @jennydoesthings

  12. THIS MAKES THE BEHAVIOR EXPECTED @jennydoesthings

  13. THIS INCLUDES 3RD PARTY APIS @jennydoesthings

  14. AND THEN THERE’S COWBOYS @jennydoesthings

  15. ….MAYBE @jennydoesthings

  16. AUTOMATION GIVES INFORMATION POINT 2:

  17. AUTOMATION HAS NEVER FOUND A DEFECT @jennydoesthings

  18. EVER. @jennydoesthings

  19. AUTOMATION HAS NEVER FOUND A DEFECT @jennydoesthings

  20. AUTOMATION PROVIDES INFORMATION @jennydoesthings

  21. HUMANS DETERMINE WHAT INFORMATION MEANS @jennydoesthings

  22. AUTOMATION PROVIDES INFORMATION ABOUT EXPECTED BEHAVIORS POINT 3:

  23. TESTS SHOULD PASS ON EXPECTED BEHAVIOR @jennydoesthings

  24. TESTS SHOULD PASS ON (SOME) DEFECTS @jennydoesthings

  25. BUT WHY

  26. KNOW THE STATE OF THE SYSTEM @jennydoesthings

  27. LIMITING FAILURE FATIGUE @jennydoesthings

  28. INCREASED CONFIDENCE IN OUR SUITE @jennydoesthings

  29. TREATING AUTOMATION AS INFORMATION @jennydoesthings

  30. I CALL THIS DESIGNING TESTS TO FAIL

  31. TESTS ARE WRITTEN TO PASS @jennydoesthings

  32. TESTS ARE PLANNED TO FAIL @jennydoesthings

  33. DOCUMENT THE BEHAVIOR @jennydoesthings

  34. HOW DO YOU DESIGN TESTS TO FAIL

  35. ALL HAIL DOCUMENTATION @jennydoesthings

  36. SERIOUSLY, DOCUMENT YOUR TESTS @jennydoesthings

  37. THE BEHAVIOR HAS CHANGED @jennydoesthings THE TEST IS FLAKY

  38. LET’S TALK ABOUT DOCUMENTATION @jennydoesthings

  39. DOCUMENTATION

  40. CREATE CONTEXT @jennydoesthings

  41. START WITH TICKETS @jennydoesthings

  42. Defect 1414 @JENNYDOESTHINGS "THE TEST IS IN THE LOGIN GROUP

    AND CURRENTLY VERIFIES THAT THE NEW PASSWORD IS AT LEAST SIX CHARACTERS. WHEN THIS DEFECT IS RESOLVED, THE TEST WILL NEED TO BE UPDATED TO THE CORRECT VALUE OF EIGHT CHARACTERS."
  43. USE CODE COMMENTS @jennydoesthings

  44. Your Code @JENNYDOESTHINGS //TODO: REVISIT WHEN DEFECT 1414 IS RESOLVED.

    TEST WILL NEED TO BE UPDATED FROM 6 CHARS TO 8 CHARS.
  45. Xcode @JENNYDOESTHINGS

  46. Xcode @JENNYDOESTHINGS

  47. Xcode @JENNYDOESTHINGS

  48. Your Code @JENNYDOESTHINGS //TODO: REVISIT WHEN DEFECT 1414 IS RESOLVED.

    TEST WILL NEED TO BE UPDATED FROM 6 CHARS TO 8 CHARS.
  49. Android Studio @JENNYDOESTHINGS

  50. Android Studio @JENNYDOESTHINGS

  51. USE CODE COMMENTS @jennydoesthings

  52. REVISIT YOUR TODOS @jennydoesthings

  53. CONSIDER THE README @jennydoesthings

  54. RESOLVE FAILED TESTS QUICKLY @jennydoesthings

  55. IN SUMMARY

  56. @JENNYDOESTHINGS WHY DO WE DESIGN TESTS TO FAIL ▸ Know

    the state of the system ▸ Limit failure fatigue ▸ Increase confidence in our suite ▸ Treat automation as information
  57. @JENNYDOESTHINGS HOW DO WE DESIGN TESTS TO FAIL ▸ Write

    the test to pass on expected behavior ▸ Document your tests ▸ Expect your tests to fail ▸ Resolve failures quickly
  58. @JENNYDOESTHINGS FIND ME ONLINE! ▸ jenny.bramble@gmail.com ▸ http://twitter.com/jennydoesthings ▸ https://linkedin.com/in/

    jennybramble ▸ http://jennydoesthings.com ▸ Willowtree!
  59. None
  60. IT’S STORY TIME! Me, probably @JENNYDOESTHINGS