Setting Your Tests Up to Fail

It’s a fact of life that we often have to write automated UI tests for features that have defects, or that interact with 3rd party APIs that aren’t returning the right responses, or for items that we know aren’t working right. When the team has decided that the behavior isn’t going to be fixed, what’s an automation engineer to do? Let the tests fail? Not write them? Champion harder for the defects?

Jenny Bramble suggests writing your tests to pass.

By creating tests that pass on the current expected behavior (the defect), we are in a perfect position to tell when the defect is resolved, or the api is returning the correct information, or any of the other error cases we may be encountering. This prevents failure fatigue (from seeing a test ‘always fail), while still providing meaningful, actionable information out of our test suite.

She will discuss several cases that she’s experienced that this method has worked for as well as how to keep the rest of the team informed through TODOs, Jira stories, and documentation. And—of course!—what to do when your test finally fails.

You’ll walk away from this talk with a clear idea of when to design your tests to fail, how to use TODOs and other indicators to let the rest of the test team know what’s going on, as well as a frank discussion of automation as information—not as a bug detection system!

Transcript

1. SETTING YOUR TESTS UP TO FAIL @jennydoesthings
2. HI, I’M JENNY HUMAN INTERFACING IS MY FAVORITE THING MY PRONOUNS ARE SHE/HER @jennydoesthings
3. A BIT OF HISTORY FIRST,
4. PAUL GRIZZAFFI TWITTER.COM/PGRIZZAFFI @jennydoesthings
5. I WANT TO WARN YOU AHEAD OF TIME
7. YES.
8. LET’S TALK ABOUT DESIGNING TESTS TO FAIL NOW,
9. TESTING VERIFIES EXPECTED BEHAVIORS POINT 1:
10. I HAVE FOUND A LOT OF DEFECTS @jennydoesthings
11. MANY DEFECTS ARE NEVER FIXED @jennydoesthings
12. THIS MAKES THE BEHAVIOR EXPECTED @jennydoesthings
13. THIS INCLUDES 3RD PARTY APIS @jennydoesthings
14. AND THEN THERE’S COWBOYS @jennydoesthings
15. ….MAYBE @jennydoesthings
16. AUTOMATION GIVES INFORMATION POINT 2:
17. AUTOMATION HAS NEVER FOUND A DEFECT @jennydoesthings
18. EVER. @jennydoesthings
19. AUTOMATION HAS NEVER FOUND A DEFECT @jennydoesthings
20. AUTOMATION PROVIDES INFORMATION @jennydoesthings
21. HUMANS DETERMINE WHAT INFORMATION MEANS @jennydoesthings
22. AUTOMATION PROVIDES INFORMATION ABOUT EXPECTED BEHAVIORS POINT 3:
23. TESTS SHOULD PASS ON EXPECTED BEHAVIOR @jennydoesthings
24. TESTS SHOULD PASS ON (SOME) DEFECTS @jennydoesthings
25. BUT WHY
26. KNOW THE STATE OF THE SYSTEM @jennydoesthings
27. LIMITING FAILURE FATIGUE @jennydoesthings
28. INCREASED CONFIDENCE IN OUR SUITE @jennydoesthings
29. TREATING AUTOMATION AS INFORMATION @jennydoesthings
30. I CALL THIS DESIGNING TESTS TO FAIL
31. TESTS ARE WRITTEN TO PASS @jennydoesthings
32. TESTS ARE PLANNED TO FAIL @jennydoesthings
33. DOCUMENT THE BEHAVIOR @jennydoesthings
34. HOW DO YOU DESIGN TESTS TO FAIL
35. ALL HAIL DOCUMENTATION @jennydoesthings
36. SERIOUSLY, DOCUMENT YOUR TESTS @jennydoesthings
37. THE BEHAVIOR HAS CHANGED @jennydoesthings THE TEST IS FLAKY
38. LET’S TALK ABOUT DOCUMENTATION @jennydoesthings
39. DOCUMENTATION
40. CREATE CONTEXT @jennydoesthings
41. START WITH TICKETS @jennydoesthings
42. Defect 1414 @JENNYDOESTHINGS "THE TEST IS IN THE LOGIN GROUP AND CURRENTLY VERIFIES THAT THE NEW PASSWORD IS AT LEAST SIX CHARACTERS. WHEN THIS DEFECT IS RESOLVED, THE TEST WILL NEED TO BE UPDATED TO THE CORRECT VALUE OF EIGHT CHARACTERS."
43. USE CODE COMMENTS @jennydoesthings
44. Your Code @JENNYDOESTHINGS //TODO: REVISIT WHEN DEFECT 1414 IS RESOLVED. TEST WILL NEED TO BE UPDATED FROM 6 CHARS TO 8 CHARS.
45. Xcode @JENNYDOESTHINGS
46. Xcode @JENNYDOESTHINGS
47. Xcode @JENNYDOESTHINGS
48. Your Code @JENNYDOESTHINGS //TODO: REVISIT WHEN DEFECT 1414 IS RESOLVED. TEST WILL NEED TO BE UPDATED FROM 6 CHARS TO 8 CHARS.
49. Android Studio @JENNYDOESTHINGS
50. Android Studio @JENNYDOESTHINGS
51. USE CODE COMMENTS @jennydoesthings
52. REVISIT YOUR TODOS @jennydoesthings
53. CONSIDER THE README @jennydoesthings
54. RESOLVE FAILED TESTS QUICKLY @jennydoesthings
55. IN SUMMARY
56. @JENNYDOESTHINGS WHY DO WE DESIGN TESTS TO FAIL ▸ Know the state of the system ▸ Limit failure fatigue ▸ Increase confidence in our suite ▸ Treat automation as information
57. @JENNYDOESTHINGS HOW DO WE DESIGN TESTS TO FAIL ▸ Write the test to pass on expected behavior ▸ Document your tests ▸ Expect your tests to fail ▸ Resolve failures quickly
58. @JENNYDOESTHINGS FIND ME ONLINE! ▸ jenny.bramble@gmail.com ▸ http://twitter.com/jennydoesthings ▸ https://linkedin.com/in/ jennybramble ▸ http://jennydoesthings.com ▸ Willowtree!
60. IT’S STORY TIME! Me, probably @JENNYDOESTHINGS