Great test automation does not absolve you from doing manual / exploratory testing

In recent years we've seen a movement to more test automation replacing manual/exploratory testing and even replacing dedicated testers altogether.
Many programmers don't like to manually test things. But not everything can (or should) be automated. Some things are hard if not impossible to automate, others are simply not worth the effort.
I'll illustrate this with a tale of a new system built by my team where we still found bugs in e2e testing even though we had solid unit and integration test coverage.

Great test automation does not absolve you from manual or exploratory testing. We've seen there's more automated testing than doing manual testing.
I've seen people, companies, teams move away from having dedicated testers at all, and I've even heard people say, you know, you should never do any manual testing at all. I have an opinion on that.

Now, I don't like manual testing, especially manual regression testing; I think it's very boring! So, when I got into test automation about five years ago, I was like “automate all the things!” Because automation is fun, you know?
I get to write code, listen to Spotify, and get paid to do that; it is awesome! However, I met lots and lots of awesome testers, and I learned that maybe we cannot automate everything.
Some things are just too hard, too brittle, it takes too much work; it's more work automating it than it would be to do it manually, especially if you have to maintain those tests (which I have done).

Automation can't tell you what it's like to actually use the thing. To quote my friend Lanette Creamer: "If you don't test it, your users will." And what does that say about how you value your users?

Transcript

1. @MaritvanDijk77 github.com/mlvandijk medium.com/@mlvandijk Great test automation does not absolve you from manual / exploratory testing TestBash Utrecht – 24 May 2019
2. About me • Technical Test Engineer at bol.com (NL) • Core contributor to Cucumber • Organizer European Testing Conference @MaritvanDijk77
3. Recent years •More test automation @MaritvanDijk77
4. Recent years •More test automation •No more dedicated testers @MaritvanDijk77
5. Recent years •More test automation •No more dedicated testers •No more manual testing - at all @MaritvanDijk77
6. Confession time @MaritvanDijk77
7. Confession time •I was not a tester @MaritvanDijk77
8. Confession time •I was not a tester •I got into test automation by “accident” @MaritvanDijk77
9. Automate all the things (meme) @MaritvanDijk77
10. But… •We cannot automate everything @MaritvanDijk77
11. But… •We cannot automate everything •Automation cannot tell you what it’s like to actually use it @MaritvanDijk77
12. “If you don’t test it, your users will.” Lanette Creamer @lanettecream @MaritvanDijk77
13. But… •We cannot automate everything •Automation cannot tell you what it’s like to actually use it •Unknown unknowns @MaritvanDijk77
14. But… •We cannot automate everything •Automation cannot tell you what it’s like to actually use it •Unknown unknowns •Costs vs benefit @MaritvanDijk77
15. E2E Tests are hard @MaritvanDijk77
16. E2E Tests are hard •Consistent state in several systems @MaritvanDijk77
17. E2E Tests are hard •Consistent state in several systems •Collaboration between teams / depts @MaritvanDijk77
18. Context •50 – 60 teams •Autonomous •Microservices @MaritvanDijk77
19. Our test strategy •Unit test – fast, cheap (close to the logic) @MaritvanDijk77 E2E Integration tests Unit tests
20. Our test strategy •Integration – slower (preferably in isolation) •Unit test – fast, cheap (close to the logic) @MaritvanDijk77 E2E Integration tests Unit tests
21. Our test strategy •E2E – very slow & brittle (esp. on shared environment) •Integration – slower (preferably in isolation) •Unit test – fast, cheap (close to the logic) @MaritvanDijk77 E2E Integration tests Unit tests
22. Tested components: unit tests •Functions / (small) classes •Errors / Exceptions @MaritvanDijk77 Unit Unit Unit Integration Integration
23. Tested components: integration •Integration between components (e.g. database) @MaritvanDijk77 Unit Unit Unit Integration Integration
24. Tested components: integration •Integration between components (e.g. database) •Communication between systems (e.g. other services) @MaritvanDijk77 Unit Unit Unit Integration Integration
25. Tested whole? If we have tested all components, the whole system should just work… right? @MaritvanDijk77 Unit Unit Unit Integration Integration
26. Here be bugs Well… @MaritvanDijk77 Unit Unit Unit Integration Integration
27. Example 1: Rebuilding a workflow system + Familiar with process @MaritvanDijk77
28. Example 1: Rebuilding a workflow system + Familiar with process + Close cooperation with BA @MaritvanDijk77
29. Example 1: Rebuilding a workflow system + Familiar with process + Close cooperation with BA + Unit & integration tests @MaritvanDijk77
30. Overview Workflow System Other System Other System Other System Legend “ours” not “ours” Other System Other System
31. Pair testing •Technical knowledge (dev) •Functional knowledge (test) @MaritvanDijk77
32. Benefits of pair testing + Less error prone @MaritvanDijk77
33. Benefits of pair testing + Less error prone + No time wasted on reproduction @MaritvanDijk77
34. Benefits of pair testing + Less error prone + No time wasted on reproduction + Bugs were fixed quickly @MaritvanDijk77
35. Bug collection Workflow System Other System Other System Other System Legend “ours” not “ours” Other System Other System
36. Example 2: Replacing an exchange system + Replace as is @MaritvanDijk77
37. Example 2: Replacing an exchange system + Replace as is + Close cooperation with developer @MaritvanDijk77
38. Example 2: Replacing an exchange system + Replace as is + Close cooperation with developer + Unit & integration tests @MaritvanDijk77
39. Example 2: Replacing an exchange system + Replace as is + Close cooperation with developer + Unit & integration tests + Shadow run on production (@ngurov) @MaritvanDijk77
40. Example 2: Replacing an exchange system + Replace as is + Close cooperation with developer + Unit & integration tests + Shadow run on production + Phased go-live @MaritvanDijk77
41. Overview Exchange System Other System Other System Other System Other System Other System Legend “ours” not “ours” Other System Other System
42. Other System Other System Bug collection Exchange System Other System Other System Other System Other System Other System Legend “ours” not “ours” Other System Other System
43. We cannot automate everything •What it’s like to actually use it •Unknown unknowns @MaritvanDijk77
44. TL; DR: Test your code! https://www.redbubble.com/people/hogfish (Trish Khoo) @MaritvanDijk77
45. @MaritvanDijk77 github.com/mlvandijk medium.com/@mlvandijk Great test automation does not absolve you from manual / exploratory testing TestBash Utrecht – 24 May 2019