Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Great test automation does not absolve you from doing manual / exploratory testing

Great test automation does not absolve you from doing manual / exploratory testing

In recent years we've seen a movement to more test automation replacing manual/exploratory testing and even replacing dedicated testers altogether.
Many programmers don't like to manually test things. But not everything can (or should) be automated. Some things are hard if not impossible to automate, others are simply not worth the effort.
I'll illustrate this with a tale of a new system built by my team where we still found bugs in e2e testing even though we had solid unit and integration test coverage.

Great test automation does not absolve you from manual or exploratory testing. We've seen there's more automated testing than doing manual testing.
I've seen people, companies, teams move away from having dedicated testers at all, and I've even heard people say, you know, you should never do any manual testing at all. I have an opinion on that.

Now, I don't like manual testing, especially manual regression testing; I think it's very boring! So, when I got into test automation about five years ago, I was like “automate all the things!” Because automation is fun, you know?
I get to write code, listen to Spotify, and get paid to do that; it is awesome! However, I met lots and lots of awesome testers, and I learned that maybe we cannot automate everything.
Some things are just too hard, too brittle, it takes too much work; it's more work automating it than it would be to do it manually, especially if you have to maintain those tests (which I have done).

Automation can't tell you what it's like to actually use the thing. To quote my friend Lanette Creamer: "If you don't test it, your users will." And what does that say about how you value your users?

62b3ae0dcd8e12a5eedd1fbe417bc6c8?s=128

Marit van Dijk

May 24, 2019
Tweet

Transcript

  1. @MaritvanDijk77 github.com/mlvandijk medium.com/@mlvandijk Great test automation does not absolve you

    from manual / exploratory testing TestBash Utrecht – 24 May 2019
  2. About me • Technical Test Engineer at bol.com (NL) •

    Core contributor to Cucumber • Organizer European Testing Conference @MaritvanDijk77
  3. Recent years •More test automation @MaritvanDijk77

  4. Recent years •More test automation •No more dedicated testers @MaritvanDijk77

  5. Recent years •More test automation •No more dedicated testers •No

    more manual testing - at all @MaritvanDijk77
  6. Confession time @MaritvanDijk77

  7. Confession time •I was not a tester @MaritvanDijk77

  8. Confession time •I was not a tester •I got into

    test automation by “accident” @MaritvanDijk77
  9. Automate all the things (meme) @MaritvanDijk77

  10. But… •We cannot automate everything @MaritvanDijk77

  11. But… •We cannot automate everything •Automation cannot tell you what

    it’s like to actually use it @MaritvanDijk77
  12. “If you don’t test it, your users will.” Lanette Creamer

    @lanettecream @MaritvanDijk77
  13. But… •We cannot automate everything •Automation cannot tell you what

    it’s like to actually use it •Unknown unknowns @MaritvanDijk77
  14. But… •We cannot automate everything •Automation cannot tell you what

    it’s like to actually use it •Unknown unknowns •Costs vs benefit @MaritvanDijk77
  15. E2E Tests are hard @MaritvanDijk77

  16. E2E Tests are hard •Consistent state in several systems @MaritvanDijk77

  17. E2E Tests are hard •Consistent state in several systems •Collaboration

    between teams / depts @MaritvanDijk77
  18. Context •50 – 60 teams •Autonomous •Microservices @MaritvanDijk77

  19. Our test strategy •Unit test – fast, cheap (close to

    the logic) @MaritvanDijk77 E2E Integration tests Unit tests
  20. Our test strategy •Integration – slower (preferably in isolation) •Unit

    test – fast, cheap (close to the logic) @MaritvanDijk77 E2E Integration tests Unit tests
  21. Our test strategy •E2E – very slow & brittle (esp.

    on shared environment) •Integration – slower (preferably in isolation) •Unit test – fast, cheap (close to the logic) @MaritvanDijk77 E2E Integration tests Unit tests
  22. Tested components: unit tests •Functions / (small) classes •Errors /

    Exceptions @MaritvanDijk77 Unit Unit Unit Integration Integration
  23. Tested components: integration •Integration between components (e.g. database) @MaritvanDijk77 Unit

    Unit Unit Integration Integration
  24. Tested components: integration •Integration between components (e.g. database) •Communication between

    systems (e.g. other services) @MaritvanDijk77 Unit Unit Unit Integration Integration
  25. Tested whole? If we have tested all components, the whole

    system should just work… right? @MaritvanDijk77 Unit Unit Unit Integration Integration
  26. Here be bugs Well… @MaritvanDijk77 Unit Unit Unit Integration Integration

  27. Example 1: Rebuilding a workflow system + Familiar with process

    @MaritvanDijk77
  28. Example 1: Rebuilding a workflow system + Familiar with process

    + Close cooperation with BA @MaritvanDijk77
  29. Example 1: Rebuilding a workflow system + Familiar with process

    + Close cooperation with BA + Unit & integration tests @MaritvanDijk77
  30. Overview Workflow System Other System Other System Other System Legend

    “ours” not “ours” Other System Other System
  31. Pair testing •Technical knowledge (dev) •Functional knowledge (test) @MaritvanDijk77

  32. Benefits of pair testing + Less error prone @MaritvanDijk77

  33. Benefits of pair testing + Less error prone + No

    time wasted on reproduction @MaritvanDijk77
  34. Benefits of pair testing + Less error prone + No

    time wasted on reproduction + Bugs were fixed quickly @MaritvanDijk77
  35. Bug collection Workflow System Other System Other System Other System

    Legend “ours” not “ours” Other System Other System
  36. Example 2: Replacing an exchange system + Replace as is

    @MaritvanDijk77
  37. Example 2: Replacing an exchange system + Replace as is

    + Close cooperation with developer @MaritvanDijk77
  38. Example 2: Replacing an exchange system + Replace as is

    + Close cooperation with developer + Unit & integration tests @MaritvanDijk77
  39. Example 2: Replacing an exchange system + Replace as is

    + Close cooperation with developer + Unit & integration tests + Shadow run on production (@ngurov) @MaritvanDijk77
  40. Example 2: Replacing an exchange system + Replace as is

    + Close cooperation with developer + Unit & integration tests + Shadow run on production + Phased go-live @MaritvanDijk77
  41. Overview Exchange System Other System Other System Other System Other

    System Other System Legend “ours” not “ours” Other System Other System
  42. Other System Other System Bug collection Exchange System Other System

    Other System Other System Other System Other System Legend “ours” not “ours” Other System Other System
  43. We cannot automate everything •What it’s like to actually use

    it •Unknown unknowns @MaritvanDijk77
  44. TL; DR: Test your code! https://www.redbubble.com/people/hogfish (Trish Khoo) @MaritvanDijk77

  45. @MaritvanDijk77 github.com/mlvandijk medium.com/@mlvandijk Great test automation does not absolve you

    from manual / exploratory testing TestBash Utrecht – 24 May 2019