$30 off During Our Annual Pro Sale. View Details »

Great test automation does not absolve you from doing manual / exploratory testing

Great test automation does not absolve you from doing manual / exploratory testing

In recent years we've seen a movement to more test automation replacing manual/exploratory testing and even replacing dedicated testers altogether.
Many programmers don't like to manually test things. But not everything can (or should) be automated. Some things are hard if not impossible to automate, others are simply not worth the effort.
I'll illustrate this with a tale of a new system built by my team where we still found bugs in e2e testing even though we had solid unit and integration test coverage.

Great test automation does not absolve you from manual or exploratory testing. We've seen there's more automated testing than doing manual testing.
I've seen people, companies, teams move away from having dedicated testers at all, and I've even heard people say, you know, you should never do any manual testing at all. I have an opinion on that.

Now, I don't like manual testing, especially manual regression testing; I think it's very boring! So, when I got into test automation about five years ago, I was like “automate all the things!” Because automation is fun, you know?
I get to write code, listen to Spotify, and get paid to do that; it is awesome! However, I met lots and lots of awesome testers, and I learned that maybe we cannot automate everything.
Some things are just too hard, too brittle, it takes too much work; it's more work automating it than it would be to do it manually, especially if you have to maintain those tests (which I have done).

Automation can't tell you what it's like to actually use the thing. To quote my friend Lanette Creamer: "If you don't test it, your users will." And what does that say about how you value your users?

Marit van Dijk

May 24, 2019
Tweet

More Decks by Marit van Dijk

Other Decks in Technology

Transcript

  1. @MaritvanDijk77
    github.com/mlvandijk
    medium.com/@mlvandijk
    Great test automation does not
    absolve you from manual /
    exploratory testing
    TestBash Utrecht – 24 May 2019

    View Slide

  2. About me
    • Technical Test Engineer at bol.com (NL)
    • Core contributor to Cucumber
    • Organizer European Testing Conference
    @MaritvanDijk77

    View Slide

  3. Recent years
    •More test automation
    @MaritvanDijk77

    View Slide

  4. Recent years
    •More test automation
    •No more dedicated testers
    @MaritvanDijk77

    View Slide

  5. Recent years
    •More test automation
    •No more dedicated testers
    •No more manual testing - at all
    @MaritvanDijk77

    View Slide

  6. Confession time
    @MaritvanDijk77

    View Slide

  7. Confession time
    •I was not a tester
    @MaritvanDijk77

    View Slide

  8. Confession time
    •I was not a tester
    •I got into test automation by “accident”
    @MaritvanDijk77

    View Slide

  9. Automate all the things
    (meme)
    @MaritvanDijk77

    View Slide

  10. But…
    •We cannot automate everything
    @MaritvanDijk77

    View Slide

  11. But…
    •We cannot automate everything
    •Automation cannot tell you
    what it’s like to actually use it
    @MaritvanDijk77

    View Slide

  12. “If you don’t test it,
    your users will.”
    Lanette Creamer
    @lanettecream
    @MaritvanDijk77

    View Slide

  13. But…
    •We cannot automate everything
    •Automation cannot tell you
    what it’s like to actually use it
    •Unknown unknowns
    @MaritvanDijk77

    View Slide

  14. But…
    •We cannot automate everything
    •Automation cannot tell you
    what it’s like to actually use it
    •Unknown unknowns
    •Costs vs benefit
    @MaritvanDijk77

    View Slide

  15. E2E Tests are hard
    @MaritvanDijk77

    View Slide

  16. E2E Tests are hard
    •Consistent state in several systems
    @MaritvanDijk77

    View Slide

  17. E2E Tests are hard
    •Consistent state in several systems
    •Collaboration between teams / depts
    @MaritvanDijk77

    View Slide

  18. Context
    •50 – 60 teams
    •Autonomous
    •Microservices
    @MaritvanDijk77

    View Slide

  19. Our test strategy
    •Unit test – fast, cheap
    (close to the logic)
    @MaritvanDijk77
    E2E
    Integration
    tests
    Unit tests

    View Slide

  20. Our test strategy
    •Integration – slower
    (preferably in isolation)
    •Unit test – fast, cheap
    (close to the logic)
    @MaritvanDijk77
    E2E
    Integration
    tests
    Unit tests

    View Slide

  21. Our test strategy
    •E2E – very slow & brittle
    (esp. on shared environment)
    •Integration – slower
    (preferably in isolation)
    •Unit test – fast, cheap
    (close to the logic)
    @MaritvanDijk77
    E2E
    Integration
    tests
    Unit tests

    View Slide

  22. Tested components: unit tests
    •Functions / (small) classes
    •Errors / Exceptions
    @MaritvanDijk77
    Unit Unit Unit
    Integration
    Integration

    View Slide

  23. Tested components: integration
    •Integration between components (e.g. database)
    @MaritvanDijk77
    Unit Unit Unit
    Integration
    Integration

    View Slide

  24. Tested components: integration
    •Integration between components (e.g. database)
    •Communication between systems (e.g. other services)
    @MaritvanDijk77
    Unit Unit Unit
    Integration
    Integration

    View Slide

  25. Tested whole?
    If we have tested all components,
    the whole system should just work… right?
    @MaritvanDijk77
    Unit Unit Unit
    Integration
    Integration

    View Slide

  26. Here be bugs
    Well…
    @MaritvanDijk77
    Unit Unit Unit
    Integration
    Integration

    View Slide

  27. Example 1: Rebuilding a workflow system
    + Familiar with process
    @MaritvanDijk77

    View Slide

  28. Example 1: Rebuilding a workflow system
    + Familiar with process
    + Close cooperation with BA
    @MaritvanDijk77

    View Slide

  29. Example 1: Rebuilding a workflow system
    + Familiar with process
    + Close cooperation with BA
    + Unit & integration tests
    @MaritvanDijk77

    View Slide

  30. Overview
    Workflow
    System
    Other
    System
    Other
    System
    Other
    System
    Legend
    “ours”
    not “ours”
    Other
    System
    Other
    System

    View Slide

  31. Pair testing
    •Technical knowledge (dev)
    •Functional knowledge (test)
    @MaritvanDijk77

    View Slide

  32. Benefits of pair testing
    + Less error prone
    @MaritvanDijk77

    View Slide

  33. Benefits of pair testing
    + Less error prone
    + No time wasted on reproduction
    @MaritvanDijk77

    View Slide

  34. Benefits of pair testing
    + Less error prone
    + No time wasted on reproduction
    + Bugs were fixed quickly
    @MaritvanDijk77

    View Slide

  35. Bug collection
    Workflow
    System
    Other
    System
    Other
    System
    Other
    System
    Legend
    “ours”
    not “ours”
    Other
    System
    Other
    System

    View Slide

  36. Example 2: Replacing an exchange system
    + Replace as is
    @MaritvanDijk77

    View Slide

  37. Example 2: Replacing an exchange system
    + Replace as is
    + Close cooperation with developer
    @MaritvanDijk77

    View Slide

  38. Example 2: Replacing an exchange system
    + Replace as is
    + Close cooperation with developer
    + Unit & integration tests
    @MaritvanDijk77

    View Slide

  39. Example 2: Replacing an exchange system
    + Replace as is
    + Close cooperation with developer
    + Unit & integration tests
    + Shadow run on production (@ngurov)
    @MaritvanDijk77

    View Slide

  40. Example 2: Replacing an exchange system
    + Replace as is
    + Close cooperation with developer
    + Unit & integration tests
    + Shadow run on production
    + Phased go-live
    @MaritvanDijk77

    View Slide

  41. Overview
    Exchange
    System
    Other
    System
    Other
    System
    Other
    System
    Other
    System
    Other
    System
    Legend
    “ours”
    not “ours”
    Other
    System
    Other
    System

    View Slide

  42. Other
    System
    Other
    System
    Bug collection
    Exchange
    System
    Other
    System
    Other
    System
    Other
    System
    Other
    System
    Other
    System
    Legend
    “ours”
    not “ours”
    Other
    System
    Other
    System

    View Slide

  43. We cannot automate everything
    •What it’s like to actually use it
    •Unknown unknowns
    @MaritvanDijk77

    View Slide

  44. TL; DR: Test your code!
    https://www.redbubble.com/people/hogfish (Trish Khoo)
    @MaritvanDijk77

    View Slide

  45. @MaritvanDijk77
    github.com/mlvandijk
    medium.com/@mlvandijk
    Great test automation does not
    absolve you from manual /
    exploratory testing
    TestBash Utrecht – 24 May 2019

    View Slide