FFS - Great test automation does not absolve you from manual / exploratory testing

In recent years we've seen a movement to more test automation replacing manual/exploratory testing and even replacing dedicated testers altogether.
Many programmers don't like to manually test things. But not everything can (or should) be automated. Some things are hard if not impossible to automate, others are simply not worth the effort.
I'll illustrate this with a tale of a new system built by my team where we still found bugs in e2e testing even though we had solid unit and integration test coverage.

Transcript

1. @MaritvanDijk77

   github.com/mlvandijk

   medium.com/@mlvandijk
   FFS - Great test automation
   does not absolve you from
   manual / exploratory testing
   
   

   View Slide

2. Recent years
   •More test automation
   •No more dedicated testers
   •No more manual testing - at all!
   @MaritvanDijk77
   
   

   View Slide

3. Automate all the things
   (meme)
   @MaritvanDijk77
   

   View Slide

4. But…
   •We cannot automate everything
   (we shouldn’t want to)
   •Automation cannot tell you what it’s
   like to actually use it
   @MaritvanDijk77
   

   View Slide

5. “If you don’t test it, your users will.”
   ~ @lanettecream
   @MaritvanDijk77
   

   View Slide

6. Tested components
   @MaritvanDijk77
   Unit Unit Unit
   Integration
   Integration
   

   View Slide

7. Tested components
   @MaritvanDijk77
   Unit Unit Unit
   Integration
   Integration
   

   View Slide

8. Tested whole?
   @MaritvanDijk77
   Unit Unit Unit
   Integration
   Integration
   

   View Slide

9. Here be bugs
   Tested whole?
   @MaritvanDijk77
   Unit Unit Unit
   Integration
   Integration
   
   
   
   
   
   
   
   
   
   
   
   
   

   View Slide

10. Example: Rebuilding a workflow system
    + Familiar with process
    + Close cooperation with BA
    + Unit tests
    + Integration tests
    @MaritvanDijk77
    

    View Slide

11. Overview
    Workflow
    System
    Other
    System
    Other
    System
    Other
    System
    

    View Slide

12. E2E Test
    •Notoriously difficult:
    •Consistent state in several systems
    •Collaboration between teams/depts
    •Hard to automate / maintain
    @MaritvanDijk77
    
    

    View Slide

13. Pair testing
    •Technical knowledge (dev)
    •Functional knowledge (test)
    •Less error prone
    •No time wasted on reproduction
    •Quickly fixed
    @MaritvanDijk77
    

    View Slide

14. Test your code!
    https://www.redbubble.com/people/hogfish
    @MaritvanDijk77
    
    

    View Slide

15. @MaritvanDijk77

    github.com/mlvandijk

    medium.com/@mlvandijk
    FFS - Great test automation
    does not absolve you from
    manual / exploratory testing
    
    

    View Slide