Automated Tests Aren't Enough

Talk given at MinneBar 14.

Recording available here: https://www.youtube.com/watch?v=Ih1-y2Vdvic

Transcript

1. Automated Tests Aren’t Enough
   @danlew42
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. Can we get by with fewer
   testers?
   

   View Slide

8. View Slide

9. View Slide

10. Markdown Rewrite!
    Experience
    Specs Testability
    https://www.flickr.com/photos/albertogp123/5843577306
    

    View Slide

11. View Slide

12. Automated Tests Weren’t Enough
    Based on a true story
    

    View Slide

13. View Slide

14. There were plenty of tests…
    …But they were incorrect!
    

    View Slide

15. Automation cannot prove
    correctness
    

    View Slide

16. View Slide

17. View Slide

18. Correctness
    Humans
    Subjective
    

    View Slide

19. View Slide

20. There were plenty of tests…
    …But the UI was ugly!
    

    View Slide

21. Automation cannot prove
    goodness
    

    View Slide

22. View Slide

23. https://www.flickr.com/photos/eugene/186551247/
    

    View Slide

24. View Slide

25. Goodness
    Humans
    Subjective
    

    View Slide

26. View Slide

27. There were plenty of tests…
    …But I had a blindspot!
    

    View Slide

28. View Slide

29. There were plenty of tests…
    …Only worked on my device!
    

    View Slide

30. Automation cannot prove
    coverage
    

    View Slide

31. https://www.flickr.com/photos/49580580@N02/41906545634/
    

    View Slide

32. View Slide

33. Coverage
    Humans
    Exploration
    

    View Slide

34. Automation Can Be Expensive
    • 100% code coverage is nearly impossible
    
    • UI testing sucks
    
    • Tests require maintenance
    

    View Slide

35. Nomenclature
    • Testing is the process of evaluating a product by learning about it through
    exploration and experimentation, which includes to some degree:
    questioning, study, modeling, observation, inference, etc.
    
    • Checking is the process of making evaluations by applying algorithmic
    decision rules to specific observations of a product.
    
    • James Bach & Michael Bolton - satisfice.com/blog/archives/856
    

    View Slide

36. ❤
    Automation
    

    View Slide

37. ❤
    Automation Manual
    +
    

    View Slide

38. Manual Testing
    Strengths Weaknesses
    • Exploratory
    
    • Tests correctness
    
    • Tests goodness
    
    • Easy to “write” and “update”
    • Black-box testing only
    
    • Unreliable
    
    • Time intensive
    
    • Difficult reproductions
    

    View Slide

39. Automated Checks
    Strengths Weaknesses
    • Test any layer
    
    • Repeatable
    
    • Fast
    
    • Easy reproduction
    • Cannot explore
    
    • Cannot prove correctness
    
    • Cannot prove goodness
    
    • Expensive to maintain
    

    View Slide

40. Automation Manual
    

    View Slide

41. So, what did we do?
    

    View Slide

42. View Slide

43. Developer Testing
    Developers Testers
    

    View Slide

44. Augment Manual Testers
    Collaboration
    Tools
    

    View Slide

45. Improve Automation Architecture
    Easy to Write Handle Everything
    

    View Slide

46. The End!
    @danlew42
    

    View Slide