Automated Tests Aren't Enough

Automated Tests Aren't Enough

Talk given at MinneBar 14.

Recording available here: https://www.youtube.com/watch?v=Ih1-y2Vdvic

D225ebf0faa666ac7655cc7e4689283c?s=128

Daniel Lew

April 27, 2019
Tweet

Transcript

  1. Automated Tests Aren’t Enough @danlew42

  2. None
  3. None
  4. None
  5. None
  6. None
  7. Can we get by with fewer testers?

  8. None
  9. None
  10. Markdown Rewrite! Experience Specs Testability https://www.flickr.com/photos/albertogp123/5843577306

  11. None
  12. Automated Tests Weren’t Enough Based on a true story

  13. None
  14. There were plenty of tests… …But they were incorrect!

  15. Automation cannot prove correctness

  16. None
  17. None
  18. Correctness Humans Subjective

  19. None
  20. There were plenty of tests… …But the UI was ugly!

  21. Automation cannot prove goodness

  22. None
  23. https://www.flickr.com/photos/eugene/186551247/

  24. None
  25. Goodness Humans Subjective

  26. None
  27. There were plenty of tests… …But I had a blindspot!

  28. None
  29. There were plenty of tests… …Only worked on my device!

  30. Automation cannot prove coverage

  31. https://www.flickr.com/photos/49580580@N02/41906545634/

  32. None
  33. Coverage Humans Exploration

  34. Automation Can Be Expensive • 100% code coverage is nearly

    impossible • UI testing sucks • Tests require maintenance
  35. Nomenclature • Testing is the process of evaluating a product

    by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc. • Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product. • James Bach & Michael Bolton - satisfice.com/blog/archives/856
  36. ❤ Automation

  37. ❤ Automation Manual +

  38. Manual Testing Strengths Weaknesses • Exploratory • Tests correctness •

    Tests goodness • Easy to “write” and “update” • Black-box testing only • Unreliable • Time intensive • Difficult reproductions
  39. Automated Checks Strengths Weaknesses • Test any layer • Repeatable

    • Fast • Easy reproduction • Cannot explore • Cannot prove correctness • Cannot prove goodness • Expensive to maintain
  40. Automation Manual

  41. So, what did we do?

  42. None
  43. Developer Testing Developers Testers

  44. Augment Manual Testers Collaboration Tools

  45. Improve Automation Architecture Easy to Write Handle Everything

  46. The End! @danlew42