Automated Tests Aren't Enough

Talk given at MinneBar 14.

Recording available here: https://www.youtube.com/watch?v=Ih1-y2Vdvic

Transcript

1. Automated Tests Aren’t Enough @danlew42
7. Can we get by with fewer testers?
10. Markdown Rewrite! Experience Specs Testability https://www.flickr.com/photos/albertogp123/5843577306
12. Automated Tests Weren’t Enough Based on a true story
14. There were plenty of tests… …But they were incorrect!
15. Automation cannot prove correctness
18. Correctness Humans Subjective
20. There were plenty of tests… …But the UI was ugly!
21. Automation cannot prove goodness
23. https://www.flickr.com/photos/eugene/186551247/
25. Goodness Humans Subjective
27. There were plenty of tests… …But I had a blindspot!
29. There were plenty of tests… …Only worked on my device!
30. Automation cannot prove coverage
31. https://www.flickr.com/photos/49580580@N02/41906545634/
33. Coverage Humans Exploration
34. Automation Can Be Expensive • 100% code coverage is nearly impossible • UI testing sucks • Tests require maintenance
35. Nomenclature • Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc. • Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product. • James Bach & Michael Bolton - satisfice.com/blog/archives/856
36. ❤ Automation
37. ❤ Automation Manual +
38. Manual Testing Strengths Weaknesses • Exploratory • Tests correctness • Tests goodness • Easy to “write” and “update” • Black-box testing only • Unreliable • Time intensive • Difficult reproductions
39. Automated Checks Strengths Weaknesses • Test any layer • Repeatable • Fast • Easy reproduction • Cannot explore • Cannot prove correctness • Cannot prove goodness • Expensive to maintain
40. Automation Manual
41. So, what did we do?
43. Developer Testing Developers Testers
44. Augment Manual Testers Collaboration Tools
45. Improve Automation Architecture Easy to Write Handle Everything
46. The End! @danlew42