$30 off During Our Annual Pro Sale. View Details »

Automated Tests Aren't Enough

Automated Tests Aren't Enough

Talk given at MinneBar 14.

Recording available here: https://www.youtube.com/watch?v=Ih1-y2Vdvic

Daniel Lew

April 27, 2019

More Decks by Daniel Lew

Other Decks in Programming


  1. Automated Tests Aren’t Enough

    View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. Can we get by with fewer

    View Slide

  8. View Slide

  9. View Slide

  10. Markdown Rewrite!
    Specs Testability

    View Slide

  11. View Slide

  12. Automated Tests Weren’t Enough
    Based on a true story

    View Slide

  13. View Slide

  14. There were plenty of tests…
    …But they were incorrect!

    View Slide

  15. Automation cannot prove

    View Slide

  16. View Slide

  17. View Slide

  18. Correctness

    View Slide

  19. View Slide

  20. There were plenty of tests…
    …But the UI was ugly!

    View Slide

  21. Automation cannot prove

    View Slide

  22. View Slide

  23. https://www.flickr.com/photos/eugene/186551247/

    View Slide

  24. View Slide

  25. Goodness

    View Slide

  26. View Slide

  27. There were plenty of tests…
    …But I had a blindspot!

    View Slide

  28. View Slide

  29. There were plenty of tests…
    …Only worked on my device!

    View Slide

  30. Automation cannot prove

    View Slide

  31. https://www.flickr.com/photos/49580580@N02/41906545634/

    View Slide

  32. View Slide

  33. Coverage

    View Slide

  34. Automation Can Be Expensive
    • 100% code coverage is nearly impossible

    • UI testing sucks

    • Tests require maintenance

    View Slide

  35. Nomenclature
    • Testing is the process of evaluating a product by learning about it through
    exploration and experimentation, which includes to some degree:
    questioning, study, modeling, observation, inference, etc.

    • Checking is the process of making evaluations by applying algorithmic
    decision rules to specific observations of a product.

    • James Bach & Michael Bolton - satisfice.com/blog/archives/856

    View Slide

  36. Automation

    View Slide

  37. Automation Manual

    View Slide

  38. Manual Testing
    Strengths Weaknesses
    • Exploratory

    • Tests correctness

    • Tests goodness

    • Easy to “write” and “update”
    • Black-box testing only

    • Unreliable

    • Time intensive

    • Difficult reproductions

    View Slide

  39. Automated Checks
    Strengths Weaknesses
    • Test any layer

    • Repeatable

    • Fast

    • Easy reproduction
    • Cannot explore

    • Cannot prove correctness

    • Cannot prove goodness

    • Expensive to maintain

    View Slide

  40. Automation Manual

    View Slide

  41. So, what did we do?

    View Slide

  42. View Slide

  43. Developer Testing
    Developers Testers

    View Slide

  44. Augment Manual Testers

    View Slide

  45. Improve Automation Architecture
    Easy to Write Handle Everything

    View Slide

  46. The End!

    View Slide