End-To-End Testing Considered Harmful

622051376c3b381ef1c786036011112d?s=47 Steve Smith
September 28, 2016

End-To-End Testing Considered Harmful

End-To-End Testing - using a large number of end-to-end tests to test a service and its unowned dependent services together - remains the norm in IT, especially in large enterprise organisations. But what if you could demonstrate its impact upon Continuous Delivery, debunk its perceived benefits, and replace it with a viable alternative?

End-To-End Testing may appear to offer a lot of test coverage for relatively little effort, but it invariably results in long lead times and astronomical maintenance costs. It is a significant impediment to Continuous Delivery, and if an organisation is to reduce its time to market a Continuous Testing portfolio emphasising fast, reliable test feedback is required.

In this talk Steve Smith will describe how End-To-End Testing has hindered Continuous Delivery adoption on a large scale UK Government project, why End-To-End Testing often seems like a good idea but ends up as a bad idea, and how to establish a Continuous Testing portfolio including techniques such as contract testing, smoke testing, and production monitoring.

622051376c3b381ef1c786036011112d?s=128

Steve Smith

September 28, 2016
Tweet

Transcript

  1. End-To-End Testing Considered Harmful Steve Smith @SteveSmithCD hello@continuousdeliveryconsulting.com

  2. Steve Smith Continuous Delivery consultant and trainer Author of “Measuring

    Continuous Delivery” Co-author of “Build Quality In” b: www.continuousdeliveryconsulting.com e: hello@continuousdeliveryconsulting.com t: @SteveSmithCD Associated with
  3. End-To-End Testing

  4. Home Move

  5. A Home Move Story

  6. A Home Move Story

  7. End-To-End Testing

  8. End-To-End Testing

  9. End-To-End Testing

  10. High Coverage?

  11. High Coverage?

  12. Low Coverage

  13. Low Coverage

  14. End-To-End Testing

  15. Low Investment?

  16. Low Investment?

  17. High Investment

  18. High Investment – Response Times

  19. High Investment – Non-determinism

  20. End-To-End Testing

  21. Continuous Testing

  22. High Coverage

  23. High Coverage

  24. Low Investment

  25. Continuous Testing

  26. Unit Testing Name Test Scope Quantity Frequency Duration Env Unit

    Testing Intent vs. impl Units of service 100 → 1000+ Per build < 30s total • Local • Build
  27. Name Test Scope Quantity Frequency Duration Env Acceptance Testing Impl

    vs. reqs Service 10 → 100+ Per build < 10m total • Local • Build Acceptance Testing
  28. Name Test Scope Quantity Frequency Duration Env Exploratory Testing Explore

    impl vs. reqs Service 10 → 100+ Per build Timebox per test • Local • 3rd party Exploratory Testing
  29. Name Test Scope Quantity Frequency Duration Env Contract Testing 3rd

    party impl vs. reqs 3rd party service(s) ~20 Per 3rd party deploy < 1m total • 3rd party • Build Contract Testing
  30. Contract Testing

  31. Name Test Scope Quantity Frequency Duration Env Smoke Testing Impl

    vs. reqs Entire system ~5 Per deploy < 5m total All Smoke Testing
  32. Monitoring Name Test Scope Quantity Frequency Duration Env Monitoring Impl

    vs. reqs Entire system 10 → 100 Always < 10s total All Monitoring
  33. Continuous Testing Name Test Scope Quantity Frequency Duration Env Unit

    Testing Intent vs. impl Units of service 100 → 1000+ Per build < 30s total • Local • Build Acceptance Testing Impl vs. reqs Service 10 → 100+ Per build < 10m total • Local • Build Exploratory Testing Explore impl vs. reqs Service 10 → 100+ Per build Timebox per test • Local • 3rd party Contract Testing 3rd party impl vs. reqs 3rd party service(s) ~20 Per 3rd party deploy < 1m total • 3rd party • Build Smoke Testing Impl vs. reqs Entire system ~5 Per deploy < 5m total All Monitoring Impl vs. reqs Entire system 10 → 100+ Always < 10s total All
  34. Caveats

  35. End-To-End Testing “Not only won’t system testing catch all the

    bugs, but it will take longer and cost more – more than you save by skipping effective acceptance testing” Gerald Weinberg End-To-End Testing looks good, but is invariably bad Continuous Delivery needs rapid, reliable feedback Continuous Testing is a worthwhile investment
  36. End-To-End Testing Considered Harmful Steve Smith @SteveSmithCD hello@continuousdeliveryconsulting.com