Deliberate Testing

Deliberate Testing

In the decade since agile has gone mainstream, testing has received more emphasis—and in many organizations an automated test suite is now a prerequisite for delivery. For the first time, testing is a core activity within an integrated dev team rather than a downstream activity to be commoditized or outsourced. However, when you scratch the surface of agile testing, it seems we have lulled ourselves into a false sense of security. In an era of “automate all tests,” testing on agile projects broadly falls into two categories: automated deterministic testing provided by TDD, BDD, ATDD, and friends; and manual exploratory testing, carried out by expert testers.

However rigorously we apply these methods, entire classes of tests aren't being considered, and entire groups of stakeholders are falling through the testing net. Not all automated testing should be deterministic; not all testing should be automated. Dan North helps identify the gaps in your testing approach. By considering testing as a risk management exercise viewed through the eyes of multiple stakeholders, Dan hopes you will share his mild panic about how much testing we aren't doing.

08145ecb1ce091d9dd3c328ea2a707fb?s=128

Daniel Terhorst-North

May 06, 2015
Tweet

Transcript

  1. 8.

    Programmers wrote lots of tests! BDD SbE Unit tests Functional

    tests TDD ATDD Performance tests Load tests Soak tests Integration tests Smoke tests
  2. 9.

    BDD SbE Unit tests Functional tests TDD ATDD Performance tests

    Load tests Soak tests Integration tests Smoke tests Testers focused on the things programmers couldn’t automate
  3. 10.

    Testers focused on the things programmers couldn’t automate Exploratory testing

    TDD BDD SbE ATDD Functional tests Integration tests Unit tests Load tests Performance tests Usability testing Soak tests Smoke tests
  4. 11.

    Testing Corners, or YATQ Automated Manual Deterministic Stochastic TDD BDD

    SbE ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Exploratory testing Integration tests Usability testing
  5. 12.

    Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests

    Unit tests Load tests Smoke tests Performance tests Soak tests Exploratory testing Integration tests Usability testing Testing Corners, or YATQ Wat? Wat?
  6. 13.

    Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests

    Unit tests Load tests Smoke tests Performance tests Soak tests Integration tests Usability testing User journey testing A/B testing Fuzzing Property-based testing Data mother Resilience Documentation testing Exploratory testing Testing Corners, or YATQ Wat?
  7. 14.

    Risk Plane B A C F G D H E

    Im pact 80% ? Likelihood
  8. 15.

    Risk Plane B A C F D G H E

    Im pact 80% ? F D 80% ? 80% ? Likelihood
  9. 16.

    Risk Plane B A C F D G H E

    80% ? Im pact Likelihood
  10. 17.

    Risk Planes B A C F D G H E

    Im pact Likelihood Context
  11. 18.

    Risk Planes B A C F D G H E

    Im pact Likelihood D Context Functional correctness
  12. 19.

    Risk Planes B A C F D G H E

    Im pact Likelihood Operational stability A Context
  13. 20.

    Risk Planes B A C F D G H E

    Im pact Likelihood Regulatory compliance C G Context
  14. 23.

    Testing Corners revisited Automated Manual Deterministic Stochastic TDD BDD SbE

    ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Integration tests Usability testing User journey testing A/B testing Fuzzing Property-based testing Data mother Chaos Monkey Documentation testing Exploratory testing
  15. 24.

    Testing Corners revisited Automated Manual Deterministic Stochastic TDD BDD SbE

    ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Integration tests Usability testing User journey testing A/B testing Fuzzing Property-based testing Data mother Chaos Monkey Documentation testing Exploratory testing Compliance Failover Forensic Diagnostic Release Alerting Penetration SQL Injection DoS Accessibility Safety Fire drills XSS DNS
  16. 25.

    Deliberate testing in an agile world Identify all the stakeholders

    Understand their risk appetite Understand their constraints Study the application Understand its components and flows Explore each Risk Plane with its stakeholder Test according to the overall risk profile