Deliberate Testing

In the decade since agile has gone mainstream, testing has received more emphasis—and in many organizations an automated test suite is now a prerequisite for delivery. For the first time, testing is a core activity within an integrated dev team rather than a downstream activity to be commoditized or outsourced. However, when you scratch the surface of agile testing, it seems we have lulled ourselves into a false sense of security. In an era of “automate all tests,” testing on agile projects broadly falls into two categories: automated deterministic testing provided by TDD, BDD, ATDD, and friends; and manual exploratory testing, carried out by expert testers.

However rigorously we apply these methods, entire classes of tests aren't being considered, and entire groups of stakeholders are falling through the testing net. Not all automated testing should be deterministic; not all testing should be automated. Dan North helps identify the gaps in your testing approach. By considering testing as a risk management exercise viewed through the eyes of multiple stakeholders, Dan hopes you will share his mild panic about how much testing we aren't doing.

Transcript

1. Deliberate Testing in an Agile World Dan North @tastapod
2. Software delivery used to be linear
3. Agile brought people together
4. Agile brought people together
5. Programmers thought testing was cool!
6. Programmers thought TDD was cool!
7. Programmers thought TDD was testing!
8. Programmers wrote lots of tests! BDD SbE Unit tests Functional tests TDD ATDD Performance tests Load tests Soak tests Integration tests Smoke tests
9. BDD SbE Unit tests Functional tests TDD ATDD Performance tests Load tests Soak tests Integration tests Smoke tests Testers focused on the things programmers couldn’t automate
10. Testers focused on the things programmers couldn’t automate Exploratory testing TDD BDD SbE ATDD Functional tests Integration tests Unit tests Load tests Performance tests Usability testing Soak tests Smoke tests
11. Testing Corners, or YATQ Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Exploratory testing Integration tests Usability testing
12. Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Exploratory testing Integration tests Usability testing Testing Corners, or YATQ Wat? Wat?
13. Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Integration tests Usability testing User journey testing A/B testing Fuzzing Property-based testing Data mother Resilience Documentation testing Exploratory testing Testing Corners, or YATQ Wat?
14. Risk Plane B A C F G D H E Im pact 80% ? Likelihood
15. Risk Plane B A C F D G H E Im pact 80% ? F D 80% ? 80% ? Likelihood
16. Risk Plane B A C F D G H E 80% ? Im pact Likelihood
17. Risk Planes B A C F D G H E Im pact Likelihood Context
18. Risk Planes B A C F D G H E Im pact Likelihood D Context Functional correctness
19. Risk Planes B A C F D G H E Im pact Likelihood Operational stability A Context
20. Risk Planes B A C F D G H E Im pact Likelihood Regulatory compliance C G Context
21. Risk Planes Im pact Likelihood Each context has its own risk profile Context
22. Risk Planes Im pact Likelihood Stakeholder Each stakeholder has their own risk appetite
23. Testing Corners revisited Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Integration tests Usability testing User journey testing A/B testing Fuzzing Property-based testing Data mother Chaos Monkey Documentation testing Exploratory testing
24. Testing Corners revisited Automated Manual Deterministic Stochastic TDD BDD SbE ATDD Functional tests Unit tests Load tests Smoke tests Performance tests Soak tests Integration tests Usability testing User journey testing A/B testing Fuzzing Property-based testing Data mother Chaos Monkey Documentation testing Exploratory testing Compliance Failover Forensic Diagnostic Release Alerting Penetration SQL Injection DoS Accessibility Safety Fire drills XSS DNS
25. Deliberate testing in an agile world Identify all the stakeholders Understand their risk appetite Understand their constraints Study the application Understand its components and flows Explore each Risk Plane with its stakeholder Test according to the overall risk profile
26. …is nuanced and multi-dimensional Deliberate testing in an agile world
27. Fortunately, so are deliberate testers
28. Thanks for listening Dan North http:/ /dannorth.net @tastapod