The Business Case for ATDD Testing

Software development teams are under increasing pressure to deliver applications cheaper, faster, and better. As a result, many groups are using some kind of iterative or "agile" development methodology, all with the promise to achieve software faster, better, cheaper. However, shorter product releases cycles mean greater opportunity and need for regression testing. This creates resource management challenges as the burden of regression testing grows and competes with resources needed to test new feature functionality. As a remedy for this situation, many have proposed the use of test automation in the form of Acceptance Test Driven Development methods. Many managers, and even front line testers, tend to resist this approach. How can we diffuse the tension so that management will understand the necessary investments and the testers on the front line will take up the challenge with enthusiasm?

Transcript

1. www.cohesion.com | 877.774.3001 By Joseph Ours The Business Case for Automated ATDD
2. www.cohesion.com | 877.774.3001 Introducing Joseph Ours • About Joseph • Owns development and delivery of Cohesion's Quality Assurance and Testing Services • More than 15 years of IT experience spanning several industries and roles • Notable success managing multi- million dollar initiatives while introducing innovative solutions for Fortune 1000 companies, resulting in improved quality, increased client satisfaction, cost avoidance, on-time delivery, and establishing a true team culture • Contacting Me • @justjoehere • Joseph.ours@gmail.com or jours@cohesion.com • 614.423.5253 • www.linkedin.com/in/josephours
3. www.cohesion.com | 877.774.3001 • Things NOT covered • Challenges of Today • Definition of ATDD • ATDD overview for waterfall • ATDD overview for agile • ATDD and Automation • ROI methods • ROI calculations • Quick tool comparison • Challenges to implementing ATDD • Picking the right time and place Agenda
4. www.cohesion.com | 877.774.3001 • This is not a lecture on the details of HOW to implement ATDD • This is not a technical lecture on automation • This assumes you have heard of the basics of ATDD • This lumps BDD with ATDD as a single solution Important things NOT covered
5. www.cohesion.com | 877.774.3001 Different interpretation of requirements – lack of common understanding Ambiguous and/or Incomplete requirements Inadequate test coverage (permutations of options) Incorrect prioritization and testing of requirements Disconnected team members – lack of collaboration Common Challenges of Today
6. www.cohesion.com | 877.774.3001 • Acceptance Test • User’s point of view • External to the system • Rooted in functional and black box testing techniques: • Verify outputs based on inputs • Follow typical business process flows • Evaluate events and state changes • Primarily via external interface • Are independent of environment and implementation • May or may not be automated • Driven Development • Written prior to development • Confirmed with stakeholders • Development not complete until test passes What is ATDD?
7. www.cohesion.com | 877.774.3001 Test Case Execution Defect Management Test Case Creation Requirements Review Test Planning Typical workflow for waterfall development Elicit Requirements Analyze Requirements Design Code Test Deploy
8. www.cohesion.com | 877.774.3001 Typical workflow for waterfall development Elicit Requirements Analyze Requirements Design Code Test Deploy
9. www.cohesion.com | 877.774.3001 Typical workflow for waterfall development Elicit Requirements Analyze Requirements Design Code Test Deploy Months later
10. www.cohesion.com | 877.774.3001 Elicit Requirements Analyze Requirements Design Code Test Deploy Identify Success Criteria Create Tests Refactor Typical workflow for waterfall development
11. www.cohesion.com | 877.774.3001 Analyze Requirements Test Planning Typical SCRUM Implementation Test Case Creation Test Case Execution Defect Management
12. www.cohesion.com | 877.774.3001 This is a simple story clarification request Typical SCRUM Implementation
13. www.cohesion.com | 877.774.3001 This is a defect clarification request Typical SCRUM Implementation
14. www.cohesion.com | 877.774.3001 Typical SCRUM Implementation
15. www.cohesion.com | 877.774.3001 Inserting ATDD into SCRUM Each Story Create Success Criteria Code and Test Refactor
16. www.cohesion.com | 877.774.3001 • ATDD is about Collaboration and Communication • Getting all players on same page • Having the team know what successful implementation means • Reducing the number of communication pathways • Decreasing collaborative resolution time • Achieving better coverage of business expectations • Having a crisp understanding of needs • Creating closer, more integrated, teams • Is development methodology agnostic • It is about eliminating the need for unnecessary clarifications Recap – This is what ATDD is about
17. www.cohesion.com | 877.774.3001 Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Release Testing in an Agile Environment Test Sprint 1 Test Sprint 2 Test Sprint 3 Test Sprint 4 Test Sprint 5 Release Test Sprint 1 Test Sprint 2 Test Sprint 1 Test Sprint 3 Test Sprint 2 Test Sprint 1 Test Sprint 4 Test Sprint 3 Test Sprint 2 Test Sprint 1 Test Sprint 5 Test Sprint 4 Test Sprint 3 Test Sprint 2 Test Sprint 1
18. www.cohesion.com | 877.774.3001 Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Release Total Testing Effort Graphical Representation of Increasing Sprint Test Load
19. www.cohesion.com | 877.774.3001 • ATDD, but Automated • Leverage the power in repetition What is Automated ATDD?
20. www.cohesion.com | 877.774.3001 Typical can run faster than manual Increases confidence in covered automation areas Frees up testers from doing mundane checking to perform true testing Unchanged tests become cheaper over repetition Benefits of Automation
21. www.cohesion.com | 877.774.3001 • Considerations • Return on investment (ROI) is usually computed as the benefits derived divided by the investments made for a given thing • True ROI calculations require in-depth analysis and an understating of NPV, IRR, and other accounting formula’s. We will omit them for simplicity sake. • Some labor cost assumptions will need to be made for this process. They will be called out. • Objectives • Evaluate ROI for Total Cost of Ownership [Implementation through Maintenance] • Ignoring NPV and IRR considerations • Compare ROI for automation versus manual testing • Compare capital investment for automation of Open Source versus Commercial Software About our ROI calculations
22. www.cohesion.com | 877.774.3001 • Blended Labor Rate - $75/hr • Team size – 5 • Hardware Needs – Desktop for each tester • License Needs – 1 per tester • Using metrics by Capers Jones’ “The Economics of Software Quality” • Tests created are based on requirement counts • Defects found are based on tests executed • Pessimistic calculations – all inefficiencies are assumed to only affect testers ROI Calculations – Assumptions and Notes
23. www.cohesion.com | 877.774.3001 Lots of Worksheets
24. www.cohesion.com | 877.774.3001 Key Metrics - Capital Investment - QTP QTP Costs Year 1 Year 2 Year 3 Year 4 Year 5 Labor Rate (per hour) $75 Number of Servers/Desktops Needed 5 Cost per Server/Desktop $3,000 Hardware (additional or upgrades to existing) $15,000 $0 $0 $0 $0 Number of testers 5 Cost of per license $8,000 $8,000 Length of License (in years) 3 3 Maintenance rate (in years) 20% 20% 20% 20% 20% Cost of Automation Software (per year) $48,000.00 $8,000 $8,000 $48,000 $8,000 Automation Framework Development (in hours) 320 Annual Automation Framework Maintenance (per year) 160 240 240 240 240 Automation Framework $36,000 $18,000 $18,000 $18,000 $18,000 Tool training (formal class) $2,000 Tool introduction and ramp up (in hours) 80 Training $8,000 $0 $0 $0 $0 Total Non-Testing Costs $107,000 $26,000 $26,000 $66,000 $26,000
25. www.cohesion.com | 877.774.3001 Key Metrics - Capital Investment – ATDD using Ruby Ruby Costs Year 1 Year 2 Year 3 Year 4 Year 5 Labor Rate (per hour) $75 Number of Servers/Desktops Needed 5 Cost per Server/Desktop $3,000 Hardware (additional or upgrades to existing) $15,000 $0 $0 $0 $0 Number of testers 5 Cost of per license 0 Length of License (in years) 0 Maintenance rate (in years) 0 Cost of Automation Software (per year) $0 $0 Automation Framework Development (in hours) 480 Annual Automation Framework Maintenance (per year) 80 120 120 120 120 Automation Framework $42,000 $9,000 $9,000 $9,000 $9,000 Tool training (formal class) $3,500 Tool introduction and ramp up (in hours) 160 Training $15,500 $0 $0 $0 $0 Total Non-Testing Costs $72,500 $9,000 $9,000 $9,000 $9,000
26. www.cohesion.com | 877.774.3001 Waterfall Agile Average Number of Significant Requirements/Release 15 3 Average Number of Releases 4 26 Requirements requiring clarification 15% Tests Written/Year ~1000 ~1400 Test Writing Time/Requirement 1hr 30 minutes Key Metric Assumptions – Test Planning • Based on Capers Jones • Most numbers are based on function points, which are reversed engineered into Requirements and Tests Needed
27. www.cohesion.com | 877.774.3001 Manual Automated Regression Test Retention 0%-40% over 5 years Test Execution Time 30 minutes 1 minute Percent Pass – First Run 85% Defects Find Rate (per Test Case) 25% Defects requiring clarification 10% Average Number of Cycles for Clean Run 3 Key Metric Assumptions – Test Execution Waterfall Agile Communication Delays - Defects 4 hours Hands on Defect Management Time 30 minutes • Based on Capers Jones • Most numbers are based on function points, which are reversed engineered into Requirements and Tests Needed
28. www.cohesion.com | 877.774.3001 • Steady state in test planning • Increasing tests executed due to carry over regression • Inefficiencies increase test planning and test execution efforts ROI Method Year 1 Year 2 Year 3 Year 4 Year 5 Newly Created Total Executed
29. www.cohesion.com | 877.774.3001 $4,362,000 $4,183,500 $4,348,875 $4,091,625 $3,865,500 $3,600,000 $3,700,000 $3,800,000 $3,900,000 $4,000,000 $4,100,000 $4,200,000 $4,300,000 $4,400,000 $4,500,000 Waterfall - Manual Waterfall - ATDD Agile - Manual Agile - ATDD Agile - Automated ATDD 5 Year Labor Cost ROI of ATDD • $178k Savings • Equivalent to 1 FTE • $257k Savings • Equivalent to 1.75 FTEs • $483k Savings • Equivalent to 3.2 FTEs
30. www.cohesion.com | 877.774.3001 $4,349 $3,866 $0 $500 $1,000 $1,500 $2,000 $2,500 $3,000 $3,500 $4,000 $4,500 $5,000 $0 $200 $400 $600 $800 $1,000 $1,200 Year 1 Year 2 Year 3 Year 4 Year 5 Running Total (Thousands) Yearly (Thousands) Manual Agile Automated ATDD Manual Agile (No ATDD) vs. Automated ATDD Agile 14 Month ROI Breakeven Point $483k Savings 3.2 FTE Savings
31. www.cohesion.com | 877.774.3001 $4,092 $3,866 $0 $500 $1,000 $1,500 $2,000 $2,500 $3,000 $3,500 $4,000 $4,500 $0 $100 $200 $300 $400 $500 $600 $700 $800 $900 $1,000 Year 1 Year 2 Year 3 Year 4 Year 5 Running Total (Thousands) Yearly (Thousands) ATDD Manual ATDD Automated ATDD - Manual vs. Automation Break Even 23 Month ROI Breakeven Point $226k Savings 1.5 FTE Savings
32. www.cohesion.com | 877.774.3001 Representative Tool Comparisons • QTP • IDE with Record and Playback • Object Recognition • VBScript syntax • Commercial support and training available • Integration with Test Script Management • Cucumber/Watir • 3rd party IDE likely needed • Full Ruby language support • Application NOT needed to write tests • No object repository maintenance needed • True linking to requirements through BDD There are other tools!
33. www.cohesion.com | 877.774.3001 Quantitative • Achieve up to $483k in savings, or 3.2 FTEs, over 5 years • 14 month break even point, for Agile environments • $70k capital investment • Studies have shown first time pass rate approaches 87% • Studies have shown defect detection rate increases 3 fold Consolidated Business Case
34. www.cohesion.com | 877.774.3001 Qualitative • Increase in operational efficiency • Helps with customer, tester, developer communication • Shortens feedback cycle in teams • Decreases rework • Ensure happy path workflows work right first time • Ensure business rules are right • Tighter cross functional team integration • Crisp, visible completion criteria • Examples over Abstraction • Reduce risk in defects (and therefore exposure) • Automation increases regression test coverage without increasing headcount Consolidated Business Case
35. www.cohesion.com | 877.774.3001 • General • Always remember, neither automation nor ATDD is a silver bullet • Communication and collaboration goes two ways • ATDD is a process approach, not a type of test • Implementing Exploratory Testing with ATDD is highly recommended, and often misunderstood • Overstating expectations • Scaling too fast • There is no "right" approach in general, but rather a right approach in the context of your team and your situation • Automation • Automation is coding • Requires good coding practices such as DRY (Don’t Repeat Yourself) • Not everything can or SHOULD be automated • Automation is not a replacement for “testing”, only for “checking” • Don’t automate if you won’t reuse Challenges
36. www.cohesion.com | 877.774.3001 “Never lose the advantage of a bad situation” “Don’t bite off more than you can chew” Always at the beginning, never in between Need the right balance of People, Process, Tools Must have support of all management involved in project Be willing and able to adapt and to learn, things will change Identifying the Right Time and Place
37. www.cohesion.com | 877.774.3001 Joseph Ours, MBA, PMP Joseph.ours@gmail.com @justjoehere www.linkedin.com/in/josephours