Feedback Loops

The role of QA is actually best served when we concentrate on providing short-burn feedback loops to the developers, analysts, and the business. This is a talk given at last year's QA or the Highway conference in Columbus, Ohio, bragging about some our success and trying to convert the masses to ATDD.

Transcript

1. SOFTWARE AND FEEDBACK LOOPS Testing at the Speed of Business Mark Harris Enterprise QA Manager, Grange Insurance June 5th, 2012
2. About our Team  Mixed processes: agile (cards, walls, metrics) & traditional  Mainframe is system-of-record  .NET Web Forms interface (C#, VB.NET, F#) with complex object model and business rules  ~180 people in App Dev (dev+test+mgt)  55k Users, both internal & external  Tight timelines, complex business rules, large money on the table
3. Tester Shout-Out The test team is the FIRST and BEST customer You’re attentive & critical You’re available and nearby (usually) You’re absolutely not shy about feedback
4. Feedback Loops  Baked into human behavior  “People are proactive, aspiring organisms.” – Albert Bandura, Stanford University Harnessing the Power of Feedback Loops, Wired Magazine, June, 2011.
5. Garmin Connect http://www.engadget.com/2007/08/29/garmin-intros-two-bike-centric-gps-units/ http://www.heart-rate-monitor- fitness.com/garmin-edge-605-review
6. Weight Watchers™ http://adsoftheworld.com/media/print/weightwatchers_bulges?size=_original
7. Software Development  Evidence  User requests, failing test cases, bugs  Relevance  Tests represent real user behavior  Consequence  Users unhappy with product  Action  Write cleaner code, fix issues faster & better
8. Tester Shout-Out QA inserts feedback loops into the software development process Positive Feedback Loops Guide the team
9. from “Agile Testing: A Practical Guide for Testers and Agile Teams”, Janet Gregory and Lisa Crispin, 2009, Addison-Wesley Professional Agile Testing Quadrants
10. Feedback loop: Automation Acceptance Test Driven Development
11. Req Design Code Test Release! Req Design Code Test Release! Ideally… Reality… Testing get squeezed at the end Combination of dev, ba. Not explicitly done today. Traditional Development
12. Customer-reported issues cost both time and lost business opportunity. Cost of (non) Quality Source: “Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report.”, http://sarpresults.ivv.nasa.gov/viewresearch/289.24.jsp , from ftp://www.cob.sfasu.edu/csc/dcook/CSC323/LECTURE%202%20-%20Life%20Cycles.pdf
13. The current approach…  Guarantees (and rewards) rework.  Every issue found by a tester causes us to go backward in phases (either re-code, re-spec, or re-write requirements)  Squeezes the testing phase.  Requirements & Code always take longer than planned, but business dates infrequently move. Testing is forced to make risky decisions about which tests to run.  Creates inconsistencies between specification and code  Even a detailed specification needs updated as we make changes in the coding phase.
14. Traditional Automation Doesn’t Help  Automating the GUI is fragile  Automating behind the GUI is difficult  Making hardened GUI tests is non-trivial  Automation solutions assume the problem is the speed of people clicking buttons. The problem is COMMUNICATION
15. From James Shore, Brian Marick, Elisabeth Hendrickson, and Grigori Melnick, http://testobsessed.com/wp- content/uploads/2011/04/atddexample.pdf, p. 3. Acceptance Test Driven Design
16. ATDD Example 16 GIVEN I am logged in as a customer service representative WHEN I enter data for a Commercial Auto policy for a 100-vehicle fleet of Dodge Neon AND I submit the policy for a quote THEN my quote should be less than $19,000
17. ATDD Example 2 17 GIVEN a Home Owner policy in zip code 43035 in Ohio WHEN I visit the billing information page THEN I should see “Pay In Full” and “Pay In Installments”
18. Requirements Design Code Test Release! Acceptance Test Driven Design Communication is REQUIRED  USE THE DESIGN TO TEST THE SYSTEM  Passing tests specify the system’s behavior  Failing tests show a difference between specification and implementation: a BUG
19. Automated Functional is not Enough  Non-functional requirements  Performance, scalability, reliability  End-User Acceptance  Even if the product works the way the users said they wanted it, things change  Edge & Error Cases  Complete examples are prohibitively expensive
20. Tester Shout-Out Human testers will find qualitative problems far sooner than machines.
21. Feedback Loop: Critical Testing End to End Scenarios Exploratory Testing User Acceptance Testing
22. from “Agile Testing: A Practical Guide for Testers and Agile Teams”, Janet Gregory and Lisa Crispin, 2009, Addison-Wesley Professional Agile Testing Quadrants
23. Business Facing & Critique Product  End-to-End catches integration issues  Error cases, performance issues, reliability concerns  Exploratory Testing engages the “What-If?” in every tester  Edge cases, “Rightness” problems  Build upon a strong foundation of functional tests
24. Example: End to End Tests  Quote a policy, issue it, renew, cancel, and reinstate  Cancel existing policy and print cancellation letter  File 2 claims in one day on the same policy  Amend a policy for a youthful driver, renew, and file a claim 1 year later
25. The Cost of (non) Quality Cost/Benefit 25
26. Customer-reported issues cost both time and lost business opportunity. Cost of (non) Quality Source: “Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report.”, http://sarpresults.ivv.nasa.gov/viewresearch/289.24.jsp , from ftp://www.cob.sfasu.edu/csc/dcook/CSC323/LECTURE%202%20-%20Life%20Cycles.pdf
27. Definitions Net ROI = Net Benefit / Net Cost Net Benefit = cost gained through defect prevention & early detection, as compared to manual testing alone. Net Cost = cost of increased team + training + infrastructure, as compared to manual testing alone.
28. Cost/Benefit  51 customer issues analyzed  33 were ATDD-findable (requirements, specification, or functional problem)  13 could have been found in end-to-end/exploratory testing (complex business rules, printing problems, UI confusing)  5 – one-off/difficult (business changed after release)  Savings (ATDD)= 33 * 368 * $60 ~ $730k  Savings (Manual)= 13 * 368 * $60 ~ $290k  Total is over $1M saved with ATDD + Manual
29.  Shared accountability  Three Amigos: dev/test/ba  Add coders to QA  Seminars, Docs, Training  Paired testing (2 testers)  Pairing (test and dev) Experiences Challenges  This level of collaboration isn’t natural (silos)  Pushing testers beyond “test scripts”  Skill-set Dev/Test/BA for tools used in ATDD  Experience levels to know when “enough is enough” Strategies Page 29 December 4, 2012
30. THANK YOU!
32. Resources  Agile Testing: A Practical Guide for Testers and Agile Teams, Lisa Crispin and Janet Gregory, Addison-Wesley Professional, 2009.  Brian Marick, exampler.com  Jeff “Cheesy” Morgan, cheezyworld.com  Elisabeth Hendricksen, testobsessed.com