Share
Embed
Feedback Loops
The role of QA is actually best served when we concentrate on providing short-burn feedback loops to the developers, analysts, and the business. This is a talk given at last year's QA or the Highway conference in Columbus, Ohio, bragging about some our success and trying to convert the masses to ATDD.
Mark Harris
December 04, 2012
Transcript
32 slides
-
SOFTWARE AND FEEDBACK LOOPS Testing at the Speed of Business
Mark Harris Enterprise QA Manager, Grange Insurance June 5th, 2012 -
About our Team Mixed processes: agile (cards, walls, metrics)
& traditional Mainframe is system-of-record .NET Web Forms interface (C#, VB.NET, F#) with complex object model and business rules ~180 people in App Dev (dev+test+mgt) 55k Users, both internal & external Tight timelines, complex business rules, large money on the table -
Tester Shout-Out The test team is the FIRST and BEST
customer You’re attentive & critical You’re available and nearby (usually) You’re absolutely not shy about feedback -
Feedback Loops Baked into human behavior “People are
proactive, aspiring organisms.” – Albert Bandura, Stanford University Harnessing the Power of Feedback Loops, Wired Magazine, June, 2011. -
Software Development Evidence User requests, failing test cases,
bugs Relevance Tests represent real user behavior Consequence Users unhappy with product Action Write cleaner code, fix issues faster & better -
Tester Shout-Out QA inserts feedback loops into the software development
process Positive Feedback Loops Guide the team -
from “Agile Testing: A Practical Guide for Testers and Agile
Teams”, Janet Gregory and Lisa Crispin, 2009, Addison-Wesley Professional Agile Testing Quadrants -
Req Design Code Test Release! Req Design Code Test Release!
Ideally… Reality… Testing get squeezed at the end Combination of dev, ba. Not explicitly done today. Traditional Development -
Customer-reported issues cost both time and lost business opportunity. Cost
of (non) Quality Source: “Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report.”, http://sarpresults.ivv.nasa.gov/viewresearch/289.24.jsp , from ftp://www.cob.sfasu.edu/csc/dcook/CSC323/LECTURE%202%20-%20Life%20Cycles.pdf -
The current approach… Guarantees (and rewards) rework. Every
issue found by a tester causes us to go backward in phases (either re-code, re-spec, or re-write requirements) Squeezes the testing phase. Requirements & Code always take longer than planned, but business dates infrequently move. Testing is forced to make risky decisions about which tests to run. Creates inconsistencies between specification and code Even a detailed specification needs updated as we make changes in the coding phase. -
Traditional Automation Doesn’t Help Automating the GUI is fragile
Automating behind the GUI is difficult Making hardened GUI tests is non-trivial Automation solutions assume the problem is the speed of people clicking buttons. The problem is COMMUNICATION -
From James Shore, Brian Marick, Elisabeth Hendrickson, and Grigori Melnick,
http://testobsessed.com/wp- content/uploads/2011/04/atddexample.pdf, p. 3. Acceptance Test Driven Design -
ATDD Example 16 GIVEN I am logged in as a
customer service representative WHEN I enter data for a Commercial Auto policy for a 100-vehicle fleet of Dodge Neon AND I submit the policy for a quote THEN my quote should be less than $19,000 -
ATDD Example 2 17 GIVEN a Home Owner policy in
zip code 43035 in Ohio WHEN I visit the billing information page THEN I should see “Pay In Full” and “Pay In Installments” -
Requirements Design Code Test Release! Acceptance Test Driven Design Communication
is REQUIRED USE THE DESIGN TO TEST THE SYSTEM Passing tests specify the system’s behavior Failing tests show a difference between specification and implementation: a BUG -
Automated Functional is not Enough Non-functional requirements Performance,
scalability, reliability End-User Acceptance Even if the product works the way the users said they wanted it, things change Edge & Error Cases Complete examples are prohibitively expensive -
Feedback Loop: Critical Testing End to End Scenarios Exploratory Testing
User Acceptance Testing -
from “Agile Testing: A Practical Guide for Testers and Agile
Teams”, Janet Gregory and Lisa Crispin, 2009, Addison-Wesley Professional Agile Testing Quadrants -
Business Facing & Critique Product End-to-End catches integration issues
Error cases, performance issues, reliability concerns Exploratory Testing engages the “What-If?” in every tester Edge cases, “Rightness” problems Build upon a strong foundation of functional tests -
Example: End to End Tests Quote a policy, issue
it, renew, cancel, and reinstate Cancel existing policy and print cancellation letter File 2 claims in one day on the same policy Amend a policy for a youthful driver, renew, and file a claim 1 year later -
Customer-reported issues cost both time and lost business opportunity. Cost
of (non) Quality Source: “Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report.”, http://sarpresults.ivv.nasa.gov/viewresearch/289.24.jsp , from ftp://www.cob.sfasu.edu/csc/dcook/CSC323/LECTURE%202%20-%20Life%20Cycles.pdf -
Definitions Net ROI = Net Benefit / Net Cost Net
Benefit = cost gained through defect prevention & early detection, as compared to manual testing alone. Net Cost = cost of increased team + training + infrastructure, as compared to manual testing alone. -
Cost/Benefit 51 customer issues analyzed 33 were ATDD-findable
(requirements, specification, or functional problem) 13 could have been found in end-to-end/exploratory testing (complex business rules, printing problems, UI confusing) 5 – one-off/difficult (business changed after release) Savings (ATDD)= 33 * 368 * $60 ~ $730k Savings (Manual)= 13 * 368 * $60 ~ $290k Total is over $1M saved with ATDD + Manual -
Shared accountability Three Amigos: dev/test/ba Add coders
to QA Seminars, Docs, Training Paired testing (2 testers) Pairing (test and dev) Experiences Challenges This level of collaboration isn’t natural (silos) Pushing testers beyond “test scripts” Skill-set Dev/Test/BA for tools used in ATDD Experience levels to know when “enough is enough” Strategies Page 29 December 4, 2012 -
Resources Agile Testing: A Practical Guide for Testers and
Agile Teams, Lisa Crispin and Janet Gregory, Addison-Wesley Professional, 2009. Brian Marick, exampler.com Jeff “Cheesy” Morgan, cheezyworld.com Elisabeth Hendricksen, testobsessed.com