Objectively Superior Software Test Sets - An Explanation

When selecting from a large number of existing regression tests, software testers and stakeholders often struggle to select a manageable number of tests that do a good job of thoroughly testing the system in an efficient, systematic way. Modern test design approaches enable teams to achieve more thorough testing coverage using fewer tests. This presentation explains how the seemingly contradictory goals of (a) achieving more testing coverage and (b) using fewer software tests can be simultaneously achieved.

Transcript

1. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Objectively Superior Software Test Sets An Explanation Create and document tests much faster. Rapidly select and document software tests using the the famously easy-to-use Hexawise test design tool. Select and document tests much faster. Achieve greater coverage. In fewer tests.
2. Copyright © 2016 Hexawise, Inc. All Rights Reserved. This presentation provides a detailed explanation of exactly how Hexawise-generated tests are objectively superior to manually- selected sets. Specifically, it explains how relatively small sets of Hexawise tests consistently achieve greater thoroughness than larger sets of traditionally-selected tests. 2 Comparison of Test Sets (with / without Hexawise)
3. Copyright © 2016 Hexawise, Inc. All Rights Reserved. 3 Benefits Occurring in the Test Execution Phase In the test execution phase, testers using Hexawise achieve both thoroughness gains and efficiency benefits.
4. Copyright © 2016 Hexawise, Inc. All Rights Reserved. System Under Test 4 This is the simplified System Under Test used in this explanation. The tests involve the process of an individual who is applying for a loan. There are almost 20,000 possible tests to choose from.
5. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Test Ideas Let’s compare test coverage achieved by two different test design strategies as the 7 test ideas listed below get included in … 1) Manually- selected tests. 2) Hexawise- generated tests. - vs - 7 Test Ideas 5
6. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Translating Test Ideas into Tests (Manual Selection) Manual Selection To translate the seven test ideas into tests, a tester would begin by identifying specific test conditions that must appear in each test. 6
7. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Translating Test Ideas into Tests (Manual Selection) Manual Selection For tests created by hand, the “other” test conditions rarely receive attention. “Blanks” typically appear - or happy path values are used. 7
8. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Translating Test Ideas into Tests (Manual Selection) Manual Selection The consistent result of manual test selection? Wasteful repetition. Wasteful  repetition New  ideas  tested 8
9. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Optimized Hexawise Tests Hexawise Selection Wasteful  repetition New  ideas  tested Hexawise tests are very different. 1st, Hexawise eliminates all wasteful repetition automatically. 9
10. Copyright © 2016 Hexawise, Inc. All Rights Reserved. By eliminating wasteful repetition and systematically covering simple interactions, Hexawise-generated tests cover more using fewer tests. Coverage Achieved in First 7 Tests Without Hexawise Using Hexawise 7 Optimized Tests: 68% 7 Manually- Selected Tests: 26% 10
11. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Coverage Achieved in First 17 Tests 17 Optimized Tests: 100% 17 Manually- Selected Tests: 51% Without Hexawise Using Hexawise Within only 17 tests, Hexawise-generated tests cover all 324 pairwise interactions within the System Under Test. In contrast, the manually- selected test set’s first 17 tests fail to cover half of them. 11
12. Copyright © 2016 Hexawise, Inc. All Rights Reserved. 2x, 3x, 4x… When test sets are selected without using Hexawise, even when using 4 times as many tests, many coverage gaps remain. 34 Manually- Selected Tests: 58% 51 Manually- Selected Tests: 65% 68 Manually- Selected Tests: 68% Twice as many tests Three times as many tests Four times as many tests 12
13. Copyright © 2016 Hexawise, Inc. All Rights Reserved. 13 Challenging Without a Structured Approach… When manual test selection methods are used, many coverage gaps result. Why? In any non-trivial System Under Test, there are simply too many interactions for humans to keep track of without the aid of coverage optimization algorithms.
14. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Case Study 1 (Leading U.S. Insurance Firm) 25 Optimized Tests: 94% 50 Manually- Selected Tests: 54% Without Hexawise Using Hexawise The prior example was illustrative. The two charts shown below provide actual “before” and “after” coverage achieved by two test sets. Large efficiency and thoroughness gains are delivered simultaneously. 14
15. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Case Study 2 (Leading Global Bank) Without Hexawise Using Hexawise In this project, a leading global bank used Hexawise to dramatically reduce how many tests they executed (across 10 different sets of tests). 15 0 3500 7000 10500 14000 1,775 Optimized Hexawise Tests 12,471 BAU Tests
16. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Case Study 2 (Leading Global Bank) Without Hexawise Using Hexawise Again, even using far fewer tests, Hexawise generated tests delivered objectively superior testing coverage as compared to the previously- existing traditionally-selected tests. 16 1st 50 BAU Tests (Not Optimized)  x 1st 50 Tests (Optimized with Hexawise) 1st 142 Tests (Optimized with Hexawise)
17. Copyright © 2016 Hexawise, Inc. All Rights Reserved. Case Study 1 For more information about Hexawise and optimized test design methods (including pairwise testing, orthogonal array testing, and combinatorial testing) please see: hexawise.com (powerful, famously easy-to-use, test design tool used by more than 100 Fortune 500 firms) hexawise.tv (bite-sized instructional videos) training.hexawise.com (clear, practical software test design instructional articles and tips and tricks) 17