Prioritizing test suites by finding Hamiltonian paths: Preliminary studies and initial results

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Bhadra2008a/

Transcript

1. Introduction Resource-Constrained Testing Conclusions and Future Work Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results Suvarshi Bhadra and Gregory M. Kapfhammer Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/ TAIC PART 2008, Fast Abstract Track, August 2008 Featuring images from www.pdclipart.org 1 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
2. Introduction Resource-Constrained Testing Conclusions and Future Work Important Contributions Technique Formulation NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score sss − covers NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score ssm − covers NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score ssl − covers Empirical Results Design, implement and empirically evaluate test suite prioritizers that leverage travelling salesperson problem (TSP) solvers to efficiently find cost-effective orderings 2 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
3. Introduction Resource-Constrained Testing Conclusions and Future Work Regression Testing Techniques Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It may expensive to run a test suite T = T1, . . . , Tn . Prioritization searches through the n! = n × n − 1 × . . . × 1 orderings for those that avoid costly database restarts, Web service calls, or memory interactions. 3 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
4. Introduction Resource-Constrained Testing Conclusions and Future Work Prioritizing When Memory is Constrained ag replacements N0 N1 N2 N3 N4 N5 Frequent Memory Rewrites PSfrag replacements N0 N1 N2 N3 N4 N5 High Testing Costs Frequent reads and writes to memory may increase execution time by as much as 600% when a Java application executes on a virtual machine with a small heap Solution: maximize memory reuse between test cases 4 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
5. Introduction Resource-Constrained Testing Conclusions and Future Work The Impact of Test Ordering m1 m2 m3 m4 m5 m6 Test 30 30 30 30 30 30 Size T1 • • • 90 T2 • • • 90 T3 • • • 90 T4 • • • 90 T5 • • 60 T = T1, T2, T3, T4, T5 transfers 750 units to and from memory T = T2, T4, T1, T3, T5 only loads and unloads 180 units 5 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
6. Introduction Resource-Constrained Testing Conclusions and Future Work Test Prioritization: Steps One and Two Program Instrumentation Adequacy Criterion Residual Test Requirements Cummulative Adequacy Calculator Instrumented Program Test Suite Execution Covered Requirements ents N0 N1 N2 N3 N4 N5 Collect method invocation and size data using test coverage monitor and profiler 553 256 553 256 256 0 256 553 256 256 O 553 553 256 256 234 553 256 256 256 678 553 256 553 234 234 553 256 553 256 PSfrag replacements N0 N1 N2 N3 N4 N5 Formulate a complete graph using equations that estimate costs for all test pairs 6 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
7. Introduction Resource-Constrained Testing Conclusions and Future Work Test Prioritization: Steps Three and Four ements N0 N1 N2 N3 N4 N5 Use TSP solvers to identify a Hamiltonian path with low estimated costs 10 20 30 40 50 60 70 80 90 100 1.0 1.5 2.0 2.5 Size of the Test Suite (n) Execution Time (sec) PSfrag replacements N0 N1 N2 N3 N4 N5 Evaluate the efficiency of the TSP solvers and the effectiveness of the test orderings 7 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
8. Introduction Resource-Constrained Testing Conclusions and Future Work Empirical Results 0 200 400 600 800 1000 0.00 0.05 0.10 0.15 0.20 NN RNN NI FI CI AI Time v No of Test Cases (n) (t) ments N0 N1 N2 N3 N4 N5 Efficient Prioritizers 10 20 30 40 50 50 60 70 80 90 100 Score v No of Test Cases − NN (n) (score) PSfrag replacements N0 N1 N2 N3 N4 N5 High Percentile Rankings 8 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
9. Introduction Resource-Constrained Testing Conclusions and Future Work Avoiding Database Restarts T0 T1 T2 T9 T13 T15 T5 T8 T12 T3 T7 T11 T14 T16 T10 T4 T6 placements N0 N1 N2 N3 N4 N5 Use prioritization to avoid costly database restarts 9 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results
10. Introduction Resource-Constrained Testing Conclusions and Future Work Concluding Remarks Program Analysis Technique Test Suite Reduction Prioritization ents N0 N1 N2 N3 N4 N5 Test Suite Prioritizers NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score sss − covers NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score ssm − covers NCO JCO TCO 0.0 0.2 0.4 0.6 0.8 1.0 Overlap Metric Overlap Score ssl − covers PSfrag replacements N0 N1 N2 N3 N4 N5 Initial Empirical Evaluation Preliminary results with synthetic test suites indicate that it is possible to prioritize test suites with TSP solvers Use different methods for solving TSP instances (e.g., order-based genetic algorithms) and include real-world applications http://www.cs.allegheny.edu/~gkapfham/research/ 10 / 10 Prioritizing Test Suites by Finding Hamiltonian Paths: Preliminary Studies and Initial Results