An empirical study of incorporating cost into test suite reduction and prioritization

An empirical study of incorporating cost into test suite reduction and prioritization

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Smith2009/

4ae30d49c8cc07e42d5a871efb9bcfba?s=128

Gregory Kapfhammer

March 08, 2009
Tweet

Transcript

  1. Introduction Regression Testing Empirical Evaluation Conclusion An Empirical Study of

    Incorporating Cost into Test Suite Reduction and Prioritization ‡Adam M. Smith and † Gregory M. Kapfhammer ‡Department of Computer Science University of Pittsburgh †Department of Computer Science Allegheny College ACM Symposium on Applied Computing March 8 - 12, 2009 Featuring an image from www.campusbicycle.com 1 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  2. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  3. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  4. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  5. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  6. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles

    Efficiency: Low wind resistance and time to destination 3 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  7. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles

    Effectiveness: Transports all required materials and no break downs 3 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  8. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles

    Cost: Frame material and components cause price to vary considerably 3 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  9. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 4 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  10. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 4 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  11. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 4 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  12. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 4 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  13. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Prioritization searches through the n! = n × n − 1 × . . . × 1 orderings for those that maximize an objective function like coverage or fault detection. 4 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  14. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in

    Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti Test suite reduction discards the test cases that redundantly cover the test requirements T = T2, T3, T6, T9 covers all of the test requirements 5 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  15. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in

    Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti Test suite reduction discards the test cases that redundantly cover the test requirements T = T2, T3, T6, T9 covers all of the test requirements 5 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  16. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in

    Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti Test suite reduction discards the test cases that redundantly cover the test requirements T = T2, T3, T6, T9 covers all of the test requirements 5 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  17. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 6 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  18. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 6 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  19. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 6 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  20. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 6 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  21. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers

    Testing Time . . . Covered Test Reqs acements T1 Done Tn−1 Done Tn Done Cover R(T1 ) Cover n−1 i=1 R(Ti ) Cover R(T) Area t(n) 0 C(T , t) C(T , t) (t) Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0 , 1] 7 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  22. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers

    Testing Time . . . Covered Test Reqs acements T1 Done Tn−1 Done Tn Done Cover R(T1 ) Cover n−1 i=1 R(Ti ) Cover R(T) Area t(n) 0 C(T , t) C(T , t) (t) Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0 , 1] 7 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  23. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers

    Reduction Factor for Size (RFFS): How small is the reduced test suite? 8 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  24. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers

    Reduction Factor for Time (RFFT): How fast is the reduced test suite? 8 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  25. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1, T4 5 T1, T4, T2, T3 0.400 time T2, T3, T4 3 T2, T3, T4, T1 0.714 ratio T2, T4, T3 3 T2, T4, T3, T1 0.743 9 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  26. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1, T4 5 T1, T4, T2, T3 0.400 time T2, T3, T4 3 T2, T3, T4, T1 0.714 ratio T2, T4, T3 3 T2, T4, T3, T1 0.743 9 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  27. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1, T4 5 T1, T4, T2, T3 0.400 time T2, T3, T4 3 T2, T3, T4, T1 0.714 ratio T2, T4, T3 3 T2, T4, T3, T1 0.743 9 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  28. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1, T4 5 T1, T4, T2, T3 0.400 time T2, T3, T4 3 T2, T3, T4, T1 0.714 ratio T2, T4, T3 3 T2, T4, T3, T1 0.743 9 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  29. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 10 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  30. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 10 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  31. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 10 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  32. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Questions: Do the greedy reducers and prioritizers efficiently identify test suites that improve effectiveness? What are the fundamental trade-offs? 10 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  33. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends

    | metric: cost alg: 2OPT,GRD 0.2101 0.4946 0.4889 Reduction Factor for Time (RFFT) The myopic focus on cost leads to low RFFT values for 2OPT and GRD 11 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  34. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends

    | metric: cost alg: 2OPT,GRD 0.2101 0.4946 0.4889 Reduction Factor for Time (RFFT) The myopic focus on cost leads to low RFFT values for 2OPT and GRD 11 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  35. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends

    | alg: 2OPT,GRD metric: cost metric: coverage 0.1130 0.5967 0.4959 0.6136 Reduction Factor for Size (RFFS) DGR and HGS are the best at creating test suites that improve RFFS 12 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  36. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends

    | alg: 2OPT,GRD metric: cost metric: coverage 0.1130 0.5967 0.4959 0.6136 Reduction Factor for Size (RFFS) DGR and HGS are the best at creating test suites that improve RFFS 12 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  37. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends

    | alg: HGS metric: coverage alg: DGR 0.7520 0.8231 0.8344 0.9388 Coverage Effectiveness (CE) Using ratio and cost improves the CE of the prioritized test suite 13 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  38. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends

    | alg: HGS metric: coverage alg: DGR 0.7520 0.8231 0.8344 0.9388 Coverage Effectiveness (CE) Using ratio and cost improves the CE of the prioritized test suite 13 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  39. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time

    - SK Reduction Technique Reduction Factor for Time (RFFT) 0.0 0.1 0.2 0.3 0.4 2OPT DGR GRD HGS SK cost coverage ratio For 2OPT and GRD, ratio and coverage create the best test suites 14 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  40. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time

    - SK Reduction Technique Reduction Factor for Time (RFFT) 0.0 0.1 0.2 0.3 0.4 2OPT DGR GRD HGS SK cost coverage ratio For 2OPT and GRD, ratio and coverage create the best test suites 14 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  41. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size

    - SK Reduction Technique Reduction Factor for Size (RFFS) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 2OPT DGR GRD HGS SK cost coverage ratio It is often easy to construct test suites with high RFFS values 15 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  42. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size

    - SK Reduction Technique Reduction Factor for Size (RFFS) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 2OPT DGR GRD HGS SK cost coverage ratio It is often easy to construct test suites with high RFFS values 15 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  43. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio DGR and HGS exhibit lackluster performance when reordering 16 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  44. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio DGR and HGS exhibit lackluster performance when reordering 16 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  45. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements Prioritization Technique

    Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD For the chosen case study applications, the techniques are efficient 17 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  46. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements Prioritization Technique

    Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD For the chosen case study applications, the techniques are efficient 17 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  47. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like

    APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness (FDE) and average percentage of faults detected (APFD). Consider search-based testing methods. 18 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  48. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - Reduce And

    prIortize SuitEs http://raise.googlecode.com/ provides tools, data sets, and resources 19 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization
  49. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 20 / 20 An Empirical Study of Incorporating Cost into Test Suite Reduction and Prioritization