Practical suggestions for improving and empirically studying greedy test suite reduction and prioritization methods

Practical suggestions for improving and empirically studying greedy test suite reduction and prioritization methods

Interested in learning more about this topic? Visit this web site to read a related post: https://www.gregorykapfhammer.com/testing/research/advice/2017/07/03/Regression-Testing-Costs/

4ae30d49c8cc07e42d5a871efb9bcfba?s=128

Gregory Kapfhammer

November 13, 2009
Tweet

Transcript

  1. Introduction Regression Testing Empirical Evaluation Conclusion Practical Suggestions for Improving

    and Empirically Studying Greedy Test Suite Reduction and Prioritization Methods Gregory M. Kapfhammer† Department of Computer Science Allegheny College http://www.cs.allegheny.edu/~gkapfham/ Department of Computer Science and Technology Nanjing University, November 2009 †In Conjunction with Adam M. Smith, Joshua J. Geiger, G. Elisabeta Mara (University of Pittsburgh) Manos Renieris (Google) Featuring an image from www.CampusBicycle.com 1 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  2. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Study Overview: Implement and evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  3. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Study Overview: Implement and evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  4. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Study Overview: Implement and evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  5. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Study Overview: Implement and evaluate the efficiency and effectiveness of cost-aware greedy methods for regression test suite reduction and prioritization 2 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  6. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Study Experiments: Use automatically generated synthetic test suites and real world case study applications during the empirical study of greedy regression testing methods 2 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  7. Introduction Regression Testing Empirical Evaluation Conclusion Important Contributions Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Study Analysis: Develop and use tree and random forest statistical models and interactive visualization techniques that help to identify efficiency and effectiveness trade-offs for testing 2 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  8. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles

    Efficiency: Low wind resistance and time to destination 3 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  9. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles

    Effectiveness: Transports all required materials and no break downs 3 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  10. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing and Bicycles

    Cost: Frame material and components cause price to vary considerably 3 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  11. Introduction Regression Testing Empirical Evaluation Conclusion What is a Test

    Case? Input Method Under Test Output Test Oracle Expected Output Verdict Tear Down Set Up Test suite executor runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values 4 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  12. Introduction Regression Testing Empirical Evaluation Conclusion What is a Test

    Case? Input Method Under Test Output Test Oracle Expected Output Verdict Tear Down Set Up Test suite executor runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values 4 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  13. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1 , . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 5 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  14. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1 , . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 5 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  15. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1 , . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 5 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  16. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1 , . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 5 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  17. Introduction Regression Testing Empirical Evaluation Conclusion Regression Testing Techniques Before

    After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1 , . . . , Tn . Prioritization searches through the n! = n × n − 1 × . . . × 1 orderings for those that maximize an objective function like coverage or fault detection. 5 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  18. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of

    a Test Suite Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Calculating Coverage Use instrumentation probes to capture and analyze a test suite’s coverage of the program state and structure Regression Testing The coverage results and adequacy measurements can support both test suite reduction and prioritization 6 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  19. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of

    a Test Suite Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Calculating Coverage Use instrumentation probes to capture and analyze a test suite’s coverage of the program state and structure Regression Testing The coverage results and adequacy measurements can support both test suite reduction and prioritization 6 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  20. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of

    a Test Suite Program Instrumentation Test Suite Adequacy Criterion Instrumented Program Test Coverage Monitoring Instrumented Test Suite Coverage Results Adequacy Calculation Test Requirements Adequacy Measurements Calculating Coverage Use instrumentation probes to capture and analyze a test suite’s coverage of the program state and structure Regression Testing The coverage results and adequacy measurements can support both test suite reduction and prioritization 6 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  21. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in

    Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti Test suite reduction discards the test cases that redundantly cover the test requirements T = T2 , T3 , T6 , T9 covers all of the test requirements 7 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  22. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in

    Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti Test suite reduction discards the test cases that redundantly cover the test requirements T = T2 , T3 , T6 , T9 covers all of the test requirements 7 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  23. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in

    Coverage T2 T3 T6 T9 R R1 R2 T4 T8 T11 R3 T12 R4 T1 R4 R4 R4 T5 R4 R4 T10 R4 T11 R5 T5 R5 R6 R6 T10 R7 R7 T7 R7 T10 Rj → Ti means that requirement Rj is covered by test Ti Test suite reduction discards the test cases that redundantly cover the test requirements T = T2 , T3 , T6 , T9 covers all of the test requirements 7 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  24. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 8 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  25. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 8 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  26. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 8 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  27. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression

    Testing Original Test Suite First Output First Residual Second Output Prioritized Test Suite Reduction Technique ts T4 T3 T2 T1 T4 T1 T3 T2 T3 T2 T3 T3 T3 T3 T2 T2 T2 T2 T4 T4 T4 T1 T1 T1 Harrold, Gupta, Soffa (HGS) Delayed Greedy (DGR) Traditional Greedy (GRD) 2-Optimal Greedy (2OPT) Hypothesis: Using the exe- cution time of a test case can improve the reduced and pri- oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms 8 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  28. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers

    Testing Time . . . Covered Test Reqs acements T1 Done Tn−1 Done Tn Done Cover R(T1 ) Cover n−1 i=1 R(Ti ) Cover R(T) Area t(n) 0 C(T, t) C(T , t) (t) Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1] 9 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  29. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers

    Testing Time . . . Covered Test Reqs acements T1 Done Tn−1 Done Tn Done Cover R(T1 ) Cover n−1 i=1 R(Ti ) Cover R(T) Area t(n) 0 C(T, t) C(T , t) (t) Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1] 9 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  30. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers

    Reduction Factor for Size (RFFS): How small is the reduced test suite? 10 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  31. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers

    Reduction Factor for Time (RFFT): How fast is the reduced test suite? 10 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  32. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers

    Common Rate (CR): How similar are differently reduced test suites? 10 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  33. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1 , T4 5 T1 , T4 , T2 , T3 0.400 time T2 , T3 , T4 3 T2 , T3 , T4 , T1 0.714 ratio T2 , T4 , T3 3 T2 , T4 , T3 , T1 0.743 11 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  34. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1 , T4 5 T1 , T4 , T2 , T3 0.400 time T2 , T3 , T4 3 T2 , T3 , T4 , T1 0.714 ratio T2 , T4 , T3 3 T2 , T4 , T3 , T1 0.743 11 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  35. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1 , T4 5 T1 , T4 , T2 , T3 0.400 time T2 , T3 , T4 3 T2 , T3 , T4 , T1 0.714 ratio T2 , T4 , T3 3 T2 , T4 , T3 , T1 0.743 11 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  36. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1 , T4 5 T1 , T4 , T2 , T3 0.400 time T2 , T3 , T4 3 T2 , T3 , T4 , T1 0.714 ratio T2 , T4 , T3 3 T2 , T4 , T3 , T1 0.743 11 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  37. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1 , T4 5 T1 , T4 , T2 , T3 0.400 time T2 , T3 , T4 3 T2 , T3 , T4 , T1 0.714 ratio T2 , T4 , T3 3 T2 , T4 , T3 , T1 0.743 11 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  38. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness

    R1 R2 R3 R4 R5 Execution Time T1 4 T2 1 T3 1 T4 1 Greedy-by Tr time(Tr ) Tp CE coverage T1 , T4 5 T1 , T4 , T2 , T3 0.400 time T2 , T3 , T4 3 T2 , T3 , T4 , T1 0.714 ratio T2 , T4 , T3 3 T2 , T4 , T3 , T1 0.743 11 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  39. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 Tree Models: Use recursive partitioning to create hierarchical view of data 12 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  40. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 Explanatory Variable: Configuration of the testing methods (e.g., GCM) 12 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  41. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 Response Variable: One of the evaluation metrics (e.g., CE or RFFT) 12 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  42. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 13 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  43. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 13 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  44. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 13 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  45. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 13 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  46. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests

    metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 metric: cost alg: 2OPT, GRD Mean 0.4889 Mean 0.2101 Mean 0.4946 Many Trees: Randomly construct a large collection of trees in order to avoid bias and identify the most important explanatory variables 13 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  47. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 14 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  48. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 14 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  49. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 14 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  50. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name

    |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Do the greedy reducers and prioritizers efficiently identify test suites that improve effectiveness? What are the fundamental trade-offs? 14 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  51. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation

    0gkapfhamExp Revision : 1.2 T1 R1 R2 R3 T2 T3 T4 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal: generate coverage and timing information that will fool the greedy technique into creating Tp = Tn , . . . , T1 even though CE(Tp ) < CE(T) for T = T1 , . . . , Tn Inspiration: Vazirani’s construction of a tight example for the greedy minimal set cover algorithm 15 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  52. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation

    0gkapfhamExp Revision : 1.2 T1 R1 R2 R3 T2 T3 T4 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal: generate coverage and timing information that will fool the greedy technique into creating Tp = Tn , . . . , T1 even though CE(Tp ) < CE(T) for T = T1 , . . . , Tn Inspiration: Vazirani’s construction of a tight example for the greedy minimal set cover algorithm 15 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  53. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation

    0gkapfhamExp Revision : 1.2 T1 R1 R2 R3 T2 T3 T4 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal: generate coverage and timing information that will fool the greedy technique into creating Tp = Tn , . . . , T1 even though CE(Tp ) < CE(T) for T = T1 , . . . , Tn Inspiration: Vazirani’s construction of a tight example for the greedy minimal set cover algorithm 15 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  54. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test

    Suites 0gkapfhamExp Revision : 1.2 T1 R1 R2 R3 T2 T3 T4 Approach: use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost(Tn ) Construction: set cost(T1 ) = cost(T2 ) = cost(T3 ) = 1 and then determine the bounds for cost(T4 ) ∈ [Cmin , Cmax ] Example: cost(T4 ) ∈ [2.138803, 2.472136] so that CEmin (Tp ) = .5838004 CEmin (T) = .6108033 CEmax (Tp ) = .5482172 CEmax (T) = .6345125 16 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  55. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test

    Suites 0gkapfhamExp Revision : 1.2 T1 R1 R2 R3 T2 T3 T4 Approach: use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost(Tn ) Construction: set cost(T1 ) = cost(T2 ) = cost(T3 ) = 1 and then determine the bounds for cost(T4 ) ∈ [Cmin , Cmax ] Example: cost(T4 ) ∈ [2.138803, 2.472136] so that CEmin (Tp ) = .5838004 CEmin (T) = .6108033 CEmax (Tp ) = .5482172 CEmax (T) = .6345125 16 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  56. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test

    Suites 0gkapfhamExp Revision : 1.2 T1 R1 R2 R3 T2 T3 T4 Approach: use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost(Tn ) Construction: set cost(T1 ) = cost(T2 ) = cost(T3 ) = 1 and then determine the bounds for cost(T4 ) ∈ [Cmin , Cmax ] Example: cost(T4 ) ∈ [2.138803, 2.472136] so that CEmin (Tp ) = .5838004 CEmin (T) = .6108033 CEmax (Tp ) = .5482172 CEmax (T) = .6345125 16 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  57. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites

    Objective: Simulate test suite execution on a centralized server 17 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  58. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites

    Objective: Simulate test suite execution on a centralized server 17 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  59. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites

    Construction: Combine all of the test suites and coverage reports 17 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  60. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization

    Random Number Input Output {1 , 2 , 3 , 4} 2 t1, t2, t3, |t4 t1, t4, t3, t2 {1 , 2 , 3} 3 t1, t4, |t3, t2 t1, t4, t3, t2 {1 , 2} 1 t1, |t4, t3, t2 t4, t1, t3, t2 18 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  61. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization

    Random Number Input Output {1 , 2 , 3 , 4} 2 t1, t2, t3, |t4 t1, t4, t3, t2 {1 , 2 , 3} 3 t1, t4, |t3, t2 t1, t4, t3, t2 {1 , 2} 1 t1, |t4, t3, t2 t4, t1, t3, t2 18 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  62. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization

    Random Number Input Output {1 , 2 , 3 , 4} 2 t1, t2, t3, |t4 t1, t4, t3, t2 {1 , 2 , 3} 3 t1, t4, |t3, t2 t1, t4, t3, t2 {1 , 2} 1 t1, |t4, t3, t2 t4, t1, t3, t2 18 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  63. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization

    Random Number Input Output {1 , 2 , 3 , 4} 2 t1, t2, t3, |t4 t1, t4, t3, t2 {1 , 2 , 3} 3 t1, t4, |t3, t2 t1, t4, t3, t2 {1 , 2} 1 t1, |t4, t3, t2 t4, t1, t3, t2 18 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  64. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization

    Random Number Input Output {1 , 2 , 3 , 4} 2 t1, t2, t3, |t4 t1, t4, t3, t2 {1 , 2 , 3} 3 t1, t4, |t3, t2 t1, t4, t3, t2 {1 , 2} 1 t1, |t4, t3, t2 t4, t1, t3, t2 Importance: Random prioritization serves as a valuable experimental control and often produces orderings better than the initial suite 18 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  65. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization

    Random Number Input Output {1 , 2 , 3 , 4} 2 t1, t2, t3, |t4 t1, t4, t3, t2 {1 , 2 , 3} 3 t1, t4, |t3, t2 t1, t4, t3, t2 {1 , 2} 1 t1, |t4, t3, t2 t4, t1, t3, t2 Strategy: Use the modern and efficient implementation of the Fisher-Yates shuffle to produce the reordered test suite Tp = t4 , t1 , t3 , t2 18 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  66. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends

    | metric: cost alg: 2OPT,GRD 0.2101 0.4946 0.4889 Reduction Factor for Time (RFFT) The myopic focus on cost leads to low RFFT values for 2OPT and GRD 19 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  67. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends

    | metric: cost alg: 2OPT,GRD 0.2101 0.4946 0.4889 Reduction Factor for Time (RFFT) The myopic focus on cost leads to low RFFT values for 2OPT and GRD 19 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  68. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends

    | alg: 2OPT,GRD metric: cost metric: coverage 0.1130 0.5967 0.4959 0.6136 Reduction Factor for Size (RFFS) DGR and HGS are the best at creating test suites that improve RFFS 20 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  69. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends

    | alg: 2OPT,GRD metric: cost metric: coverage 0.1130 0.5967 0.4959 0.6136 Reduction Factor for Size (RFFS) DGR and HGS are the best at creating test suites that improve RFFS 20 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  70. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends

    | alg: HGS metric: coverage alg: DGR 0.7520 0.8231 0.8344 0.9388 Coverage Effectiveness (CE) Using ratio and cost improves the CE of the prioritized test suite 21 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  71. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends

    | alg: HGS metric: coverage alg: DGR 0.7520 0.8231 0.8344 0.9388 Coverage Effectiveness (CE) Using ratio and cost improves the CE of the prioritized test suite 21 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  72. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time

    - SK Reduction Technique Reduction Factor for Time (RFFT) 0.0 0.1 0.2 0.3 0.4 2OPT DGR GRD HGS SK cost coverage ratio For 2OPT and GRD, ratio and coverage create the best test suites 22 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  73. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time

    - SK Reduction Technique Reduction Factor for Time (RFFT) 0.0 0.1 0.2 0.3 0.4 2OPT DGR GRD HGS SK cost coverage ratio For 2OPT and GRD, ratio and coverage create the best test suites 22 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  74. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size

    - SK Reduction Technique Reduction Factor for Size (RFFS) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 2OPT DGR GRD HGS SK cost coverage ratio It is often easy to construct test suites with high RFFS values 23 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  75. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size

    - SK Reduction Technique Reduction Factor for Size (RFFS) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 2OPT DGR GRD HGS SK cost coverage ratio It is often easy to construct test suites with high RFFS values 23 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  76. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio 24 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  77. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio DGR and HGS exhibit lackluster performance when reordering 24 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  78. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio Greedily reordered test suites are better than randomly prioritized ones 24 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  79. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio Greedy fooling test suites cause GRD and DGR to make low CE suites 24 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  80. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results -

    RP Prioritization Technique Coverage Effectiveness (CE) 0.4 0.5 0.6 0.7 0.8 0.9 2OPT DGR GRD HGS SK cost coverage ratio 2OPT uses lookahead and can construct high CE test prioritizations 24 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  81. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application

    CommonRate(Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 25 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  82. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application

    CommonRate(Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 25 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  83. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application

    CommonRate(Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 25 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  84. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application

    CommonRate(Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 25 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  85. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application

    CommonRate(Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Value of the common rate is relatively stable across methods 25 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  86. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Parameter

    Permutation Importance (PI) 0 10 20 30 alg metric CE alg metric RFFS alg metric RFFT 26 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  87. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Parameter

    Permutation Importance (PI) 0 10 20 30 alg metric CE alg metric RFFS alg metric RFFT Algorithm choice is most important for improving the CE of ordering 26 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  88. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Parameter

    Permutation Importance (PI) 0 10 20 30 alg metric CE alg metric RFFS alg metric RFFT Greedy choice metric has the greatest impact on the test suite reducers 26 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  89. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements Prioritization Technique

    Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD For the chosen case study applications, the techniques are fast 27 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  90. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements Prioritization Technique

    Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD For the chosen case study applications, the techniques are fast 27 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  91. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like

    APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness (FDE) and average percentage of faults detected (APFD). Consider search-based testing methods. 28 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  92. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like

    APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness (FDE) and average percentage of faults detected (APFD). Consider search-based testing methods. 28 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  93. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like

    APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness (FDE) and average percentage of faults detected (APFD). Consider search-based testing methods. 28 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  94. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like

    APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness (FDE) and average percentage of faults detected (APFD). Consider search-based testing methods. 28 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  95. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - Reduce And

    prIortize SuitEs http://raise.googlecode.com/ provides tools, data sets, and resources 29 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  96. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - Reduce And

    prIortize SuitEs Interactive visualization methods enable testers to find best ordering 29 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  97. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 30 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  98. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 30 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  99. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 30 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  100. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 30 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods
  101. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks Program Analysis

    Technique Test Suite Reduction Prioritization Regression Testing Techniques Prioritization Technique Execution Time (ms) 0 20 40 60 80 100 2OPT DGR GRD HGS JD Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 30 / 30 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods