Using synthetic test suites to empirically compare search-based and greedy prioritizers

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Williams2010/

Transcript

1. Using Synthetic Test Suites to Empirically Compare Search-Based and Greedy Prioritizers Zachary Williams Gregory M. Kapfhammer Department of Computer Science Allegheny College http://www.cs.allegheny.edu/ Genetic and Evolutionary Computation Conference Late Breaking Abstract Workshop July 2010 Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 1 / 11
2. Important Contributions Synthetic Test Suites Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11
3. Important Contributions Synthetic Test Suites Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11
4. Important Contributions Synthetic Test Suites Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC 0.5 1 1.5 200 20 40 60 80 100 0.5 1 1.5 200 20 40 60 80 100 0.5 1 1.5 200 Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11
5. Important Contributions Synthetic Test Suites Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC 0.5 1 1.5 200 20 40 60 80 100 0.5 1 1.5 200 20 40 60 80 100 0.5 1 1.5 200 Detailed Empirical Study Use synthetic test suites to empirically evaluate the efficiency and effectiveness of search-based and greedy prioritizers Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 2 / 11
6. Overview of Regression Testing P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
7. Overview of Regression Testing P T Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
8. Overview of Regression Testing P T Execute Test Suite Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
9. Overview of Regression Testing P T Execute Test Suite Report Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
10. Overview of Regression Testing P T Execute Test Suite P Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
11. Overview of Regression Testing P T Execute Test Suite P Report Correct programing defect Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
12. Overview of Regression Testing P T Execute Test Suite P P Add new functionallity Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
13. Overview of Regression Testing P T Execute Test Suite P P Report Add new functionallity Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
14. Overview of Regression Testing P T Execute Test Suite P P T P Modify test suite Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
15. Overview of Regression Testing P T Execute Test Suite P P T P Report Modify test suite Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
16. Overview of Regression Testing P T Execute Test Suite P P T P Report Complete retesting is often prohibitively expensive Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 3 / 11
17. Regression Test Suite Prioritization R Requirements necessitate the coverage of the state and/or structure of a program under test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
18. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 Requirements necessitate the coverage of the state and/or structure of a program under test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
19. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 Requirements necessitate the coverage of the state and/or structure of a program under test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
20. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 Each test covers specific requirements in a certain amount of time and thus the ordering is critical Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
21. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 R3 R2 T2 R4 Each test covers specific requirements in a certain amount of time and thus the ordering is critical Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
22. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
23. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
24. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
25. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
26. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
27. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
28. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
29. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
30. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 Prioritized test suites cover requirements faster thus enabling the rapid detection of defects Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
31. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 Testers can use greedy (Rothermel et al. TSE 2001) and search-based (Li et al. TSE 2007) methods to reorder suites Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
32. Regression Test Suite Prioritization R R3 R4 R2 R1 R5 T2 T1 T3 T4 T5 T6 T7 T1 T2 T3 T4 T5 T6 T7 T2 T3 T5 T2 T3 T5 T1 T4 T6 T7 T6 T7 T2 T3 T5 T6 T7 T1 T4 T6 T7 T1 T4 T1 T4 T1 T4 QUESTION: Which prioritization technique is the best? Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 4 / 11
33. Existing Prioritization Techniques Greedy approaches select the next best test case Hill climbers search the state space for improved orderings Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11
34. Existing Prioritization Techniques Greedy approaches select the next best test case Hill climbers search the state space for improved orderings Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11
35. Existing Prioritization Techniques Greedy approaches select the next best test case Hill climbers search the state space for improved orderings Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11
36. Existing Prioritization Techniques Greedy approaches select the next best test case Hill climbers search the state space for improved orderings Conventional wisdom dictates that greedy generally out performs hill climbing in terms of both efficiency and effectiveness Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 5 / 11
37. Conducting an Empirical Evaluation Determine Problem These highlighted tasks are manual, expensive, and prone to error Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
38. Conducting an Empirical Evaluation Determine Problem Download Case Studies These highlighted tasks are manual, expensive, and prone to error Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
39. Conducting an Empirical Evaluation Determine Problem Download Case Studies Configure Applica- tions and Test Suites These highlighted tasks are manual, expensive, and prone to error Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
40. Conducting an Empirical Evaluation Determine Problem Download Case Studies Configure Applica- tions and Test Suites Produce Coverage Report These highlighted tasks are manual, expensive, and prone to error Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
41. Conducting an Empirical Evaluation Determine Problem Download Case Studies Configure Applica- tions and Test Suites Produce Coverage Report Conduct Experiment These highlighted tasks are manual, expensive, and prone to error Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
42. Conducting an Empirical Evaluation Determine Problem Download Case Studies Configure Applica- tions and Test Suites Produce Coverage Report Conduct Experiment Download Case Study Configure Applica- tions and Test Suites Produce Coverage Report These highlighted tasks are manual, expensive, and prone to error Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
43. Conducting an Empirical Evaluation Determine Problem Download Case Studies Configure Applica- tions and Test Suites Produce Coverage Report Conduct Experiment Generate Synthetic Test Suites Synthetically generating a test suite is automated, effective, and efficient Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
44. Conducting an Empirical Evaluation Determine Problem Download Case Studies Configure Applica- tions and Test Suites Produce Coverage Report Conduct Experiment Generate Synthetic Test Suites Generate Synthetic Test Suites Synthetically generating a test suite is automated, effective, and efficient Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 6 / 11
45. Generating Synthetic Test Suite Tests The total number of tests controls how many tests the suite will contain Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11
46. Generating Synthetic Test Suite Tests Requirements The total number of requirements governs how many requirements the test suite will cover Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11
47. Generating Synthetic Test Suite Tests Requirements Coverage Points The total number of coverage points controls how many unique test-requirement pairs the test suite will contain Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11
48. Generating Synthetic Test Suite Tests Requirements Coverage Points Balancing Approach The balancing configuration dictates how the coverage points will be distributed in the synthetic test suite Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11
49. Generating Synthetic Test Suite Tests Requirements Coverage Points Balancing Approach Synthetic Test Suite Generator Our empirical results show that synthetic generation takes less then 0.2 seconds for extremely large test suites Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11
50. Generating Synthetic Test Suite Tests Requirements Coverage Points Balancing Approach Synthetic Test Suite Generator Synthetic Test Suite (Coverage Report) Contains information concerning the requirements covered and the execution time of each test Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 7 / 11
51. Empirical Results – Prioritizer Efficiency Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC 0.5 1 1.5 200 GRD HC 0.5 1 1.5 200 GRD HC 0.5 1 1.5 200 HC demonstrated to be more efficient then GRD for large test suites Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 8 / 11
52. Empirical Results – Prioritizer Efficiency Prioritization Technique Execution Time (sec) 20 40 60 80 100 GRD HC 0.5 1 1.5 200 GRD HC 0.5 1 1.5 200 GRD HC 0.5 1 1.5 200 HC demonstrated to be more efficient then GRD for large test suites Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 8 / 11
53. Empirical Results – Prioritizer Effectiveness Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC 1.5 200 0.1 0.25 0.5 As the amount of coverage points in the test suite increases the performance of HC becomes comparable to that of GRD Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 9 / 11
54. Empirical Results – Prioritizer Effectiveness Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC 1.5 200 0.1 0.25 0.5 As the amount of coverage points in the test suite increases the performance of HC becomes comparable to that of GRD Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 9 / 11
55. Empirical Results – Prioritizer Effectiveness Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC 0.5 1 1.5 2 50 100 200 GRD HC 0.5 1 1.5 2 50 100 200 GRD HC 0.5 1 1.5 2 50 100 200 GRD HC 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.80 0.85 0.90 0.95 1.00 0.5 1 1.5 2 50 100 200 0.80 0.85 0.90 0.95 1.00 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.1 0.25 0.5 The trend is evident over a wide range of experimental configurations Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 10 / 11
56. Empirical Results – Prioritizer Effectiveness Prioritization Technique Coverage Effectiveness (CE) 0.80 0.85 0.90 0.95 1.00 GRD HC 0.5 1 1.5 2 50 100 200 GRD HC 0.5 1 1.5 2 50 100 200 GRD HC 0.5 1 1.5 2 50 100 200 GRD HC 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.80 0.85 0.90 0.95 1.00 0.5 1 1.5 2 50 100 200 0.80 0.85 0.90 0.95 1.00 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.5 1 1.5 2 50 100 200 0.1 0.25 0.5 The trend is evident over a wide range of experimental configurations Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 10 / 11
57. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11
58. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11
59. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11
60. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11
61. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11
62. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11
63. Conclusion and Future Work Conclusion Synthetic test suite generation is efficient Enable the identification of fundemental trade-offs Future Work Apply our technique to genetic and other algorithms Implement and evaluate new and different synthetic generators http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Zachary Williams (Allegheny College) Synthetic Test Suite Prioritization GECCO 2010 11 / 11