Regression testing: Theoretical underpinnings, practical techniques, and empirical insights

Transcript

1. 1.

   Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights Gregory

   M. Kapfhammer∗ and Jonathan Miller Kauffman† Department of Computer Science Allegheny College ∗http://www.cs.allegheny.edu/∼gkapfham/ †https://sites.google.com/a/allegheny.edu/jonathan-kauffman/ 15th International Conference on Software Engineering and Applications December 14 - 16, 2011
2. 2.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Important Points

   Accessing the Presentation Scan this QR Code with your smartphone! ... or, visit this Web site: http://www.cs.allegheny.edu/˜gkapfham/sea2011.pdf ... or, ask us for a USB drive! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
3. 3.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Important Points

   Presenter Introduction: Gregory M. Kapfhammer test testing software suites prioritization components coverage empirically regression suite applications evaluating algorithm analysis data effectiveness empirical genetic performance understanding using approach commercialofftheshelf comparison comprehensive constrained creation databasecentric environments execution finding framework identifying interactive java methods multiplots mutation party prioritized reduction relational study third timeaware towards 105 adequacy approaches array automatically building call challenges chapter communication compare compressing computer conditional cost cots covering creating criteria database databaseaware databasedriven databases declarative dependable detection devices distributed distributing duringsearchbased dynamic efficiency efficient engineering environment evaluate examination executing experimental family flow forward frameworks free generation greedy gui hamiltonian handbook heaps implementation improve incorporating increase information initial intranode invariant javaspace javaspacebased kernel knapsack linux measurement memory method monitoring operators paths poster potential preliminary primitives prioritizations prioritizers prioritizing problematic receive remote reports resource resourceconstrained results role runtime science searchbased selection solutions solvers space studies studying supported synthetic techniques transmission transparently trees tuple unstructured wrappers Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
4. 4.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Important Points

   Presenter Introduction: Jonathan Miller Kauffman ACM FOSS Best Genetic Array Alden R Scholar XInteract Algorithm Meadville tools project Java computer Covering student undergraduate learning per−test science Generation Science teaching assistant research updates recitation laboratory coverage scientist Introduction Computer Principles programming software development regression President Empirically Identifying Organization Allegheny replication validation Bachelor contributions demonstrations publications presentations experimentation Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
5. 5.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
6. 6.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
7. 7.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
8. 8.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
9. 9.

   Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
10. 10.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scientific Device Program Household Appliance Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
11. 11.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scientific Device Program Household Appliance Program Network Router Program Scientific Device Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
12. 12.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
13. 13.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
14. 14.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
15. 15.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
16. 16.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Execution Environments Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
17. 17.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Execution Environments Software entities are more complex for their size than per- haps any other human construct - Frederick P. Brooks, Jr. Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
18. 18.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
19. 19.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
20. 20.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Program Changed because of the addition of a new feature or the correction of a defect Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
21. 21.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
22. 22.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Execution Environment Program Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
23. 23.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
24. 24.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
25. 25.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
26. 26.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Complex Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
27. 27.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Complex Evolving Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
28. 28.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Complex Evolving Regression Testing supports the efficient construction of pervasive software that is complex and rapidly evolving Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
29. 29.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
30. 30.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
31. 31.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Input Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
32. 32.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Input Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
33. 33.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
34. 34.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
35. 35.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
36. 36.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
37. 37.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
38. 38.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
39. 39.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
40. 40.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
41. 41.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
42. 42.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
43. 43.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
44. 44.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
45. 45.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
46. 46.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
47. 47.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
48. 48.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
49. 49.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
50. 50.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
51. 51.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
52. 52.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
53. 53.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
54. 54.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Basic Concepts

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Eclipse Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
55. 55.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
56. 56.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
57. 57.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
58. 58.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
59. 59.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
60. 60.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
61. 61.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
62. 62.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
63. 63.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
64. 64.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn What if Only Certain Tests are Needed? T2 Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
65. 65.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn What if Only Certain Tests are Needed? T2 Tn Selection T1 T2 T3 T4 . . . Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
66. 66.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
67. 67.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
68. 68.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
69. 69.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
70. 70.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
71. 71.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
72. 72.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
73. 73.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
74. 74.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
75. 75.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
76. 76.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Use the Coverage Report During the Next Round of Regression Testing Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
77. 77.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Use the Same Test Suite for the Next Round of Regression Testing Standard Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
78. 78.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Make a New Test Suite for the Next Round of Regression Testing Version Specific Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
79. 79.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Make a New Test Suite for the Next Round of Regression Testing Version Specific Repeat Our Tools Support All of the Phases in this Model! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
80. 80.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
81. 81.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
82. 82.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
83. 83.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
84. 84.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
85. 85.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
86. 86.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
87. 87.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
88. 88.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
89. 89.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
90. 90.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
91. 91.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
92. 92.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
93. 93.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
94. 94.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
95. 95.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
96. 96.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
97. 97.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
98. 98.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
99. 99.

    Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 Run Test Case Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
100. 100.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 Passing Test Case: OA = OE Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
101. 101.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
102. 102.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
103. 103.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 Run Test Case Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
104. 104.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
105. 105.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
106. 106.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
107. 107.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
108. 108.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Continue Running T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
109. 109.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Continue Running T T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
110. 110.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
111. 111.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
112. 112.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Cobertura Test Coverage Monitor Proteja Test Suite Manager Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
113. 113.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Cobertura Test Coverage Monitor Proteja Test Suite Manager T1 Run Test Case Collect Per-Test Case Coverage Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
114. 114.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
115. 115.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
116. 116.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
117. 117.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Requirements Set R for ... Statement Coverage Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
118. 118.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Requirements Set R for ... Mutation Coverage Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
119. 119.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Requirements Set R for ... Definition-Use Coverage Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
120. 120.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Hands-on: Preparing the Applications and Tools 1 Compile the Sudoku Case Study Application cd proteja-installation-directory cd apps/SK ant init ant compile cd ../../ Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
121. 121.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Hands-on: Preparing the Applications and Tools 1 Compile the Sudoku Case Study Application cd proteja-installation-directory cd apps/SK ant init ant compile cd ../../ 2 Compile the Proteja Test Coverage Monitor cd ../../ ant init ant build Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
122. 122.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Hands-on: Executing Tests and Monitoring Coverage 1 Run Sudoku’s Test Suite Without Coverage Monitoring cp data/settingsFiles/SK/* . ant addAnnotations ant proteja Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
123. 123.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Hands-on: Executing Tests and Monitoring Coverage 1 Run Sudoku’s Test Suite Without Coverage Monitoring cp data/settingsFiles/SK/* . ant addAnnotations ant proteja 2 Run Sudoku’s Test Suite With Coverage Monitoring ant proteja -Dcobertura=true -DtestTimer=true -DapplicationName=Sudoku Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
124. 124.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Screenshot: Configuring Test Suite Execution Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
125. 125.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Screenshot: Configuring the Test Coverage Monitor Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
126. 126.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Screenshot: Running the Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
127. 127.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Screenshot: Performing Test Coverage Monitoring Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
128. 128.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Screenshot: Viewing a Test Coverage Report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
129. 129.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Screenshot: Viewing a Test Case Timings Report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
130. 130.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
131. 131.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1] Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
132. 132.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 1,2,3 Test Orderings Original ordering exhibits poor effectiveness score - CE = 0.3789 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
133. 133.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 3, 2 CE = 0.5053 1,2,3 Test Orderings 1,2,3 1,3,2 Different ordering improves the effectiveness score - CE = 0.5053 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
134. 134.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 2, 3, 1 CE = 0.4316 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 Some orderings have less improved scores - CE = 0.4316 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
135. 135.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Best ordering shows a higher effectiveness scores - CE = 0.5789 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
136. 136.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Greedy methods often produce high-effectiveness orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
137. 137.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Search-based techniques may have some desirable characteristics Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
138. 138.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
139. 139.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
140. 140.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
141. 141.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
142. 142.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
143. 143.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
144. 144.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 ratio(T1) = cost(T1) coverage(T1) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
145. 145.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 ratio(T1) = 5 3 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
146. 146.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 ratio(T1) = 1.66 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
147. 147.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
148. 148.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
149. 149.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 ratio(T5) = cost(T5) coverage(T5) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
150. 150.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 ratio(T5) = 8 3 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
151. 151.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 ratio(T5) = 2.66 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
152. 152.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
153. 153.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 ratio(T1) < ratio(T5) Prefer T1 over T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
154. 154.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 Proceed incrementally, picking the test case with the lowest ratio value for the uncovered requirements Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
155. 155.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 Test Suite T = T8, T4, T9, T1, T10, T3, T7, T2, T6, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
156. 156.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
157. 157.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
158. 158.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
159. 159.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
160. 160.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
161. 161.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
162. 162.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
163. 163.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
164. 164.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
165. 165.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
166. 166.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 T2 T6 T5 Test Suite T = T8, T4, T9, T1, T10, T3, T7 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
167. 167.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 T2 T6 T5 Test Suite T = T8, T4, T9, T1, T10, T3, T7 R12 R12 R12 R12 R12 R12 R12 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
168. 168.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 1 Load the Regression Testing Techniques cd modificare-installation-directory R source(‘‘FDP Start.R’’) zLoadFit() zLoadGRD() zLoadHC() zLoadGA() zLoadProduceOrderingReductionDataFile() Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
169. 169.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 2 Run the Greedy Reduction Algorithm x1<-GRD reduction(coverageMatrixName= "reqMatrices/StatementCoverageMatrices/ Sudoku 1 line true Coverage.dat", timingsFileName="reqMatrices/ TimingsFiles/Sudoku Timing.dat") Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
170. 170.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 3 Run the Greedy Prioritization Algorithm x2<-GRD(coverageMatrixName= "reqMatrices/StatementCoverageMatrices/ Sudoku 1 line true Coverage.dat", timingsFileName="reqMatrices/ TimingsFiles/Sudoku Timing.dat") Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
171. 171.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 4 Map the Test Indices to Test Names y1<-produceOrderingReductionDataFile (testSuite=x1$Ord, timingsFile="reqMatrices/ TimingsFiles/Sudoku Timing.dat") y2<-produceOrderingReductionDataFile (testSuite=x2$Ord, timingsFile="reqMatrices/ TimingsFiles/Sudoku Timing.dat") Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
172. 172.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 5 Store the Test Case Timings write.table(y1,file="path-to-proteja/ SK ordering1.dat", row.names=FALSE, col.names=FALSE,quote=FALSE) write.table(y2,file="path-to-proteja/ SK ordering2.dat", row.names=FALSE, col.names=FALSE,quote=FALSE) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
173. 173.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 6 Execute the Reduced Test Suite cd proteja-installation-directory ant modifyTestSuite -DmodificationFile= "SK ordering1.dat" ant proteja Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
174. 174.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Greedy Algorithms 7 Execute the Prioritized Test Suite cd proteja-installation-directory ant modifyTestSuite -DmodificationFile= "SK ordering2.dat" ant proteja Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
175. 175.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Loading Regression Testing Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
176. 176.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Running the Greedy Reducer Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
177. 177.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Running the Greedy Prioritizer Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
178. 178.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Test Index to Test Name Mapping Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
179. 179.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Storing the Test Case Timings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
180. 180.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: The Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
181. 181.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: The Prioritized Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
182. 182.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Preparing the Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
183. 183.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Viewing the Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
184. 184.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Running the Reduced Test Suite Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
185. 185.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 Possible configuration of the coverage report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
186. 186.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
187. 187.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
188. 188.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 Execution time of the test cases may mislead greedy Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
189. 189.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 Execution time of the test cases may mislead greedy Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
190. 190.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 Original ordering has low effectiveness score Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
191. 191.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 CE(T) = 0.54 Original ordering has low effectiveness score Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
192. 192.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 Greedy method constructs suite with marginal improvement Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
193. 193.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 CE(T ) = 0.55 Greedy method constructs suite with marginal improvement Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
194. 194.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 CE(T ) = 0.55 Greedy can exhibit high run-times (Jiang et al. ASE 2009) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
195. 195.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 Genetic may find better orderings (Conrad et al. GECCO 2010) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
196. 196.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Genetic may find better orderings (Conrad et al. GECCO 2010) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
197. 197.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Search-based algorithms are amenable to parallelization Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
198. 198.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Search-based algorithms support “human in the loop” Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
199. 199.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Search-based algorithms construct diverse test orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
200. 200.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
201. 201.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
202. 202.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
203. 203.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
204. 204.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 T4, T2, T3, T1, T5 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
205. 205.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 T4, T2, T3, T1, T5 T5, T2, T3, T4, T1 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
206. 206.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
207. 207.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
208. 208.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
209. 209.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
210. 210.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
211. 211.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T T T T Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
212. 212.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Randomly create suites by repeatedly shuffling T1, . . . , Tn Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
213. 213.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Execute the phases until the genetic algorithm stagnates Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
214. 214.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Use a “higher is better” effectiveness metric Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
215. 215.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators CE Use a “higher is better” effectiveness metric Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
216. 216.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators CE APFD APRC Use a “higher is better” effectiveness metric Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
217. 217.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Choose orderings to become parents of next generation Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
218. 218.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Choose orderings to become parents of next generation Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
219. 219.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators ROU TOU TRU Choose orderings to become parents of next generation Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
220. 220.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Five possible operators combine parents to produce children Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
221. 221.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Five possible operators combine parents to produce children Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
222. 222.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators MPX PMX POS Five possible operators combine parents to produce children Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
223. 223.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Six possible operators make random changes to orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
224. 224.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Six possible operators make random changes to orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
225. 225.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators ISM IVM SIM Six possible operators make random changes to orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
226. 226.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Genetic Algorithm Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators ISM IVM SIM See (Conrad et. al, GECCO 2010) for additional details Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
227. 227.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Search-Based Algorithms 1 Run the Hill Climbing Prioritization Algorithm HC FA(lFM=makeLogFM(read.table( "reqMatrices/StatementCoverageMatrices/ Sudoku 1 line true Coverage.dat")), NG="NG FS",Seed=100) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
228. 228.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hands-on: Running the Search-Based Algorithms 2 Run the Genetic Prioritization Algorithm GA(lFM=makeLogFM(read.table("reqMatrices/ StatementCoverageMatrices/ Sudoku 1 line true Coverage.dat")), cChildDens=.6,cMutRate=.2,cPopSize=20, CrossOp="CO OX1",MutOp="MO DM", SelOp="SO ROU", TransOp="TO EXP", TermCond="TermTotalIt", EndCond=10,Seed=100) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
229. 229.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Running the Hill Climbing Algorithm Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
230. 230.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Screenshot: Running the Genetic Algorithm Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
231. 231.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 1: read(”Enter 3 numbers:”,x,y,z); 2: m=z; 3: if(y<z) 4: if(x< y) 5: m=y; 6: else if(x<z) 7: m=y; // *** bug *** 8: else 9: if(x>y) 10: m=y; 11: else if (x>z) 12: m=z; 13: print(”Middle number is:”,m); } Pass/Fail Status P P P P P P F F Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
232. 232.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization suspiciousness(s) = failed(s) totalfailed ∗ (failed(s) + passed(s)) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
233. 233.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization suspiciousness(s) = failed(s) totalfailed ∗ (failed(s) + passed(s)) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
234. 234.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization suspiciousness(s) = failed(s) totalfailed ∗ (failed(s) + passed(s)) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
235. 235.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization suspiciousness(s) = failed(s) totalfailed ∗ (failed(s) + passed(s)) Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
236. 236.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
237. 237.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
238. 238.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
239. 239.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Examine the most suspicious statement first Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
240. 240.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Assumption: Developer examines source code statements in order Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
241. 241.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Assumption: Developer can identify faulty statement Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
242. 242.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Fault Localization mid() { int x,y,z,m; T1 T2 T3 T4 T5 T6 T7 T8 Susp Rank 1: read(”Enter 3 numbers:”,x,y,z); 0.5 7 2: m=z; 0.5 7 3: if(y<z) 0.5 7 4: if(x< y) 0.63 3 5: m=y; 0.0 13 6: else if(x<z) 0.71 2 7: m=y; // *** bug *** 0.82 1 8: else 0.0 13 9: if(x>y) 0.0 13 10: m=y; 0.0 13 11: else if (x>z) 0.0 13 12: m=z; 0.0 13 13: print(”Middle number is:”,m); 0.5 7 } Pass/Fail Status P P P P P P F F Our framework supports fault localization with a wide variety of metrics Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
243. 243.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation 1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
244. 244.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
245. 245.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
246. 246.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
247. 247.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
248. 248.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
249. 249.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
250. 250.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
251. 251.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Iteratively Perform Visualization and Statistical Analysis Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
252. 252.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Iteratively Perform Visualization and Statistical Analysis Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
253. 253.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Our tools support all of these tasks! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
254. 254.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     1 Introduction Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
255. 255.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Case Study Applications Application Program Size Lines Methods Classes Faults Test Cases CommissionEmployee 34 18 2 95 15 Point 125 26 3 44 13 DataStructures 189 57 8 324 106 Employee 192 52 7 84 14 LoopFinder 193 26 4 34 13 Sudoku 231 58 4 414 25 JDepend 1,462 282 35 2,659 39 Reduction and Prioritization 2,050 211 19 1,412 38 Barbecue 2,501 422 59 18,312 140 JodaTime 12,687 3,644 223 20,894 206 CommonsMath 20,763 4,185 556 6,077 268 Total 40,427 8,981 920 50,349 877 Average 3,675 816 84 4,577 80 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
256. 256.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Case Study Applications Application Program Size Lines Methods Classes Faults Test Cases CommissionEmployee 34 18 2 95 15 Point 125 26 3 44 13 DataStructures 189 57 8 324 106 Employee 192 52 7 84 14 LoopFinder 193 26 4 34 13 Sudoku 231 58 4 414 25 JDepend 1,462 282 35 2,659 39 Reduction and Prioritization 2,050 211 19 1,412 38 Barbecue 2,501 422 59 18,312 140 JodaTime 12,687 3,644 223 20,894 206 CommonsMath 20,763 4,185 556 6,077 268 Total 40,427 8,981 920 50,349 877 Average 3,675 816 84 4,577 80 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
257. 257.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Case Study Applications Application Program Size Lines Methods Classes Faults Test Cases CommissionEmployee 34 18 2 95 15 Point 125 26 3 44 13 DataStructures 189 57 8 324 106 Employee 192 52 7 84 14 LoopFinder 193 26 4 34 13 Sudoku 231 58 4 414 25 JDepend 1,462 282 35 2,659 39 Reduction and Prioritization 2,050 211 19 1,412 38 Barbecue 2,501 422 59 18,312 140 JodaTime 12,687 3,644 223 20,894 206 CommonsMath 20,763 4,185 556 6,077 268 Total 40,427 8,981 920 50,349 877 Average 3,675 816 84 4,577 80 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
258. 258.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Case Study Applications Application Program Size Lines Methods Classes Faults Test Cases CommissionEmployee 34 18 2 95 15 Point 125 26 3 44 13 DataStructures 189 57 8 324 106 Employee 192 52 7 84 14 LoopFinder 193 26 4 34 13 Sudoku 231 58 4 414 25 JDepend 1,462 282 35 2,659 39 Reduction and Prioritization 2,050 211 19 1,412 38 Barbecue 2,501 422 59 18,312 140 JodaTime 12,687 3,644 223 20,894 206 CommonsMath 20,763 4,185 556 6,077 268 Total 40,427 8,981 920 50,349 877 Average 3,675 816 84 4,577 80 Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
259. 259.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
260. 260.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
261. 261.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
262. 262.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
263. 263.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
264. 264.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
265. 265.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Greedy and hill climbing produce comparable orderings Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
266. 266.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime However, hill climbing is slightly more efficient than greedy Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
267. 267.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Greedy produces a slightly better ordering than hill climbing Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
268. 268.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime But the hill climbing algorithm executes over four times faster! Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
269. 269.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime A small fitness increase may result in a large runtime increase Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
270. 270.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Insights Prioritizer Fitness Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Our tools support efficient experimentation through parallelism Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
271. 271.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion 1 Introduction

     Important Points Software Challenges 2 Software Testing Basic Concepts Testing Opportunities 3 Regression Testing Supporting Methods Key Algorithms 4 Empirical Evaluation Model for Experimentation Concrete Examples 5 Conclusion Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
272. 272.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Conclusions and

     Future Work Concluding Remarks • Comprehensive framework for regression testing • Interesting empirical results demonstrate trade-offs • Free/open source tools are available for download • http://proteja.googlecode.com • http://modificare.googlecode.com Future Work • Add new algorithms for regression testing • Conduct experiments with more case study applications • Develop statistically meaningful empirical results Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
273. 273.

     Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Conclusions and

     Future Work Concluding Remarks • Comprehensive framework for regression testing • Interesting empirical results demonstrate trade-offs • Free/open source tools are available for download • http://proteja.googlecode.com • http://modificare.googlecode.com Future Work • Add new algorithms for regression testing • Conduct experiments with more case study applications • Develop statistically meaningful empirical results Kapfhammer and Kauffman Allegheny College Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights
274. 274.

     Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights Gregory

     M. Kapfhammer∗ and Jonathan Miller Kauffman† ∗http://www.cs.allegheny.edu/∼gkapfham/ †https://sites.google.com/a/allegheny.edu/jonathan-kauffman/ Thank you for your attention! Contact us with questions and/or comments!