Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

Transcript

1. Software Quality Improvement through Repeated Test Execution: An Exploration of

   the Present and Future of Regression Testing Gregory M. Kapfhammer† Department of Computer Science Allegheny College http://www.cs.allegheny.edu/∼gkapfham/ University of Delhi – May 9, 2012 †Joint with Jonathan Miller Kauffman (Allegheny College)

2. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Important Points

   Presenter Introduction: Gregory M. Kapfhammer test testing software suites prioritization components coverage empirically regression suite applications evaluating algorithm analysis data effectiveness empirical genetic performance understanding using approach commercialofftheshelf comparison comprehensive constrained creation databasecentric environments execution finding framework identifying interactive java methods multiplots mutation party prioritized reduction relational study third timeaware towards 105 adequacy approaches array automatically building call challenges chapter communication compare compressing computer conditional cost cots covering creating criteria database databaseaware databasedriven databases declarative dependable detection devices distributed distributing duringsearchbased dynamic efficiency efficient engineering environment evaluate examination executing experimental family flow forward frameworks free generation greedy gui hamiltonian handbook heaps implementation improve incorporating increase information initial intranode invariant javaspace javaspacebased kernel knapsack linux measurement memory method monitoring operators paths poster potential preliminary primitives prioritizations prioritizers prioritizing problematic receive remote reports resource resourceconstrained results role runtime science searchbased selection solutions solvers space studies studying supported synthetic techniques transmission transparently trees tuple unstructured wrappers Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

3. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

4. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

5. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

6. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

7. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scientific Device Program Household Appliance Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

8. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Everywhere Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scientific Device Program Household Appliance Program Network Router Program Scientific Device Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

9. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

   Software is Complex Computer Software Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

10. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

11. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

12. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

13. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Execution Environments Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

14. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Complex Computer Software Lines of Code Numerous Features Feature Interactions Execution Environments Software entities are more complex for their size than per- haps any other human construct - Frederick P. Brooks, Jr. Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

15. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

16. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

17. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Program Changed because of the addition of a new feature or the correction of a defect Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

18. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

19. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Execution Environment Program Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

20. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Software is Evolving Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

21. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

22. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

23. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Complex Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

24. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Complex Evolving Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

25. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Software Challenges

    Regression Testing to the Rescue Computer Software Pervasive Complex Evolving Regression Testing supports the efficient construction of pervasive software that is complex and rapidly evolving Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

26. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

27. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Input Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

28. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Input Output Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

29. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

30. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

31. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

32. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

33. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

34. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

35. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

36. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

37. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct! Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

38. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

39. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found! Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

40. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

41. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

42. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

43. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

44. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

45. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

46. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

47. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

48. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

49. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

50. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    What is a Test Suite? T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Eclipse Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

51. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

52. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

53. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

54. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

55. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique What if Some Test Cases are More Effective? T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

56. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

57. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

58. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn What if Some Test Cases are Redundant? T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

59. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn What if Only Certain Tests are Needed? T2 Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

60. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Testing Opportunities

    Test Suite Management T1 T2 T3 T4 . . . Tn Regression Testing Technique T3 Tn Prioritization T3 Tn T1 T4 . . . T2 T1 T2 T3 T4 . . . Tn T1 T2 T3 T4 . . . Tn Reduction T4 T1 T2 T3 T4 . . . Tn What if Only Certain Tests are Needed? T2 Tn Selection T1 T2 T3 T4 . . . Tn Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

61. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

62. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

63. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

64. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

65. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

66. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

67. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

68. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

69. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

70. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Use the Coverage Report During the Next Round of Regression Testing Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

71. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Use the Same Test Suite for the Next Round of Regression Testing Standard Repeat Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

72. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Make a New Test Suite for the Next Round of Regression Testing Version Specific Repeat Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

73. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Model of Regression Testing Start Coverage Report Selection Reduction Prioritization Original Test Suite Modified Test Suite Test Suite Execution Test Coverage Monitoring Program Adequacy Criterion Test Results End Standard Repeat Make a New Test Suite for the Next Round of Regression Testing Version Specific Repeat Our Tools Support All of the Phases in this Model! Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

74. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

75. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

76. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

77. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

78. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

79. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

80. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

81. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

82. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

83. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

84. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

85. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

86. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

87. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

88. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

89. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Adequacy T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

90. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

91. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

92. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

93. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 Run Test Case Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

94. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 Passing Test Case: OA = OE Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

95. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

96. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

97. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 Run Test Case Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

98. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

99. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

    Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

100. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

101. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

102. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Continue Running T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

103. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Suite Execution T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework T1 T1 T2 T3 T4 T5 T6 T7 T7 Failing Test Case: OA = OE Stop Running T T8 T9 T10 T8 T9 T10 Continue Running T T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

104. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

105. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

106. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Cobertura Test Coverage Monitor Proteja Test Suite Manager Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

107. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 JUnit Test Automation Framework Cobertura Test Coverage Monitor Proteja Test Suite Manager T1 Run Test Case Collect Per-Test Case Coverage Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

108. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

109. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

110. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

111. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Requirements Set R for ... Statement Coverage Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

112. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Requirements Set R for ... Mutation Coverage Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

113. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Supporting Methods

     Test Coverage Monitoring T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Requirements Set R for ... Definition-Use Coverage Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

114. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0, 1] Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

115. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 1,2,3 Test Orderings Original ordering exhibits poor effectiveness score - CE = 0.3789 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

116. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 3, 2 CE = 0.5053 1,2,3 Test Orderings 1,2,3 1,3,2 Different ordering improves the effectiveness score - CE = 0.5053 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

117. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 2, 3, 1 CE = 0.4316 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 Some orderings have less improved scores - CE = 0.4316 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

118. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Best ordering shows a higher effectiveness scores - CE = 0.5789 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

119. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Greedy methods often produce high-effectiveness orderings Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

120. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Importance of Test Suite Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Search-based techniques may have some desirable characteristics Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

121. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

122. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

123. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

124. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

125. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 R12 R12 R12 R12 R12 R12 R12 R12 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

126. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

127. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 ratio(T1) = cost(T1) coverage(T1) Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

128. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 ratio(T1) = 5 3 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

129. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} R12 ratio(T1) = 1.66 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

130. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

131. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

132. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 ratio(T5) = cost(T5) coverage(T5) Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

133. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 ratio(T5) = 8 3 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

134. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 R12 ratio(T5) = 2.66 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

135. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

136. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 ratio(T1) < ratio(T5) Prefer T1 over T5 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

137. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} 1.66 2.66 Proceed incrementally, picking the test case with the lowest ratio value for the uncovered requirements Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

138. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 Test Suite T = T8, T4, T9, T1, T10, T3, T7, T2, T6, T5 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

139. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

140. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

141. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

142. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

143. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

144. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

145. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

146. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

147. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

148. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 R12 1 2 3 5 2 4 4 6 10 8 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

149. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 T2 T6 T5 Test Suite T = T8, T4, T9, T1, T10, T3, T7 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

150. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Greedy Algorithms T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Requirements Set R = {R1, R2, . . . , R11, R12} T8 T4 T9 T1 T10 T3 T7 T2 T6 T5 T2 T6 T5 Test Suite T = T8, T4, T9, T1, T10, T3, T7 R12 R12 R12 R12 R12 R12 R12 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

151. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 Possible configuration of the coverage report Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

152. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

153. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

154. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 Execution time of the test cases may mislead greedy Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

155. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 Execution time of the test cases may mislead greedy Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

156. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 Original ordering has low effectiveness score Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

157. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 CE(T) = 0.54 Original ordering has low effectiveness score Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

158. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 Greedy method constructs suite with marginal improvement Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

159. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 CE(T ) = 0.55 Greedy method constructs suite with marginal improvement Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

160. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 CE(T ) = 0.55 Greedy can exhibit high run-times (Jiang et al. ASE 2009) Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

161. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 Genetic may find better orderings (Conrad et al. GECCO 2010) Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

162. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Genetic may find better orderings (Conrad et al. GECCO 2010) Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

163. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Search-based algorithms are amenable to parallelization Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

164. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Search-based algorithms support “human in the loop” Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

165. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Limitations of Greedy Algorithms T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1) = 1 time(T2) = 1 time(T4) = 1 time(T3) = 2.45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T ) = 0.63 Search-based algorithms construct diverse test orderings Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

166. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

167. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

168. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

169. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

170. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 T4, T2, T3, T1, T5 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

171. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm Explore the “neighborhood” of test suites from a starting point T1, T2, T3, T4, T5 T2, T1, T3, T4, T5 T3, T2, T1, T4, T5 T4, T2, T3, T1, T5 T5, T2, T3, T4, T1 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

172. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

173. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

174. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

175. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

176. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T T T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

177. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Key Algorithms

     Hill Climbing Algorithm fitness time T T T T T Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

178. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

179. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

180. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

181. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

182. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

183. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

184. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

185. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Iteratively Perform Visualization and Statistical Analysis Repeat Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

186. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Iteratively Perform Visualization and Statistical Analysis Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

187. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Our tools support all of these tasks! Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

188. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Model for

     Experimentation How Do I Evaluate Regression Testing Methods? Start Programs Test Suites Conduct Experiments Regression Testing Techniques Technique Configurations Data Sets Visualization Statistical Analysis Graphs Diagrams Summarized Data Sets Models Repeat End Conduct Experiments with Additional Programs, Test Suites, and Techniques Repeat Greedy, Hill Climbing, Random, Adaptive Random, Simulated Annealing, Genetic Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

189. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Case Study Applications Application Program Size Lines Methods Classes Faults Test Cases CommissionEmployee 34 18 2 95 15 Point 125 26 3 44 13 DataStructures 189 57 8 324 106 Employee 192 52 7 84 14 LoopFinder 193 26 4 34 13 Sudoku 231 58 4 414 25 JDepend 1,462 282 35 2,659 39 Reduction and Prioritization 2,050 211 19 1,412 38 Barbecue 2,501 422 59 18,312 140 JodaTime 12,687 3,644 223 20,894 206 CommonsMath 20,763 4,185 556 6,077 268 Total 40,427 8,981 920 50,349 877 Average 3,675 816 84 4,577 80 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

190. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

191. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

192. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

193. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

194. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

195. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

196. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Greedy and hill climbing produce comparable orderings Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

197. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime However, hill climbing is slightly more efficient than greedy Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

198. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime Greedy produces a slightly better ordering than hill climbing Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

199. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime But the hill climbing algorithm executes over four times faster! Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

200. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Greedy and Search-Based Prioritizer NAPSC Runtime (sec) Application GRD 0.98 0.22 Sudoku GRD 0.36 0.38 ReductionAndPrioritization GRD 0.71 33.88 JodaTime HC SA FS 0.98 0.01 Sudoku HC SA FS 0.36 0.04 ReductionAndPrioritization HC SA FS 0.69 7.88 JodaTime A small NAPSC increase may result in a large runtime increase Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

201. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Population Size NAPSC 0.3240 0.3245 0.3250 0.3255 0.3260 0.3265 10 20 30 JodaTime (JT) Statement Original NAPSC Score: 0.2784 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

202. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Population Size NAPSC 0.3240 0.3245 0.3250 0.3255 0.3260 0.3265 10 20 30 JodaTime (JT) Statement Negligible NAPSC increase as population size increases Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

203. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Population Size Runtime (seconds) 10 15 20 10 20 30 JodaTime (JT) Statement Increases in runtime are more marked Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

204. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Similarity Metric NAPSC 0.3194 0.3195 0.3196 0.3197 Euclidean Jaccard Manhattan JodaTime (JT) Statement Original NAPSC Score: 0.2784 Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

205. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Similarity Metric NAPSC 0.3194 0.3195 0.3196 0.3197 Euclidean Jaccard Manhattan JodaTime (JT) Statement NAPSC changes little as similarity metric is varied Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

206. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Similarity Metric NAPSC 0.3194 0.3195 0.3196 0.3197 Euclidean Jaccard Manhattan JodaTime (JT) Statement Scores are comparable to those produced by random (0.3240 - 0.3265) Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

207. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Similarity Metric Runtime (seconds) 80 90 100 110 120 Euclidean Jaccard Manhattan JodaTime (JT) Statement Adaptive random executes more slowly than random Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

208. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Concrete Examples

     Empirical Results: Random and Adaptive Random Similarity Metric Runtime (seconds) 80 90 100 110 120 Euclidean Jaccard Manhattan JodaTime (JT) Statement Choose random because it produces comparable NAPSC scores in less time Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

209. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Future The

     Future of Regression Testing Research • Reproducible research by releasing software tools and data • Integrate many existing algorithms into a single framework • Develop new forums for publishing important results • Software Quality Journal special issue • International Workshop on Regression Testing Practice • Encourage the use of coarse-grained information • Try to apply existing tools to industrial programs • Participate in community events; publish experience reports Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

210. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Future The

     Future of Regression Testing Research • Reproducible research by releasing software tools and data • Integrate many existing algorithms into a single framework • Develop new forums for publishing important results • Software Quality Journal special issue • International Workshop on Regression Testing Practice • Encourage the use of coarse-grained information • Try to apply existing tools to industrial programs • Participate in community events; publish experience reports Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

211. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Conclusions Conclusions

     and Future Work Concluding Remarks • Comprehensive framework for regression testing • Interesting empirical results demonstrate trade-offs • Free/open source tools are available for download • http://proteja.googlecode.com • http://modificare.googlecode.com Future Work • Add new algorithms for regression testing • Conduct experiments with more case study applications • Further develop statistically meaningful empirical results Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

212. Introduction Software Testing Regression Testing Empirical Evaluation Conclusion Conclusions Conclusions

     and Future Work Concluding Remarks • Comprehensive framework for regression testing • Interesting empirical results demonstrate trade-offs • Free/open source tools are available for download • http://proteja.googlecode.com • http://modificare.googlecode.com Future Work • Add new algorithms for regression testing • Conduct experiments with more case study applications • Further develop statistically meaningful empirical results Kapfhammer Allegheny College Software Quality Improvement through Repeated Test Execution: An Exploration of the Present and Future of Regression Testing

213. Software Quality Improvement through Repeated Test Execution: An Exploration of

     the Present and Future of Regression Testing Gregory M. Kapfhammer http://www.cs.allegheny.edu/∼gkapfham/ Thank you for your attention! Contact me with questions and/or comments!