Searching for the best tests

Transcript

1. “Searching” for the Best Tests An Introduction to Automated Software

   Testing with Search-Based Techniques Gregory M. Kapfhammer Department of Computer Science Allegheny College March 31, 2015

2. Software is Everywhere Software is pervasive — and so it

   must be thoroughly tested! Computer Server Program

3. Software is Everywhere Software is pervasive — and so it

   must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program

4. Software is Everywhere Software is pervasive — and so it

   must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program

5. Software is Everywhere Software is pervasive — and so it

   must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program

6. Software is Everywhere Software is pervasive — and so it

   must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scienti c Device Program Household Appliance Program

7. Software is Everywhere Software is pervasive — and so it

   must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scienti c Device Program Household Appliance Program Network Router Program Scienti c Device Program

8. Software is Complex Even simple programs are intricate — and

   difficult to test! Computer Software

9. Software is Complex Even simple programs are intricate — and

   difficult to test! Computer Software Lines of Code

10. Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features

11. Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions

12. Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions Runtime Environs

13. Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions Runtime Environs “Software entities are more complex for their size than perhaps any other hu- man construct” — Frederick P. Brooks, Jr.

14. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program

15. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program

16. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Program Changed because of the addition of a new feature or the correction of a defect

17. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program

18. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Execution Environment Program

19. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine

20. Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine “Release early, release often” means that programs are regularly updated

21. Motivating Example The computation of an object’s velocity presents challenges!

    K = 1 2 m × v2

22. Motivating Example The computation of an object’s velocity presents challenges!

    K = 1 2 m × v2

23. Motivating Example The computation of an object’s velocity presents challenges!

    K = 1 2 m × v2

24. Motivating Example The computation of an object’s velocity presents challenges!

    K = 1 2 m × v2

25. Computing Velocity public static String computeVelocity(int kinetic, int mass) {

    int velocitySquared = 0; int velocity = 0; StringBuffer finalVelocity = new StringBuffer(); if( mass != 0 ) { velocitySquared = 3 * (kinetic / mass); velocity = (int)Math.sqrt(velocitySquared); finalVelocity.append(velocity); } else { finalVelocity.append("Undefined"); } return finalVelocity.toString(); }

26. Important Questions Finding software defects is a challenging and rewarding

    task Can you nd the defect in this program?

27. Important Questions Finding software defects is a challenging and rewarding

    task Are there general purpose strategies for defect isolation?

28. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

29. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

30. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

31. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

32. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test

33. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Input

34. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Input Output

35. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up

36. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input

37. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output

38. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up

39. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle

40. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output

41. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict

42. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output

43. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct!

44. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output

45. What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found!

46. What is a Test Suite? A test suite is an

    organized collection of test cases T1

47. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2

48. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3

49. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4

50. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . .

51. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn

52. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite

53. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing?

54. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit

55. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant

56. What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Eclipse

57. A JUnit Test Case @Test public void testOne() { String

    expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); }

58. Another JUnit Test @Test public void testTwo() { String expected

    = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); }

59. Important Questions Not all tests have the same fault detection

    effectiveness! Will these test cases nd the fault in the example program?

60. Important Questions Not all tests have the same fault detection

    effectiveness! T1 assigns K = 5, m = 0 — Pass or fail?

61. Important Questions Not all tests have the same fault detection

    effectiveness! T2 assigns K = 0, m = 5 — Pass or fail?

62. Important Questions Not all tests have the same fault detection

    effectiveness! How do we study the e ectiveness of di erent test cases?

63. The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?

64. The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?

65. The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?

66. The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?

67. The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?

68. The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using the PIE model, will the test cases nd the defect in the program?

69. A JUnit Test Case — T1 @Test public void testOne()

    { String expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); } E I P   

70. A JUnit Test Case — T1 @Test public void testOne()

    { String expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); } E I P   

71. A JUnit Test Case — T2 @Test public void testTwo()

    { String expected = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); } E I P   

72. A JUnit Test Case — T2 @Test public void testTwo()

    { String expected = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); } E I P   

73. A JUnit Test Case — T3 @Test public void testThree()

    { String expected = new String("4"); String actual = Kinetic. computeVelocity(8,1); assertEquals(expected, actual); } E I P   

74. A JUnit Test Case — T3 @Test public void testThree()

    { String expected = new String("4"); String actual = Kinetic. computeVelocity(8,1); assertEquals(expected, actual); } E I P   

75. A JUnit Test Case — T4 @Test public void testFour()

    { String expected = new String("20"); String actual = Kinetic. computeVelocity(1000,5); assertEquals(expected, actual); } E I P   

76. A JUnit Test Case — T4 @Test public void testFour()

    { String expected = new String("20"); String actual = Kinetic. computeVelocity(1000,5); assertEquals(expected, actual); } E I P   

77. Test Suite Summary A test case must create specific inputs

    in order to cause failure! Test Case Status T1 Pass T2 Pass T3 Pass T4 Fail

78. Important Insights Software testing is fundamentally challenging — is there

    help? I shall not deny that the construction of these testing programs has been a major intellectual e ort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. Edsger W. Dijkstra, Communications of the ACM, 1968

79. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

80. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

81. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

82. The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

83. Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing

84. Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious

85. Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming

86. Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming Very Tedious

87. Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming Very Tedious Di cult

88. Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming Very Tedious Di cult Can we develop and employ methods that will automatically generate high- quality test cases for real-world software?

89. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing

90. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious

91. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming

92. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious

93. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious Di cult

94. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious Di cult Laborious Time Consuming Very Tedious Testing is less laborious and tedious because an algorithm generates the tests. While computational time is needed, a human can be less involved!

95. Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious Di cult Laborious Time Consuming Very Tedious Laborious Time Consuming Very Tedious Di cult Automated testing is less di cult since a good tness function can guide the algorithm to inputs that nd the faults

96. Random Testing It is easy to randomly generate tests —

    but how good are they? −6 −4 −2 0 2 4 6 −1 −0.5 0 0.5 1 0 2 4 6 8 10 12 14

97. Search-Based Testing Use a fitness function to guide the search

    to “good” values 0 0.2 0.4 0.6 0.8 1 0 0.5 1 0 0.5 1

98. Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2

99. Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4

100. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6

101. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8

102. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10

103. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10

104. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2

105. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4

106. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6

107. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8

108. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10

109. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12

110. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12}

111. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12

112. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12

113. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12

114. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4

115. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1

116. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1

117. Mutation Testing Let’s purposefully insert faults into the program under

     test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1 M8

118. Testing with EvoSuite This prototype can automatically generate real JUnit

     test suites! Evolutionary Testing

119. Testing with EvoSuite This prototype can automatically generate real JUnit

     test suites! Evolutionary Testing Representation

120. Testing with EvoSuite This prototype can automatically generate real JUnit

     test suites! Evolutionary Testing Representation Fitness Function

121. Testing with EvoSuite This prototype can automatically generate real JUnit

     test suites! Evolutionary Testing Representation Fitness Function Modify Program

122. Testing with EvoSuite This prototype can automatically generate real JUnit

     test suites! Evolutionary Testing Representation Fitness Function Modify Program Operators

123. Testing with EvoSuite This prototype can automatically generate real JUnit

     test suites! Evolutionary Testing Representation Fitness Function Modify Program Operators “1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite”, Empirical Software Engineering

124. Configuring EvoSuite This tool has many unique configurations — which

     are best? EvoSuite’s Con gurations

125. Configuring EvoSuite This tool has many unique configurations — which

     are best? EvoSuite’s Con gurations Random

126. Configuring EvoSuite This tool has many unique configurations — which

     are best? EvoSuite’s Con gurations Random Fixed-Random

127. Configuring EvoSuite This tool has many unique configurations — which

     are best? EvoSuite’s Con gurations Random Fixed-Random Genetic

128. Configuring EvoSuite This tool has many unique configurations — which

     are best? EvoSuite’s Con gurations Random Fixed-Random Genetic Regression

129. Configuring EvoSuite This tool has many unique configurations — which

     are best? EvoSuite’s Con gurations Random Fixed-Random Genetic Regression The tness function computes branch coverage, weak mutation score, or strong mutation score to guide the search to the “best” test cases

130. A Test from EvoSuite @Test public void test001() throws Throwable

     { int int0 = 0; String string0 = Kinetic.computeVelocity(int0, int0); assertEquals("Undefined", string0); assertNotNull(string0); int int1 = 1262; String string1 = Kinetic.computeVelocity(int0, int0); int int2 = 5349; String string2 = Kinetic.computeVelocity(int1, int2); int int3 = 0; int int4 = 3; String string3 = Kinetic.computeVelocity(int3, int4); Kinetic kinetic0 = new Kinetic(); }

131. Important Questions EvoSuite is an advanced, yet sometimes limited, testing

     tool Will EvoSuite’s test cases nd the fault in the example program?

132. Important Questions EvoSuite is an advanced, yet sometimes limited, testing

     tool What is missing from the test cases that EvoSuite generates?

133. Important Questions EvoSuite is an advanced, yet sometimes limited, testing

     tool How does the oracle problem in uence the e ectiveness of EvoSuite?

134. Important Questions EvoSuite is an advanced, yet sometimes limited, testing

     tool What are the fundamental limitations of automated testing?

135. No Silver Bullet Software tools are fundamentally limited — what

     is our hope? There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity. Frederick P. Brooks, Jr., Proceedings of the IFIP Tenth World Computing Conference, 1986

136. The Challenges of Software Development Pervasiveness of Software Complexity of

     Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

137. The Challenges of Software Development Pervasiveness of Software Complexity of

     Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

138. The Challenges of Software Development Pervasiveness of Software Complexity of

     Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

139. The Challenges of Software Development Pervasiveness of Software Complexity of

     Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion

140. The Software Crisis Solutions are available — but not always

     obvious or popular What are the solutions to the software crisis?

141. The Software Crisis Solutions are available — but not always

     obvious or popular Great Designers

142. The Software Crisis Solutions are available — but not always

     obvious or popular Unremitting Care

143. The Software Crisis Solutions are available — but not always

     obvious or popular Incremental Advances

144. Final Admonishment An epilogue called “Fifty Years of Wonder, Excitement,

     and Joy” Too many interests, too many exciting opportunities for learning, research, and thought. What a marvelous predicament! Not only is the end not in sight, but the pace is not slackening. We have many future joys. Frederick P. Brooks, Jr., The Mythical Man-Month, 1995