Searching for the best tests

Searching for the best tests

Interested in learning more about this topic? Visit this web site to see related software: https://www.gregorykapfhammer.com/software/

4ae30d49c8cc07e42d5a871efb9bcfba?s=128

Gregory Kapfhammer

May 31, 2015
Tweet

Transcript

  1. 1.

    “Searching” for the Best Tests An Introduction to Automated Software

    Testing with Search-Based Techniques Gregory M. Kapfhammer Department of Computer Science Allegheny College March 31, 2015
  2. 2.

    Software is Everywhere Software is pervasive — and so it

    must be thoroughly tested! Computer Server Program
  3. 3.

    Software is Everywhere Software is pervasive — and so it

    must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program
  4. 4.

    Software is Everywhere Software is pervasive — and so it

    must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program
  5. 5.

    Software is Everywhere Software is pervasive — and so it

    must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program
  6. 6.

    Software is Everywhere Software is pervasive — and so it

    must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scienti c Device Program Household Appliance Program
  7. 7.

    Software is Everywhere Software is pervasive — and so it

    must be thoroughly tested! Computer Server Program Computer Server Program Desktop Computer Program Desktop Computer Program Mobile Computer Program Mobile Computer Program Household Appliance Program Scienti c Device Program Household Appliance Program Network Router Program Scienti c Device Program
  8. 8.
  9. 9.

    Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code
  10. 10.

    Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features
  11. 11.

    Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions
  12. 12.

    Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions Runtime Environs
  13. 13.

    Software is Complex Even simple programs are intricate — and

    difficult to test! Computer Software Lines of Code Numerous Features Feature Interactions Runtime Environs “Software entities are more complex for their size than perhaps any other hu- man construct” — Frederick P. Brooks, Jr.
  14. 15.

    Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program
  15. 16.

    Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Program Changed because of the addition of a new feature or the correction of a defect
  16. 17.

    Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program
  17. 18.

    Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Execution Environment Program
  18. 19.

    Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine
  19. 20.

    Software is Evolving Software is continuously updated — making testing

    critical! Execution Environment Program Execution Environment Program Execution Environment Program Execution Environment Changed due to an up- grade in a kernel, device driver, or virtual machine “Release early, release often” means that programs are regularly updated
  20. 25.

    Computing Velocity public static String computeVelocity(int kinetic, int mass) {

    int velocitySquared = 0; int velocity = 0; StringBuffer finalVelocity = new StringBuffer(); if( mass != 0 ) { velocitySquared = 3 * (kinetic / mass); velocity = (int)Math.sqrt(velocitySquared); finalVelocity.append(velocity); } else { finalVelocity.append("Undefined"); } return finalVelocity.toString(); }
  21. 26.
  22. 27.

    Important Questions Finding software defects is a challenging and rewarding

    task Are there general purpose strategies for defect isolation?
  23. 28.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  24. 29.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  25. 30.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  26. 31.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  27. 32.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test
  28. 33.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Input
  29. 34.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Input Output
  30. 35.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up
  31. 36.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input
  32. 37.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output
  33. 38.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up
  34. 39.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle
  35. 40.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output
  36. 41.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict
  37. 42.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output
  38. 43.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct!
  39. 44.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output
  40. 45.

    What is a Test Case? A test case calls a

    method and checks its output with an oracle Method Un- der Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found!
  41. 46.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1
  42. 47.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2
  43. 48.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3
  44. 49.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4
  45. 50.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . .
  46. 51.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn
  47. 52.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite
  48. 53.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing?
  49. 54.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit
  50. 55.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant
  51. 56.

    What is a Test Suite? A test suite is an

    organized collection of test cases T1 T2 T3 T4 . . . Tn Organize the Test Cases into a Test Suite Tool Support for Software Testing? JUnit Apache Ant Eclipse
  52. 57.

    A JUnit Test Case @Test public void testOne() { String

    expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); }
  53. 58.

    Another JUnit Test @Test public void testTwo() { String expected

    = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); }
  54. 59.

    Important Questions Not all tests have the same fault detection

    effectiveness! Will these test cases nd the fault in the example program?
  55. 60.

    Important Questions Not all tests have the same fault detection

    effectiveness! T1 assigns K = 5, m = 0 — Pass or fail?
  56. 61.

    Important Questions Not all tests have the same fault detection

    effectiveness! T2 assigns K = 0, m = 5 — Pass or fail?
  57. 62.

    Important Questions Not all tests have the same fault detection

    effectiveness! How do we study the e ectiveness of di erent test cases?
  58. 63.

    The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?
  59. 64.

    The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?
  60. 65.

    The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?
  61. 66.

    The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?
  62. 67.

    The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using PIE, will the test cases nd the defect in the program?
  63. 68.

    The PIE Model There are necessary and sufficient conditions for

    fault detection Execute the faulty source code Infect the program’s data state Propagate to the program’s output All of these must occur before the fault manifests itself as a failure! Using the PIE model, will the test cases nd the defect in the program?
  64. 69.

    A JUnit Test Case — T1 @Test public void testOne()

    { String expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); } E I P   
  65. 70.

    A JUnit Test Case — T1 @Test public void testOne()

    { String expected = new String("Undefined"); String actual = Kinetic. computeVelocity(5,0); assertEquals(expected, actual); } E I P   
  66. 71.

    A JUnit Test Case — T2 @Test public void testTwo()

    { String expected = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); } E I P   
  67. 72.

    A JUnit Test Case — T2 @Test public void testTwo()

    { String expected = new String("0"); String actual = Kinetic. computeVelocity(0,5); assertEquals(expected, actual); } E I P   
  68. 73.

    A JUnit Test Case — T3 @Test public void testThree()

    { String expected = new String("4"); String actual = Kinetic. computeVelocity(8,1); assertEquals(expected, actual); } E I P   
  69. 74.

    A JUnit Test Case — T3 @Test public void testThree()

    { String expected = new String("4"); String actual = Kinetic. computeVelocity(8,1); assertEquals(expected, actual); } E I P   
  70. 75.

    A JUnit Test Case — T4 @Test public void testFour()

    { String expected = new String("20"); String actual = Kinetic. computeVelocity(1000,5); assertEquals(expected, actual); } E I P   
  71. 76.

    A JUnit Test Case — T4 @Test public void testFour()

    { String expected = new String("20"); String actual = Kinetic. computeVelocity(1000,5); assertEquals(expected, actual); } E I P   
  72. 77.

    Test Suite Summary A test case must create specific inputs

    in order to cause failure! Test Case Status T1 Pass T2 Pass T3 Pass T4 Fail
  73. 78.

    Important Insights Software testing is fundamentally challenging — is there

    help? I shall not deny that the construction of these testing programs has been a major intellectual e ort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. Edsger W. Dijkstra, Communications of the ACM, 1968
  74. 79.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  75. 80.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  76. 81.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  77. 82.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  78. 84.
  79. 85.

    Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming
  80. 86.

    Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming Very Tedious
  81. 87.

    Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming Very Tedious Di cult
  82. 88.

    Manual Testing While it has benefits, this industry standard may

    be limited! Manual Testing Laborious Time Consuming Very Tedious Di cult Can we develop and employ methods that will automatically generate high- quality test cases for real-world software?
  83. 91.

    Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming
  84. 92.

    Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious
  85. 93.

    Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious Di cult
  86. 94.

    Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious Di cult Laborious Time Consuming Very Tedious Testing is less laborious and tedious because an algorithm generates the tests. While computational time is needed, a human can be less involved!
  87. 95.

    Automated Testing Automatically generating tests is amazing — but does

    it work? Automated Testing Laborious Time Consuming Very Tedious Di cult Laborious Time Consuming Very Tedious Laborious Time Consuming Very Tedious Di cult Automated testing is less di cult since a good tness function can guide the algorithm to inputs that nd the faults
  88. 96.

    Random Testing It is easy to randomly generate tests —

    but how good are they? −6 −4 −2 0 2 4 6 −1 −0.5 0 0.5 1 0 2 4 6 8 10 12 14
  89. 97.

    Search-Based Testing Use a fitness function to guide the search

    to “good” values 0 0.2 0.4 0.6 0.8 1 0 0.5 1 0 0.5 1
  90. 103.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10
  91. 104.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2
  92. 105.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4
  93. 106.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6
  94. 107.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8
  95. 108.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10
  96. 109.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12
  97. 110.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12}
  98. 111.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12
  99. 112.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12
  100. 113.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12
  101. 114.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4
  102. 115.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1
  103. 116.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1
  104. 117.

    Mutation Testing Let’s purposefully insert faults into the program under

    test! T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 Set of Program Methods M = {M1, M2, . . . , M11, M12} M12 M12 M12 M12 M12 M12 M12 M12 M12 M12 M4 T1 M8
  105. 119.

    Testing with EvoSuite This prototype can automatically generate real JUnit

    test suites! Evolutionary Testing Representation
  106. 120.

    Testing with EvoSuite This prototype can automatically generate real JUnit

    test suites! Evolutionary Testing Representation Fitness Function
  107. 121.

    Testing with EvoSuite This prototype can automatically generate real JUnit

    test suites! Evolutionary Testing Representation Fitness Function Modify Program
  108. 122.

    Testing with EvoSuite This prototype can automatically generate real JUnit

    test suites! Evolutionary Testing Representation Fitness Function Modify Program Operators
  109. 123.

    Testing with EvoSuite This prototype can automatically generate real JUnit

    test suites! Evolutionary Testing Representation Fitness Function Modify Program Operators “1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite”, Empirical Software Engineering
  110. 125.
  111. 126.

    Configuring EvoSuite This tool has many unique configurations — which

    are best? EvoSuite’s Con gurations Random Fixed-Random
  112. 127.

    Configuring EvoSuite This tool has many unique configurations — which

    are best? EvoSuite’s Con gurations Random Fixed-Random Genetic
  113. 128.

    Configuring EvoSuite This tool has many unique configurations — which

    are best? EvoSuite’s Con gurations Random Fixed-Random Genetic Regression
  114. 129.

    Configuring EvoSuite This tool has many unique configurations — which

    are best? EvoSuite’s Con gurations Random Fixed-Random Genetic Regression The tness function computes branch coverage, weak mutation score, or strong mutation score to guide the search to the “best” test cases
  115. 130.

    A Test from EvoSuite @Test public void test001() throws Throwable

    { int int0 = 0; String string0 = Kinetic.computeVelocity(int0, int0); assertEquals("Undefined", string0); assertNotNull(string0); int int1 = 1262; String string1 = Kinetic.computeVelocity(int0, int0); int int2 = 5349; String string2 = Kinetic.computeVelocity(int1, int2); int int3 = 0; int int4 = 3; String string3 = Kinetic.computeVelocity(int3, int4); Kinetic kinetic0 = new Kinetic(); }
  116. 131.

    Important Questions EvoSuite is an advanced, yet sometimes limited, testing

    tool Will EvoSuite’s test cases nd the fault in the example program?
  117. 132.

    Important Questions EvoSuite is an advanced, yet sometimes limited, testing

    tool What is missing from the test cases that EvoSuite generates?
  118. 133.

    Important Questions EvoSuite is an advanced, yet sometimes limited, testing

    tool How does the oracle problem in uence the e ectiveness of EvoSuite?
  119. 134.

    Important Questions EvoSuite is an advanced, yet sometimes limited, testing

    tool What are the fundamental limitations of automated testing?
  120. 135.

    No Silver Bullet Software tools are fundamentally limited — what

    is our hope? There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity. Frederick P. Brooks, Jr., Proceedings of the IFIP Tenth World Computing Conference, 1986
  121. 136.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  122. 137.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  123. 138.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  124. 139.

    The Challenges of Software Development Pervasiveness of Software Complexity of

    Software Evolving Nature of Software Motivating Example Important Questions Bene ts of Software Testing Test Cases Test Suites Examples of Tests The PIE Model Test Case E ectiveness Search-Based Software Testing Testing Methods Random Testing Testing with EvoSuite Conclusion
  125. 140.

    The Software Crisis Solutions are available — but not always

    obvious or popular What are the solutions to the software crisis?
  126. 141.
  127. 142.
  128. 143.

    The Software Crisis Solutions are available — but not always

    obvious or popular Incremental Advances
  129. 144.

    Final Admonishment An epilogue called “Fifty Years of Wonder, Excitement,

    and Joy” Too many interests, too many exciting opportunities for learning, research, and thought. What a marvelous predicament! Not only is the end not in sight, but the pace is not slackening. We have many future joys. Frederick P. Brooks, Jr., The Mythical Man-Month, 1995