Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Empirically studying the role of selection operators during search-based test suite prioritization

Empirically studying the role of selection operators during search-based test suite prioritization

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Conrad2010a/

Gregory Kapfhammer

July 02, 2010
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Research

Transcript

  1. Introduction Regression Testing Empirical Study Conclusion Empirically Studying the Role

    of Selection Operators During Search-Based Test Suite Prioritization Alexander P. Conrad‡, Robert S. Roos†, and Gregory M. Kapfhammer† ‡Department of Computer Science University of Pittsburgh †Department of Computer Science Allegheny College Genetic and Evolutionary Computation Conference Search-Based Software Engineering Track July 2010 1 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  2. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Empirical Results Genetic algorithm-based test prioritizer that uses many mutation, crossover, selection, and transformation operators 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  3. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Empirical Results Genetic algorithm-based test prioritizer that uses many mutation, crossover, selection, and transformation operators 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  4. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Genetic algorithm-based test prioritizer that uses many mutation, crossover, selection, and transformation operators 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  5. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Genetic algorithm-based test prioritizer that uses many mutation, crossover, selection, and transformation operators 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  6. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Automatically constructed tree models highlight the unique role that the selection operator plays during prioritization 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  7. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Genetic algorithm is superior to random search and hill climbing and often suitable for many testing environments 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  8. Introduction Regression Testing Empirical Study Conclusion Important Contributions Search-Based Prioritizers

    Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Complete genetic algorithm-based prioritization framework is available from http://gelations.googlecode.com/ 2 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  9. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Reorder the test suite in order to improve effectiveness 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  10. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization Reorder the test suite in order to improve effectiveness 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  11. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Reorder the test suite in order to improve effectiveness 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  12. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Execution Reorder the test suite in order to improve effectiveness 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  13. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Execution P Reorder the test suite in order to improve effectiveness 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  14. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Execution P S(T) R(T) Reorder the test suite in order to improve effectiveness 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  15. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Execution P S(T) R(T) R(T) T Prioritization Version Specific Regression Testing T′ Execution P S(T) R(T) Re-prioritize each time the suite or program changes 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  16. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Execution P S(T) R(T) R(T) T Prioritization T′ Execution P S(T) R(T) R(T) T Prioritization General Regression Testing T′ Execution P S(T) R(T) Use the same suite for multiple rounds of test execution 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  17. Introduction Regression Testing Empirical Study Conclusion Process of Regression Testing

    R(T) T Prioritization T′ Execution P S(T) R(T) R(T) T Prioritization T′ Execution P S(T) R(T) R(T) T Prioritization General Regression Testing T′ Execution P S(T) R(T) Do et al. “the worst thing that JUnit users can do is not practice some form of prioritization” (ISSRE 2004) 3 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  18. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite

    Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [0 , 1] 4 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  19. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite

    Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 2, 3 CE = 0.3789 1,2,3 Test Orderings Original ordering exhibits poor effectiveness score - CE = 0.3789 4 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  20. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite

    Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 1, 3, 2 CE = 0.5053 1,2,3 Test Orderings 1,2,3 1,3,2 Different ordering improves the effectiveness score - CE = 0.5053 4 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  21. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite

    Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 2, 3, 1 CE = 0.4316 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 Some orderings have less improved scores - CE = 0.4316 4 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  22. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite

    Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Best ordering shows a higher effectiveness scores - CE = 0.5789 4 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  23. Introduction Regression Testing Empirical Study Conclusion Importance of Test Suite

    Prioritization 0 5 10 15 0 1 2 3 4 5 Testing Time (l) Covered Test Requirements (C(T, l)) 3, 1, 2 CE = 0.5789 1,2,3 Test Orderings 1,2,3 1,3,2 1,2,3 1,3,2 2,3,1 1,2,3 1,3,2 2,3,1 3,1,2 Greedy methods often produce high effectiveness orderings 4 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  24. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 Possible configuration of the coverage report 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  25. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  26. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  27. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 Possible configuration of the coverage report 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  28. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 Execution time of the test cases may mislead greedy 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  29. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 Execution time of the test cases may mislead greedy 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  30. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 Original ordering has low effectiveness score 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  31. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 CE(T) = 0 . 54 Original ordering has low effectiveness score 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  32. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 Greedy method constructs suite with marginal improvement 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  33. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 CE(T′) = 0 . 55 Greedy method constructs suite with marginal improvement 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  34. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 CE(T′) = 0 . 55 Greedy can exhibit high run-times (Jiang et al. ASE 2009) 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  35. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 Genetic algorithm finds a higher quality ordering 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  36. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T′) = 0 . 63 Genetic algorithm finds a higher quality ordering 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  37. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T′) = 0 . 63 Genetic algorithm is amenable to parallelization 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  38. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T′) = 0 . 63 Genetic algorithm supports “human in the loop” 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  39. Introduction Regression Testing Empirical Study Conclusion Limitations of Greedy Methods

    T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 T1 T3 T2 T4 R1 R2 R3 time(T1 ) = 1 time(T2 ) = 1 time(T3 ) = 1 time(T4 ) = 2 . 45 T1 T2 T3 T4 T3 T1 T2 T4 T1 T2 T4 T3 CE(T′) = 0 . 63 Genetic algorithm constructs diverse test orderings 5 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  40. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Randomly create suites by repeatedly shuffling T1, . . . , Tn 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  41. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Execute the phases until the genetic algorithm stagnates 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  42. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Use coverage effectiveness in this study - others possible 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  43. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators CE Use coverage effectiveness in this study - others possible 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  44. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators CE APFD APRC Use coverage effectiveness in this study - others possible 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  45. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Choose orderings to become parents of next generation 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  46. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Choose orderings to become parents of next generation 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  47. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators ROU TOU TRU Choose orderings to become parents of next generation 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  48. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Seven possible operators combine parents to produce children 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  49. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Seven possible operators combine parents to produce children 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  50. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators MPX PMX POS Seven possible operators combine parents to produce children 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  51. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Six possible operators make random changes to orderings 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  52. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators Six possible operators make random changes to orderings 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  53. Introduction Regression Testing Empirical Study Conclusion Test Prioritization with Genetic

    Algorithms Fitness Initialize Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Fitness Execute Selection Crossover Mutation Operators ISM IVM SIM Six possible operators make random changes to orderings 6 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  54. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Explored a wide variety of genetic algorithm configurations 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  55. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy How frequently do we modify individual test orderings? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  56. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values How frequently do we modify individual test orderings? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  57. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values 0 . 10 0 . 33 0 . 67 How frequently do we modify individual test orderings? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  58. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy How many children should be in the next population? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  59. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values How many children should be in the next population? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  60. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values 0 . 50 0 . 75 1 . 00 How many children should be in the next population? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  61. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy How many test suites should exist in the population? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  62. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values How many test suites should exist in the population? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  63. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values 75 150 225 How many test suites should exist in the population? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  64. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy How many generations without fitness improvement? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  65. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values How many generations without fitness improvement? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  66. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values 20 30 40 How many generations without fitness improvement? 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  67. Introduction Regression Testing Empirical Study Conclusion Configuration of the Genetic

    Algorithm Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Mutation Rate Possible Configurations Child Density Population Stagnancy Parameter Values 20 30 40 See the paper for further operator and configuration details 7 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  68. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 Tree Models: Recursive partitioning creates hierarchical view of data 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  69. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 Tree Models: Recursive partitioning creates hierarchical view of data 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  70. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 Tree Models: Recursive partitioning creates hierarchical view of data 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  71. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 pop_size < 187.5 Tree Models: Recursive partitioning creates hierarchical view of data 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  72. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 pop_size < 187.5 sel_method child_density pop_size Explanatory Variable: Configuration of the genetic algorithm 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  73. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 pop_size < 187.5 sel_method child_density pop_size Categorical sel_method Non-parametric techniques that handles different variable types 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  74. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 pop_size < 187.5 sel_method child_density pop_size Categorical Categorical Numerical child_density pop_size Non-parametric techniques that handles different variable types 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  75. Introduction Regression Testing Empirical Study Conclusion Analysis Techniques: Tree Models

    sel_method: TOU3, TOU4, TOU5 child_density < 0.875 0.9674 0.9573 pop_size < 187.5 sel_method child_density pop_size Categorical Categorical Numerical Categorical Numerical 0.9674 0.9573 Response Variable: Fitness of the final test ordering (CE score) 8 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  76. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design

    Name |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Several applications and test suites - coverage reports derived from call-tree based adequacy (McMaster and Memon ICSM 2005) 9 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  77. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design

    Name |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Several applications and test suites - coverage reports derived from call-tree based adequacy (McMaster and Memon ICSM 2005) 9 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  78. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design

    Name |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Several applications and test suites - coverage reports derived from call-tree based adequacy (McMaster and Memon ICSM 2005) 9 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  79. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design

    Name |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Use additional case study applications and adequacy criteria as future work in order to control threats to external validity 9 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  80. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design

    Name |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Use random and hill climbing (first and steepest ascent) as control methods for comparison to the genetic algorithm prioritizer 9 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  81. Introduction Regression Testing Empirical Study Conclusion Experimental Goals and Design

    Name |T| |R(T)| CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 See the paper for more details about the design of the empirical study (e.g., configuration of random and hill climbing prioritizers) 9 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  82. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  83. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  84. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  85. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  86. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method The sel_method variable is always the most important parameter 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  87. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method Importance of sel_method holds for all case study applications 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  88. Introduction Regression Testing Empirical Study Conclusion Results: Selection Method Importance

    sel_method: TOU3, TOU4, TOU5 RM child_density < 0.875 0.9674 sel_method sel_method: TOU3, TOU4, TOU5 GB mutation_rate < 0.17 cross_operator: OX1, VR sel_method How does the selection method impact the genetic algorithm? 10 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  89. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Except for the smallest application (LF), the CE scores of the evolved orderings are better than the initial and reverse test suites 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  90. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Study a type of operator as it increases in intensity, or the change in average fitness due to selection (Blickle & Thiele, Evol Comp 1997) 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  91. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Increasing selection intensity improves the CE scores of test orderings, even though it does not cause more rapid convergence 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  92. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Increasing selection intensity improves the CE scores of test orderings, even though it does not cause more rapid convergence 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  93. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Increasing selection intensity improves the CE scores of test orderings, even though it does not cause more rapid convergence 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  94. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 Low intensity selection causes search to meander around low quality test suite prioritizations, making fitness stagnate and the GA terminate 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  95. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 High intensity selection focuses on a local optimum of high quality instead of hunting for hard-to-find global optimum 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  96. Introduction Regression Testing Empirical Study Conclusion Results: Selection Intensity Name

    ROUE ROUL TRU60 TRU40 TOU2 TOU5 DS 0.9742 0.9837 0.9893 0.9915 0.9514 0.9706 GB 0.9500 0.9572 0.9668 0.9700 0.9062 0.9402 JD 0.9247 0.9328 0.9431 0.9451 0.8993 0.9192 LF 0.9903 0.9903 0.9903 0.9903 0.9903 0.9903 RM 0.9665 0.9670 0.9681 0.9682 0.9328 0.9475 RP 0.9774 0.9824 0.9868 0.9879 0.9570 0.9705 SK 0.9859 0.9878 0.9911 0.9915 0.9667 0.9763 TM 0.9585 0.9605 0.9662 0.9672 0.9503 0.9579 Avg. 0.9659 0.9702 0.9752 0.9765 0.9443 0.9591 One Explanation: the fitness landscape for coverage effectiveness contains many local optima that are good test orderings 11 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  97. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Random

    0.0 0.2 0.4 0.6 0.8 1.0 50 500 5000 GB Number of Random Prioritizations Coverage Effectiveness (CE) GB: Random orderings have average CE scores around 0.6 12 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  98. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Random

    0.2 0.4 0.6 0.8 1.0 50 500 5000 SK Number of Random Prioritizations Coverage Effectiveness (CE) SK: Random orderings have average CE scores around 0.7 12 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  99. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Random

    0.2 0.4 0.6 0.8 1.0 50 500 5000 SK Number of Random Prioritizations Coverage Effectiveness (CE) Conclusion: Random is not as effective as the genetic algorithm 12 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  100. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Hill

    Climbing 0.2 0.4 0.6 0.8 1.0 DS GB JD LF RM RP SK TM HC−FA−FN Case Study Application Coverage Effectiveness (CE) First Ascent: Across all applications, average CE score below 0.8 13 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  101. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Hill

    Climbing 0.2 0.4 0.6 0.8 1.0 DS GB JD LF RM RP SK TM HC−SA−FN Case Study Application Coverage Effectiveness (CE) Steepest Ascent: Larger neighborhoods slightly improve the CE scores 13 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  102. Introduction Regression Testing Empirical Study Conclusion Results: Comparison to Hill

    Climbing 0.2 0.4 0.6 0.8 1.0 DS GB JD LF RM RP SK TM HC−SA−FN Case Study Application Coverage Effectiveness (CE) Conclusion: Hill climber is not as effective as the genetic algorithm 13 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  103. Introduction Regression Testing Empirical Study Conclusion GELATIONS Framework for Prioritization

    http://gelations.googlecode.com/ provides our framework 14 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  104. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Empirical Results Developed a genetic algorithm-based test prioritizer supporting many evolutionary operators and configurations 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  105. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Empirical Results Developed a genetic algorithm-based test prioritizer supporting many evolutionary operators and configurations 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  106. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Developed a genetic algorithm-based test prioritizer supporting many evolutionary operators and configurations 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  107. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Developed a genetic algorithm-based test prioritizer supporting many evolutionary operators and configurations 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  108. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Used automatically constructed tree models to highlight the role that the selection operator plays during prioritization 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  109. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Genetic algorithm is superior to random search and hill climbing and thus suitable for certain testing environments 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  110. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Future Work: After extending the genetic algorithm, use fitness landscape analysis to understand impact of adequacy criteria 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  111. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Future Work: Use additional applications (e.g., SIR, XML, DBA) and test adequacy criteria (e.g., data and control flow) 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  112. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Future Work: Comprehensive empirical study of all major search-based and greedy algorithms for test suite prioritization 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  113. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Get involved and stay in touch! 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  114. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Get involved and stay in touch! Read paper, 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  115. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Get involved and stay in touch! Read paper, download and use GELATIONS, 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  116. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Get involved and stay in touch! Read paper, download and use GELATIONS, replicate experiments, 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  117. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results Get involved and stay in touch! Read paper, download and use GELATIONS, replicate experiments, and share applications 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization
  118. Introduction Regression Testing Empirical Study Conclusion Conclusions and Future Work

    Search-Based Prioritizers Execution Time (in ms) Coverage Effectiveness 0.92 0.94 0.96 0.98 0 1000 2000 3000 4000 5000 GB CX Control (RS) MPX OX1 OX2 PMX POS VR Empirical Results http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ 15 / 15 Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization