Parameter tuning for search-based test-data generation revisited

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Kotelyanskii2014a/

Transcript

1. Parameter Tuning for Search-Based Test-Data Generation Revisited Support for Previous Results Anton Kotelyanskii Gregory M. Kapfhammer shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
2. Software Testing
3. Software Testing Test Suites
4. Software Testing Test Suites Automatic Generation
5. Software Testing Test Suites Automatic Generation Confronting Challenges
6. Software Testing Test Suites Automatic Generation Confronting Challenges Evaluation Strategies
7. Empirical Studies
8. Empirical Studies Challenges
9. Empirical Studies Challenges Importance
10. Empirical Studies Challenges Importance Replication
11. Empirical Studies Challenges Importance Replication Rarity
13. EvoSuite shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie
14. EvoSuite Amazing test suite generator shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie
15. EvoSuite Amazing test suite generator Uses a genetic algorithm shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie
16. EvoSuite Amazing test suite generator Uses a genetic algorithm Input: A Java class shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie
17. EvoSuite Amazing test suite generator Uses a genetic algorithm Input: A Java class Output: A JUnit test suite shared by creative commons licensed ( BY-SA ) ickr photo mcclanahoochie
18. EvoSuite Amazing test suite generator Uses a genetic algorithm Input: A Java class Output: A JUnit test suite shared by http://www.evosuite.org/ creative commons licensed ( BY-SA ) ickr photo mcclanahoochie
19. Parameter Tuning
20. Parameter Tuning RSM: Response surface methodology
21. Parameter Tuning RSM: Response surface methodology SPOT: Sequential parameter optimization toolbox
22. Parameter Tuning RSM: Response surface methodology SPOT: Sequential parameter optimization toolbox Successfully applied to many diverse problems!
23. Defaults or Tuned Values?
25. Experiment Design shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
26. Experiment Design Eight EvoSuite parameters shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
27. Experiment Design Eight EvoSuite parameters Ten projects from SF100 shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
28. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
29. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
30. Experiment Design Eight EvoSuite parameters Ten projects from SF100 475 Java classes for subjects 100 trials after parameter tuning Aiming to improve statement coverage shared by creative commons licensed ( BY-NC ) ickr photo Michael Kappel
31. Parameters Parameter Name Minimum Maximum Population Size 5 99 Chromosome Length 5 99 Rank Bias 1.01 1.99 Number of Mutations 1 10 Max Initial Test Count 1 10 Crossover Rate 0.01 0.99 Constant Pool Use Probability 0.01 0.99 Test Insertion Probability 0.01 0.99
32. Experiments
33. Experiments 184 days of computation time estimated
34. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks
35. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi ed 139 "easy" and 21 "hard" classes
36. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi ed 139 "easy" and 21 "hard" classes Mann-Whitney U-test and
37. Experiments 184 days of computation time estimated Cluster of 70 computers running for weeks Identi ed 139 "easy" and 21 "hard" classes Mann-Whitney U-test and Vargha-Delaney e ect size
38. Results Category E ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314
39. Results Using lower-is-better inverse statement coverage Category E ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314
40. Results Using lower-is-better inverse statement coverage E ect size greater than 0.5 means that tuning is worse Category E ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314
41. Results Using lower-is-better inverse statement coverage E ect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis Category E ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314
42. Results Using lower-is-better inverse statement coverage E ect size greater than 0.5 means that tuning is worse Testing shows we do not always reject the null hypothesis Additional empirical results in the QSIC 2014 paper! Category E ect Size p-value Results Across Trials and Classes 0.5029 0.1045 No "Easy" and "Hard" Classes 0.5048 0.0314
44. Discussion shared by creative commons licensed ( BY ) photo Startup Stock Photos
45. Discussion Tuning improved scores for 11 classes shared by creative commons licensed ( BY ) photo Startup Stock Photos
46. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults shared by creative commons licensed ( BY ) photo Startup Stock Photos
47. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft oor" may exist for parameter tuning shared by creative commons licensed ( BY ) photo Startup Stock Photos
48. Discussion Tuning improved scores for 11 classes Otherwise, same as or worse than defaults A "soft oor" may exist for parameter tuning Additional details in the QSIC 2014 paper! shared by creative commons licensed ( BY ) photo Startup Stock Photos
49. Practical Implications
50. Practical Implications Fundamental Challenges
51. Practical Implications Fundamental Challenges Tremendous Con dence
52. Practical Implications Fundamental Challenges Tremendous Con dence Great Opportunities
54. Important Contributions shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
55. Important Contributions Comprehensive Experiments shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
56. Important Contributions Comprehensive Experiments Conclusive Con rmation shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13
57. Important Contributions Comprehensive Experiments Conclusive Con rmation For EvoSuite, Defaults = Tuned shared by creative commons licensed ( BY-NC-ND ) ickr photo sunface13