Empirically identifying the best genetic algorithm for covering array generation

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Yalan2011/

Transcript

1. EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING ARRAY GENERATION Liang Yalan1, Changhai Nie1, Jonathan M. Kauffman2, Gregory M. Kapfhammer2, Hareton Leung3 1Department of Computer Science and Technology, Nanjing University 2Department of Computer Science, Allegheny College 3Department of Computing, Hong Kong Polytechnic University 1 3rd International Symposium on Search Based Software Engineering Szeged, Hungary September 10-12, 2011
2. Combinatorial Testing 2 Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
3. Combinatorial Testing 3 Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
4. Combinatorial Testing 4 Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
5. Combinatorial Testing 5 Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
6. Combinatorial Testing 6 Modern software systems are highly configurable and involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
7. 2-way Covering Arrays 7 Suppose there are 4 parameters (pa1 , pa2 , pa3 , and pa4 ) in a system under test (SUT), each with 3 values (0, 1, 2) If we want to cover all 54 pair-wise interactions between every 2 parameters in the SUT, then only 9 test cases are needed What is the most efficient and effective method for generating covering arrays?
8. 2-way Covering Arrays 8 Suppose there are 4 parameters (pa1 , pa2 , pa3 , and pa4 ) in a system under test (SUT), each with 3 values (0, 1, 2) If we want to cover all 54 pair-wise interactions between every 2 parameters in the SUT, then only 9 test cases are needed What is the most efficient and effective method for generating covering arrays?
9. Covering Array Generation 9 • OFOT: One Factor One Time Method • AETG: Automatic Efficient Tests Generator Mathematical and Greedy Methods • Particle swarm optimization • Simulated annealing • Ant colony optimization Evolutionary Search Techniques
10. Covering Array Generation 10 • OFOT: One Factor One Time Method • AETG: Automatic Efficient Tests Generator Mathematical and Greedy Methods • Particle swarm optimization • Simulated annealing • Ant colony optimization Evolutionary Search Techniques
11. Covering Array Generation 11 • OFOT: One Factor One Time Method • AETG: Automatic Efficient Tests Generator Mathematical and Greedy Methods • Particle swarm optimization • Simulated annealing • Ant colony optimization Evolutionary Search Techniques This paper studies and improves genetic algorithms for covering array generation
12. Genetic Algorithm Phases 12 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Genetic algorithms solve complex problems
13. Genetic Algorithm Phases 13 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Genetic algorithms are hard to configure
14. Genetic Algorithm Parameters 14 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 System under test (SUT) description (e.g., 313)
15. Genetic Algorithm Parameters 15 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Number of uncovered pair-wise interactions
16. Genetic Algorithm Parameters 16 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Pc controls the probability of crossover
17. Genetic Algorithm Parameters 17 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pc is too high, then break good individuals
18. Genetic Algorithm Parameters 18 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pc is too low, then miss good solutions
19. Genetic Algorithm Parameters 19 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Pm controls the probability of mutation
20. Genetic Algorithm Parameters 20 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pm is too small, then cannot escape minima
21. Genetic Algorithm Parameters 21 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pm is too large, then degrade into random
22. Genetic Algorithm Parameters 22 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Standard GA: Select the superior individuals
23. Genetic Algorithm Parameters 23 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GA-: Select the inferior individuals
24. Genetic Algorithm Parameters 24 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GAr: Randomly select the individuals
25. Genetic Algorithm Parameters 25 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GA climb: Use elitism to keep best individual
26. Genetic Algorithm Parameters 26 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GA, GA-, GAr, GA climb, GA- climb, GAr climb
27. Experimental Design 27 Population Size Number of Generations Crossover Probability Mutation Probability
28. Experimental Design 28 • 100, 2100, • 4100, 6100 Population Size Number of Generations Crossover Probability Mutation Probability
29. Experimental Design 29 Population Size • 100, 600, 1100 Number of Generations Crossover Probability Mutation Probability
30. Experimental Design 30 Population Size Number of Generations • 0.2, 0.4, 0.6, 0.8, 1.0 Crossover Probability Mutation Probability
31. Experimental Design 31 Population Size Number of Generations Crossover Probability • 0.2, 0.4, 0.6, 0.8, 1.0 Mutation Probability
32. Experimental Design 32 Experiment Classes Pair-wise Base choice Hill climbing Is there an improved configuration of genetic algorithm for a particular pair-wise SUT?
33. Experimental Design 33 Experiment Classes Pair-wise Base choice Hill climbing Is there a common improved configuration for all pair-wise SUTs?
34. Experimental Design 34 Experiment Classes Pair-wise Base choice Hill climbing Produce a 2-way covering array with 34 configurations and input into the next phases
35. Experimental Design 35 Experiment Classes Pair-wise Base choice Hill climbing Create configurations by changing the value of one parameter and not modifying others
36. Experimental Design 36 Experiment Classes Pair-wise Base choice Hill climbing Iteratively refine the configurations in order to find the best one for each SUT
37. 37 For the chosen SUTs, there is no single genetic algorithm configuration that is the best Experimental Results
38. 38 Different values for the effectiveness of the genetic algorithm (e.g., CA size) Experimental Results
39. 39 Different values for the efficiency of the genetic algorithm (e.g., run time) Experimental Results
40. 40 The VGAs of all the improved configurations all use a climbing genetic algorithm Experimental Results
41. 41 GA- and GAr yield the best configuration for CA generation in 10 out of 15 SUTs Experimental Results
42. 42 For all SUTs, a lengthier evolutionary process improves CA generation Experimental Results
43. 43 In 13 out of 15 SUTS, creating fewer mutated individuals leads to better CAs Experimental Results
44. 44 There is no common best value of Pc or m for the chosen SUTs Experimental Results
45. 45 Please see the paper for additional insights concerning the experimental results Experimental Results
46. Conclusions 46 Initialize • Accept input parameters • Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Genetic algorithms for covering array generation
47. Conclusions 47 Systematic study on the impact of GA parameters Experiment Classes Pair-wise Base choice Hill climbing
48. Conclusions 48 Fundamental insights into the genetic algorithm Counter- intuitive selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
49. Conclusions 49 Fundamental insights into the genetic algorithm Counter- intuitive selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
50. Conclusions 50 Fundamental insights into the genetic algorithm Counter- intuitive selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
51. Conclusions 51 Fundamental insights into the genetic algorithm Counter- intuitive selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
52. Conclusions 52 Fundamental insights into the genetic algorithm Counter- intuitive selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
53. Future Work 53 Improved Understanding of Parameters Parallel Computing Discrete Parameters Continuous Parameters
54. Future Work 54 Improved Understanding of Parameters Parallel Computing Discrete Parameters Continuous Parameters
55. Future Work 55 Improved Understanding of Parameters Parallel Computing Discrete Parameters Continuous Parameters
56. Future Work 56 Improved Understanding of Parameters Parallel Computing Discrete Parameters Continuous Parameters
57. Future Work 57 Efficient and Effective Genetic Algorithms Parallel Computing Discrete Parameters Continuous Parameters
58. Future Work 58 Enhanced Covering Array Generators Parallel Computing Discrete Parameters Continuous Parameters
59. Future Work 59 Better Tested and Higher Quality Software Parallel Computing Discrete Parameters Continuous Parameters
60. Thank you for your attention! 60 EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING ARRAY GENERATION Liang Yalan, Changhai Nie, Jonathan M. Kauffman, Gregory M. Kapfhammer, Hareton Leung QUESTIONS OR COMMENTS? 3rd International Symposium on Search Based Software Engineering Szeged, Hungary September 10-12, 2011