Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Empirically identifying the best genetic algorithm for covering array generation

Empirically identifying the best genetic algorithm for covering array generation

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Yalan2011/

Gregory Kapfhammer

September 10, 2011
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Research

Transcript

  1. EMPIRICALLY IDENTIFYING THE BEST GENETIC ALGORITHM FOR COVERING ARRAY GENERATION

    Liang Yalan1, Changhai Nie1, Jonathan M. Kauffman2, Gregory M. Kapfhammer2, Hareton Leung3 1Department of Computer Science and Technology, Nanjing University 2Department of Computer Science, Allegheny College 3Department of Computing, Hong Kong Polytechnic University 1 3rd International Symposium on Search Based Software Engineering Szeged, Hungary September 10-12, 2011
  2. Combinatorial Testing 2 Modern software systems are highly configurable and

    involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
  3. Combinatorial Testing 3 Modern software systems are highly configurable and

    involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
  4. Combinatorial Testing 4 Modern software systems are highly configurable and

    involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
  5. Combinatorial Testing 5 Modern software systems are highly configurable and

    involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
  6. Combinatorial Testing 6 Modern software systems are highly configurable and

    involve many interacting parameters Combinatorial testing is a widely used and practical technique for detecting failures caused by the parameter interactions One of the key challenges in combinatorial testing is covering array generation, which is an noteworthy area of research From Kuhn et al., 70% of failures can be detected by 2-way interactions of the software system’s parameters Covering arrays can save testing time while still detecting many important software faults
  7. 2-way Covering Arrays 7 Suppose there are 4 parameters (pa1

    , pa2 , pa3 , and pa4 ) in a system under test (SUT), each with 3 values (0, 1, 2) If we want to cover all 54 pair-wise interactions between every 2 parameters in the SUT, then only 9 test cases are needed What is the most efficient and effective method for generating covering arrays?
  8. 2-way Covering Arrays 8 Suppose there are 4 parameters (pa1

    , pa2 , pa3 , and pa4 ) in a system under test (SUT), each with 3 values (0, 1, 2) If we want to cover all 54 pair-wise interactions between every 2 parameters in the SUT, then only 9 test cases are needed What is the most efficient and effective method for generating covering arrays?
  9. Covering Array Generation 9 • OFOT: One Factor One Time

    Method • AETG: Automatic Efficient Tests Generator Mathematical and Greedy Methods • Particle swarm optimization • Simulated annealing • Ant colony optimization Evolutionary Search Techniques
  10. Covering Array Generation 10 • OFOT: One Factor One Time

    Method • AETG: Automatic Efficient Tests Generator Mathematical and Greedy Methods • Particle swarm optimization • Simulated annealing • Ant colony optimization Evolutionary Search Techniques
  11. Covering Array Generation 11 • OFOT: One Factor One Time

    Method • AETG: Automatic Efficient Tests Generator Mathematical and Greedy Methods • Particle swarm optimization • Simulated annealing • Ant colony optimization Evolutionary Search Techniques This paper studies and improves genetic algorithms for covering array generation
  12. Genetic Algorithm Phases 12 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Genetic algorithms solve complex problems
  13. Genetic Algorithm Phases 13 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Genetic algorithms are hard to configure
  14. Genetic Algorithm Parameters 14 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 System under test (SUT) description (e.g., 313)
  15. Genetic Algorithm Parameters 15 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Number of uncovered pair-wise interactions
  16. Genetic Algorithm Parameters 16 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Pc controls the probability of crossover
  17. Genetic Algorithm Parameters 17 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pc is too high, then break good individuals
  18. Genetic Algorithm Parameters 18 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pc is too low, then miss good solutions
  19. Genetic Algorithm Parameters 19 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Pm controls the probability of mutation
  20. Genetic Algorithm Parameters 20 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pm is too small, then cannot escape minima
  21. Genetic Algorithm Parameters 21 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 If Pm is too large, then degrade into random
  22. Genetic Algorithm Parameters 22 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Standard GA: Select the superior individuals
  23. Genetic Algorithm Parameters 23 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GA-: Select the inferior individuals
  24. Genetic Algorithm Parameters 24 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GAr: Randomly select the individuals
  25. Genetic Algorithm Parameters 25 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GA climb: Use elitism to keep best individual
  26. Genetic Algorithm Parameters 26 Initialize • Accept input parameters •

    Generate population Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 GA, GA-, GAr, GA climb, GA- climb, GAr climb
  27. Experimental Design 28 • 100, 2100, • 4100, 6100 Population

    Size Number of Generations Crossover Probability Mutation Probability
  28. Experimental Design 29 Population Size • 100, 600, 1100 Number

    of Generations Crossover Probability Mutation Probability
  29. Experimental Design 30 Population Size Number of Generations • 0.2,

    0.4, 0.6, 0.8, 1.0 Crossover Probability Mutation Probability
  30. Experimental Design 32 Experiment Classes Pair-wise Base choice Hill climbing

    Is there an improved configuration of genetic algorithm for a particular pair-wise SUT?
  31. Experimental Design 33 Experiment Classes Pair-wise Base choice Hill climbing

    Is there a common improved configuration for all pair-wise SUTs?
  32. Experimental Design 34 Experiment Classes Pair-wise Base choice Hill climbing

    Produce a 2-way covering array with 34 configurations and input into the next phases
  33. Experimental Design 35 Experiment Classes Pair-wise Base choice Hill climbing

    Create configurations by changing the value of one parameter and not modifying others
  34. Experimental Design 36 Experiment Classes Pair-wise Base choice Hill climbing

    Iteratively refine the configurations in order to find the best one for each SUT
  35. 37 For the chosen SUTs, there is no single genetic

    algorithm configuration that is the best Experimental Results
  36. 40 The VGAs of all the improved configurations all use

    a climbing genetic algorithm Experimental Results
  37. 41 GA- and GAr yield the best configuration for CA

    generation in 10 out of 15 SUTs Experimental Results
  38. 43 In 13 out of 15 SUTS, creating fewer mutated

    individuals leads to better CAs Experimental Results
  39. 44 There is no common best value of Pc or

    m for the chosen SUTs Experimental Results
  40. 45 Please see the paper for additional insights concerning the

    experimental results Experimental Results
  41. Conclusions 46 Initialize • Accept input parameters • Generate population

    Fitness • Higher is better • Pair-wise interactions Crossover • Create new individuals • Combine two parents Mutation • Modify individuals • Increase diversity Selection • Choose individuals • Many variants 1 2 3 4 5 Genetic algorithms for covering array generation
  42. Conclusions 47 Systematic study on the impact of GA parameters

    Experiment Classes Pair-wise Base choice Hill climbing
  43. Conclusions 48 Fundamental insights into the genetic algorithm Counter- intuitive

    selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
  44. Conclusions 49 Fundamental insights into the genetic algorithm Counter- intuitive

    selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
  45. Conclusions 50 Fundamental insights into the genetic algorithm Counter- intuitive

    selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
  46. Conclusions 51 Fundamental insights into the genetic algorithm Counter- intuitive

    selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
  47. Conclusions 52 Fundamental insights into the genetic algorithm Counter- intuitive

    selection strategies Elitist selection methods Efficient and effective genetic algorithm for covering array generation
  48. Future Work 59 Better Tested and Higher Quality Software Parallel

    Computing Discrete Parameters Continuous Parameters
  49. Thank you for your attention! 60 EMPIRICALLY IDENTIFYING THE BEST

    GENETIC ALGORITHM FOR COVERING ARRAY GENERATION Liang Yalan, Changhai Nie, Jonathan M. Kauffman, Gregory M. Kapfhammer, Hareton Leung QUESTIONS OR COMMENTS? 3rd International Symposium on Search Based Software Engineering Szeged, Hungary September 10-12, 2011