Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using non-redundant mutation operators and test suite prioritization to achieve efficient and scalable mutation analysis

Using non-redundant mutation operators and test suite prioritization to achieve efficient and scalable mutation analysis

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Just2012b/

Gregory Kapfhammer

November 28, 2012
Tweet

More Decks by Gregory Kapfhammer

Other Decks in Research

Transcript

  1. Using Non-Redundant Mutation Operators and Test Suite Prioritization to Achieve

    Efficient and Scalable Mutation Analysis René Just1,2 & Gregory M. Kapfhammer3 & Franz Schweiggert2 1University of Washington, USA 2Ulm University, Germany 3Allegheny College, USA 23rd International Symposium on Software Reliability Engineering November 28, 2012
  2. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis

    assesses the quality of a test suite with artificial faults (mutants) Program Test suite
  3. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background Mutation analysis

    assesses the quality of a test suite with artificial faults (mutants) Program Test suite Generate mutants Mutants
  4. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background public int

    max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Mutation analysis assesses the quality of a test suite with artificial faults (mutants) Program Test suite Generate mutants Mutants Original
  5. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background public int

    max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Mutation analysis assesses the quality of a test suite with artificial faults (mutants) Program Test suite Generate mutants Mutants Original
  6. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background public int

    max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } public int max(int a, int b){ int max = a; if (b>=a){ max=b; } return max; } Mutation analysis assesses the quality of a test suite with artificial faults (mutants) Program Test suite Generate mutants Mutants Contains a small syntactic change Original Mutant
  7. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis Background public int

    max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } public int max(int a, int b){ int max = a; if (b>=a){ max=b; } return max; } Mutation analysis assesses the quality of a test suite with artificial faults (mutants) Program Test suite Generate mutants Mutants Execute mutants Mutation score Contains a small syntactic change Original Mutant
  8. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public

    int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original
  9. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public

    int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a)
  10. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public

    int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a) Many mutants can be generated for large programs
  11. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public

    int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a) Many mutants can be generated for large programs Large programs include comprehensive test suites
  12. Introduction Reduction Characteristics Prioritization Conclusion Mutation Analysis is Expensive public

    int max(int a, int b){ int max = a; if (b>a){ max=b; } return max; } Original if (b < a) if (b <= a) if (b >= a) if (b != a) if (b == a) Many mutants can be generated for large programs Large programs include comprehensive test suites Executing the entire test suite for all mutants in large programs is prohibitive!
  13. Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute

    fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants
  14. Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute

    fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants Test suite prioritization Test suite characteristics Reordering and splitting
  15. Introduction Reduction Characteristics Prioritization Conclusion Overview: Efficient Mutation Analysis Execute

    fewer mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants 27% Test suite prioritization Test suite characteristics Reordering and splitting 29% Empirical evaluation of 10 open-source projects with 560,000 mutants
  16. Introduction Reduction Characteristics Prioritization Conclusion Reduction of Mutants Execute fewer

    mutants fewer times Mutant reduction Generate fewer mutants Execute fewer mutants 27% Empirical evaluation of 10 open-source projects with 560,000 mutants
  17. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants

    Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b (replace binary operator) a + b → a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants
  18. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants

    Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b (replace binary operator) a + b → a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants Number of generated mutants reduced by 27%
  19. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants

    Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b (replace binary operator) a + b → a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants Number of generated mutants reduced by 27% More than 410,000 gen- erated mutants remaining
  20. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Generated Mutants

    Mutation operators may introduce redundancy: Redundant mutants are subsumed by other mutants a + b → a - b (replace binary operator) a + b → a + (-b) (insert unary operator) Use only non-redundant mutation operators Avoid the generation of such subsumed mutants Number of generated mutants reduced by 27% More than 410,000 gen- erated mutants remaining Executing all non-redundant mutants is still prohibitive!
  21. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants

    Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants
  22. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants

    Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50%
  23. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants

    Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50% Mutation analysis runtime still up to 13 hours
  24. Introduction Reduction Characteristics Prioritization Conclusion Reduce Number of Executed Mutants

    Exploit necessary conditions: Mutants not covered (reached) cannot be detected Determine covered mutants for the test suite Only execute the covered mutants Total reduction of executed mutants of more than 50% Mutation analysis runtime still up to 13 hours Further optimizations beyond the reduction of mutants are necessary!
  25. Introduction Reduction Characteristics Prioritization Conclusion Optimized Workflow for Mutation Analysis

    Execute fewer mutants fewer times Test suite prioritization Test suite characteristics Reordering and splitting 29% Empirical evaluation of 10 open-source projects with 560,000 mutants
  26. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Test case t1: 5 seconds Test case t2: 2 seconds Test case t3: 1 second
  27. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Test case t1: 5 seconds 1, 2, 3, 4, 5 Test case t2: 2 seconds 1, 3, 4, 5 Test case t3: 1 second 1, 2, 3
  28. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3
  29. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 :
  30. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5
  31. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5 3 4
  32. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5 3 4 3
  33. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5 3 4 3 t3 t2 t1 :
  34. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5 3 4 3 t3 t2 t1 : 1 2 3
  35. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5 3 4 3 t3 t2 t1 : 1 2 3 1 4 5
  36. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Reordering Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t2: 2 seconds 1, 3, 4, 5 1, 4 Test case t3: 1 second 1, 2, 3 3 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 t2 t3 : 1 2 3 4 5 3 4 3 t3 t2 t1 : 1 2 3 1 4 5 2 5
  37. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Splitting Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5
  38. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Splitting Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t 1 : 3 seconds 1, 2, 3, 4 1, 2 Test case t 1 : 2 seconds 2, 3, 4, 5 2, 5
  39. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Splitting Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t 1 : 3 seconds 1, 2, 3, 4 1, 2 Test case t 1 : 2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5
  40. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Splitting Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t 1 : 3 seconds 1, 2, 3, 4 1, 2 Test case t 1 : 2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5 t 1 t 1 :
  41. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Splitting Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t 1 : 3 seconds 1, 2, 3, 4 1, 2 Test case t 1 : 2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5 t 1 t 1 : 1 2 3 4
  42. Introduction Reduction Characteristics Prioritization Conclusion Motivating Example for Splitting Mutants:

    1, 2, 3, 4, 5 Covered: Detected: Test case t1: 5 seconds 1, 2, 3, 4, 5 1, 2, 5 Test case t 1 : 3 seconds 1, 2, 3, 4 1, 2 Test case t 1 : 2 seconds 2, 3, 4, 5 2, 5 Once a mutant is detected, it is not executed again! Executed mutants and total runtime: t1 : 1 2 3 4 5 t 1 t 1 : 1 2 3 4 3 4 5
  43. Introduction Reduction Characteristics Prioritization Conclusion Runtime Distribution of Tests within

    Test Suites q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q trove chart itext math time lang jdom jaxen io num4j 0 5 10 15 20 Test runtime in seconds
  44. Introduction Reduction Characteristics Prioritization Conclusion Runtime Distribution of Tests within

    Test Suites q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q trove chart itext math time lang jdom jaxen io num4j 0 5 10 15 20 Test runtime in seconds Most tests have short runtime
  45. Introduction Reduction Characteristics Prioritization Conclusion Runtime Distribution of Tests within

    Test Suites q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q trove chart itext math time lang jdom jaxen io num4j 0 5 10 15 20 Test runtime in seconds Most tests have short runtime A few long- running outliers
  46. Introduction Reduction Characteristics Prioritization Conclusion Runtime Distribution of Tests within

    Test Suites q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q trove chart itext math time lang jdom jaxen io num4j 0 5 10 15 20 Test runtime in seconds Most tests have short runtime A few long- running outliers A few tests constitute most of the total runtime: Reduce number of executions for these tests
  47. Introduction Reduction Characteristics Prioritization Conclusion Mutation Coverage Overlap Overlap measures

    the similarity of a test case with its enclosing test suite Pair-wise comparison of test cases is infeasible
  48. Introduction Reduction Characteristics Prioritization Conclusion Mutation Coverage Overlap Overlap measures

    the similarity of a test case with its enclosing test suite Pair-wise comparison of test cases is infeasible Definition: Overlap O(ti , T), ti ∈ T O(ti , T)        1, |Cov(ti )| = 0 |Cov(ti)∩Cov(T\ti)| |Cov(ti)| , |Cov(ti )| > 0
  49. Introduction Reduction Characteristics Prioritization Conclusion Mutation Coverage Overlap Overlap measures

    the similarity of a test case with its enclosing test suite Pair-wise comparison of test cases is infeasible Definition: Overlap O(ti , T), ti ∈ T O(ti , T)        1, |Cov(ti )| = 0 |Cov(ti)∩Cov(T\ti)| |Cov(ti)| , |Cov(ti )| > 0 Most of the test cases exhibit high overlap: Does test runtime correlate with overlap?
  50. Introduction Reduction Characteristics Prioritization Conclusion Correlation of Test Runtime and

    Mutation Coverage 0 1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 0 10 20 30 40 50 60 Index of test in original test suite 0 50 100 150 200 250 Runtime of test in milliseconds
  51. Introduction Reduction Characteristics Prioritization Conclusion Correlation of Test Runtime and

    Mutation Coverage 0 1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 0 10 20 30 40 50 60 Index of test in original test suite 0 50 100 150 200 250 Runtime of test in milliseconds Test case with longest runtime
  52. Introduction Reduction Characteristics Prioritization Conclusion Correlation of Test Runtime and

    Mutation Coverage 0 1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 0 10 20 30 40 50 60 Index of test in original test suite 0 50 100 150 200 250 Runtime of test in milliseconds Test case with longest runtime Overlapping test cases
  53. Introduction Reduction Characteristics Prioritization Conclusion Correlation of Test Runtime and

    Mutation Coverage 0 1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 0 10 20 30 40 50 60 Index of test in original test suite 0 50 100 150 200 250 Runtime of test in milliseconds Test case with longest runtime Overlapping test cases Reorder to exploit mutation coverage overlap
  54. Introduction Reduction Characteristics Prioritization Conclusion Correlation of Test Runtime and

    Mutation Coverage 0 1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 0 10 20 30 40 50 60 Index of test in original test suite 0 50 100 150 200 250 Runtime of test in milliseconds Test case with longest runtime Overlapping test cases Reorder to exploit mutation coverage overlap Large mutation coverage
  55. Introduction Reduction Characteristics Prioritization Conclusion Correlation of Test Runtime and

    Mutation Coverage 0 1000 2000 3000 4000 5000 6000 Index of mutant in set of generated mutants 0 10 20 30 40 50 60 Index of test in original test suite 0 50 100 150 200 250 Runtime of test in milliseconds Test case with longest runtime Overlapping test cases Reorder to exploit mutation coverage overlap Large mutation coverage Split test cases to increase coverage precision
  56. Introduction Reduction Characteristics Prioritization Conclusion Mutation Coverage of Test suites

    test suite class#1 method#1 ... ... ... class#m ... method#n Higher precision
  57. Introduction Reduction Characteristics Prioritization Conclusion Mutation Coverage of Test suites

    test suite class#1 method#1 ... ... ... class#m ... method#n Lower overhead Higher precision
  58. Introduction Reduction Characteristics Prioritization Conclusion Mutation Coverage of Test suites

    test suite class#1 method#1 ... ... ... class#m ... method#n Lower overhead Higher precision Only split long-running test classes
  59. Introduction Reduction Characteristics Prioritization Conclusion Splitting Test Classes Two splitting

    strategies Split entire long- running test class High overhead and coverage precision
  60. Introduction Reduction Characteristics Prioritization Conclusion Splitting Test Classes Two splitting

    strategies Split entire long- running test class High overhead and coverage precision Extract only long- running test methods Lower overhead and coverage precision
  61. Introduction Reduction Characteristics Prioritization Conclusion Splitting Test Classes Two splitting

    strategies Split entire long- running test class High overhead and coverage precision Extract only long- running test methods Lower overhead and coverage precision Trade-off between overhead and precision: Splitting based on threshold for test runtime
  62. Introduction Reduction Characteristics Prioritization Conclusion Optimized workflow Original program Generate

    mutants Set of non- redundant mutants Execute test suite Original test suite
  63. Introduction Reduction Characteristics Prioritization Conclusion Optimized workflow Original program Generate

    mutants Set of non- redundant mutants Execute test suite Original test suite Runtime of test cases Mutation coverage
  64. Introduction Reduction Characteristics Prioritization Conclusion Optimized workflow Original program Generate

    mutants Set of non- redundant mutants Execute test suite Original test suite Runtime of test cases Mutation coverage Order/split test cases Prioritized test suite
  65. Introduction Reduction Characteristics Prioritization Conclusion Optimized workflow Original program Generate

    mutants Set of non- redundant mutants Execute test suite Original test suite Runtime of test cases Mutation coverage Order/split test cases Prioritized test suite Mutation analysis
  66. Introduction Reduction Characteristics Prioritization Conclusion Example with Original Test Suite

    0 0.2 0.4 0.6 0.8 1 Mutation score Original test suite 0 7 14 21 0 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes
  67. Introduction Reduction Characteristics Prioritization Conclusion Example with Original Test Suite

    0 0.2 0.4 0.6 0.8 1 Mutation score Original test suite 0 7 14 21 0 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants
  68. Introduction Reduction Characteristics Prioritization Conclusion Example with Original Test Suite

    0 0.2 0.4 0.6 0.8 1 Mutation score Original test suite 0 7 14 21 0 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants
  69. Introduction Reduction Characteristics Prioritization Conclusion Example with Original Test Suite

    0 0.2 0.4 0.6 0.8 1 Mutation score Original test suite 0 7 14 21 0 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants Reorder
  70. Introduction Reduction Characteristics Prioritization Conclusion Example with Original Test Suite

    0 0.2 0.4 0.6 0.8 1 Mutation score Original test suite 0 7 14 21 0 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes Total runtime of test executing all covered, yet not killed, mutants Reorder Split
  71. Introduction Reduction Characteristics Prioritization Conclusion Example with Prioritized Test Suite

    0 0.2 0.4 0.6 0.8 1 Mutation score Prioritized test suite 0 7 14 21 0 100 200 300 400 500 600 700 800 Test-runtime in seconds Total runtime in minutes
  72. Introduction Reduction Characteristics Prioritization Conclusion Empirical Results Reordering: Reordering decreases

    the runtime by 20% Splitting strategies: Extracting long test methods reduces the runtime by 29% Splitting entire test classes increases the runtime by 27% Splitting may increase runtime if: Test suite has a very low mutation detection rate Test methods exhibit huge mutation coverage overlap
  73. Introduction Reduction Characteristics Prioritization Conclusion Empirical Results Reordering: Reordering decreases

    the runtime by 20% Splitting strategies: Extracting long test methods reduces the runtime by 29% Splitting entire test classes increases the runtime by 27% Splitting may increase runtime if: Test suite has a very low mutation detection rate Test methods exhibit huge mutation coverage overlap Prioritizing test suites improves the efficiency of mutation analysis by 29% on average! 29%
  74. Introduction Reduction Characteristics Prioritization Conclusion Related Work Reduction of generated

    mutants: Sufficient mutation operators Offutt et al., TOSEM’96 Namin et al., ICSE’08 Non-redundant mutation operators Kaminski et al., AST’11 Just et al., Mutation’12 Mutation-based test suite optimization: Test case prioritization Elbaum et al. TSE’02 Do and Rothermel, TSE’06
  75. Introduction Reduction Characteristics Prioritization Conclusion Related Work Reduction of generated

    mutants: Sufficient mutation operators Offutt et al., TOSEM’96 Namin et al., ICSE’08 Non-redundant mutation operators Kaminski et al., AST’11 Just et al., Mutation’12 Mutation-based test suite optimization: Test case prioritization Elbaum et al. TSE’02 Do and Rothermel, TSE’06 Still contain redundancies
  76. Introduction Reduction Characteristics Prioritization Conclusion Related Work Reduction of generated

    mutants: Sufficient mutation operators Offutt et al., TOSEM’96 Namin et al., ICSE’08 Non-redundant mutation operators Kaminski et al., AST’11 Just et al., Mutation’12 Mutation-based test suite optimization: Test case prioritization Elbaum et al. TSE’02 Do and Rothermel, TSE’06 Still contain redundancies Used in empirical study
  77. Introduction Reduction Characteristics Prioritization Conclusion Related Work Reduction of generated

    mutants: Sufficient mutation operators Offutt et al., TOSEM’96 Namin et al., ICSE’08 Non-redundant mutation operators Kaminski et al., AST’11 Just et al., Mutation’12 Mutation-based test suite optimization: Test case prioritization Elbaum et al. TSE’02 Do and Rothermel, TSE’06 Still contain redundancies Used in empirical study Do not address efficiency
  78. Introduction Reduction Characteristics Prioritization Conclusion Conclusions Reduction of mutants: Non-redundant

    operators reduce number of mutants by 27% Test suite characteristics: Most of the tests exhibit mutation coverage overlap Notable difference in runtime of tests Optimized workflow: Exploits mutation coverage overlap and runtime differences Further reduces total runtime of mutation analysis by 29%
  79. Introduction Reduction Characteristics Prioritization Conclusion Conclusions Reduction of mutants: Non-redundant

    operators reduce number of mutants by 27% Test suite characteristics: Most of the tests exhibit mutation coverage overlap Notable difference in runtime of tests Optimized workflow: Exploits mutation coverage overlap and runtime differences Further reduces total runtime of mutation analysis by 29% Non-redundant operators and optimized workflow implemented in the MAJOR mutation system