Do redundant mutants affect the effectiveness and efficiency of mutation analysis?

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Just2012a/

Transcript

1. Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? René Just1 & Gregory M. Kapfhammer2 & Franz Schweiggert1 1Ulm University, Germany 2Allegheny College, USA 7th International Workshop on Mutation Analysis Montreal, Canada April 17, 2012
2. Introduction Operator Empirical Study Conclusion Overview of the Presentation Challenges in Mutation Analysis Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
3. Introduction Operator Empirical Study Conclusion Overview of the Presentation Challenges in Mutation Analysis Equivalent Mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
4. Introduction Operator Empirical Study Conclusion Overview of the Presentation Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
5. Introduction Operator Empirical Study Conclusion Overview of the Presentation Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Redundant Mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
6. Introduction Operator Empirical Study Conclusion Overview of the Presentation Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Redundant Mutants Subsumed by other mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
7. Introduction Operator Empirical Study Conclusion Overview of the Presentation Challenges in Mutation Analysis Equivalent Mutants Cannot be killed Redundant Mutants Subsumed by other mutants Harmful to efficiency and effectiveness Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
8. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
9. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
10. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
11. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
12. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study on the impact of redundant mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
13. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study on the impact of redundant mutants Efficiency Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
14. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study on the impact of redundant mutants Efficiency Number of mutants and runtime Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
15. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study on the impact of redundant mutants Efficiency Number of mutants and runtime Effectiveness Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
16. Introduction Operator Empirical Study Conclusion Overview of the Presentation Redundant mutants Operator for Condi- tional Expressions without redundancy Conditional Operator Replacement (COR) Unary Operator Insertion (UOI) Empirical Study on the impact of redundant mutants Efficiency Number of mutants and runtime Effectiveness Accuracy of mutation score Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
17. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
18. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b && || Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
19. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b && || Conditional Operator Replacement (COR) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
20. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
21. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b lhs rhs true false Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
22. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b lhs rhs true false Unary Operator Insertion (UOI) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
23. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions a <op> b && || Conditional Operator Replacement (COR) a && b a || b a == b a != b lhs rhs true false Unary Operator Insertion (UOI) !a <op> b !(a <op> b) a <op> !b Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
24. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Literals Expression a b a && b 0 0 0 0 1 0 1 0 0 1 1 1 Literals Expression a b a b 0 0 0 0 1 1 1 0 1 1 1 1 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
25. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Literals Expression a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 Literals Expression a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 0 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
26. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 0 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
27. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 0 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 4 Mutants are sufficient Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
28. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 0 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 4 Mutants are sufficient UOI Operator completely subsumed Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
29. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a && b false lhs rhs a == b a b a != b true !a && b !(a && b) a && !b 0 0 0 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 Literals Expression Sufficient mutations Subsumed mutations Subsumed operator UOI a b a b a != b rhs lhs true a && b a == b false !a b !(a b) a !b 0 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 4 Mutants are sufficient UOI Operator completely subsumed A reduction of exactly 60% ? Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
30. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
31. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
32. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
33. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
34. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
35. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
36. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
37. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())!=null) { v.bar(x); } ... } Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
38. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } Properly handled by MAJOR Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
39. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())==null) { return; } v.bar(x); } Properly handled by MAJOR Properly handled by MAJOR Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
40. Introduction Operator Empirical Study Conclusion Mutating Conditional Expressions Two common patterns for short-circuit operators public void foo(int x){ Var v; if(flag&&(v=getVar())!=null) { v.bar(x); } ... } public void foo(int x){ Var v; if(flag||(v=getVar())==null) { return; } v.bar(x); } Properly handled by MAJOR Properly handled by MAJOR A reduction of up to 60% Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
41. Introduction Operator Empirical Study Conclusion COR and ROR Mutation Operators Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
42. Introduction Operator Empirical Study Conclusion COR and ROR Mutation Operators Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
43. Introduction Operator Empirical Study Conclusion COR and ROR Mutation Operators Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Kaminski et al. AST’11 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
44. Introduction Operator Empirical Study Conclusion COR and ROR Mutation Operators Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Kaminski et al. AST’11 Reduction of 50% Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
45. Introduction Operator Empirical Study Conclusion COR and ROR Mutation Operators Subsumption hierarchy Conditional Operator Replacement (COR) Reduction of up to 60% Relational Operator Replacement (ROR) Kaminski et al. AST’11 Reduction of 50% How prevalent are COR and ROR mutants? Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
46. Introduction Operator Empirical Study Conclusion Investigated Applications Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547 Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
47. Introduction Operator Empirical Study Conclusion Investigated Applications Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547 Application differ in size and complexity Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
48. Introduction Operator Empirical Study Conclusion Investigated Applications Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547 Application differ in size and complexity Differences in mutation coverage Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
49. Introduction Operator Empirical Study Conclusion Investigated Applications Files LOC Tests Mutants Generated Mutants Covered commons-math 408 39,991 2,169 80,372 72,203 commons-lang 99 19,495 2,039 31,130 29,069 commons-io 100 7,908 309 9,547 4,935 numerics4j 73 3,647 218 6,835 6,547 Application differ in size and complexity Differences in mutation coverage How prevalent are COR and ROR mutants? Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
50. Introduction Operator Empirical Study Conclusion Ratio of COR and ROR Mutants 0 20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
51. Introduction Operator Empirical Study Conclusion Ratio of COR and ROR Mutants 0 20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants COR and ROR generate up to 58% of all mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
52. Introduction Operator Empirical Study Conclusion Ratio of COR and ROR Mutants 0 20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants COR and ROR generate up to 58% of all mutants Indicates notable potential for improvements Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
53. Introduction Operator Empirical Study Conclusion Ratio of COR and ROR Mutants 0 20,000 40,000 60,000 80,000 numerics4j commons-io commons-lang commons-math 19.5% 14.5% 22.3% 5.1% 19.3% 26.9% 35.5% 24.3% Number of generated mutants All mutants ROR mutants COR mutants COR and ROR generate up to 58% of all mutants Indicates notable potential for improvements How much is the overall reduction? Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
54. Introduction Operator Empirical Study Conclusion Decrease in Number of Mutants Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
55. Introduction Operator Empirical Study Conclusion Decrease in Number of Mutants Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%) Overall reduction of up to 34% Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
56. Introduction Operator Empirical Study Conclusion Decrease in Number of Mutants Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%) Overall reduction of up to 34% Depends on mutation coverage Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
57. Introduction Operator Empirical Study Conclusion Decrease in Number of Mutants Generated Generated Covered Covered (original set) (reduced set) (original set) (reduced set) commons-math 80,372 66,787 (-16.9%) 72,203 59,195 (-18.0%) commons-lang 31,130 21,074 (-32.3%) 29,069 19,112 (-34.3%) commons-io 9,547 7,319 (-23.3%) 4,935 4,168 (-15.5%) numerics4j 6,835 5,437 (-20.5%) 6,547 5,149 (-21.4%) Overall reduction of up to 34% Depends on mutation coverage How much is the saving in runtime? Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
58. Introduction Operator Empirical Study Conclusion Runtime Improvement Runtime Runtime (original set) (reduced set) commons-math 300.77 271.10 (-09.9%) commons-lang 28.25 18.70 (-33.8%) commons-io 6.95 4.58 (-34.1%) numerics4j 2.85 2.08 (-26.9%) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
59. Introduction Operator Empirical Study Conclusion Runtime Improvement Runtime Runtime (original set) (reduced set) commons-math 300.77 271.10 (-09.9%) commons-lang 28.25 18.70 (-33.8%) commons-io 6.95 4.58 (-34.1%) numerics4j 2.85 2.08 (-26.9%) Divergence due to differences in test suite runtime and coverage Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
60. Introduction Operator Empirical Study Conclusion Runtime Improvement Runtime Runtime (original set) (reduced set) commons-math 300.77 271.10 (-09.9%) commons-lang 28.25 18.70 (-33.8%) commons-io 6.95 4.58 (-34.1%) numerics4j 2.85 2.08 (-26.9%) Divergence due to differences in test suite runtime and coverage Significant speed-up for all applications Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
61. Introduction Operator Empirical Study Conclusion Accuracy of the Mutation Score Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%) Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
62. Introduction Operator Empirical Study Conclusion Accuracy of the Mutation Score Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%) Mutation Score up to 10% overestimated Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
63. Introduction Operator Empirical Study Conclusion Accuracy of the Mutation Score Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%) Mutation Score up to 10% overestimated Low mutation coverage may lead to underestimation Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
64. Introduction Operator Empirical Study Conclusion Accuracy of the Mutation Score Mutation Score Mutation Score (original set) (reduced set) commons-math 0.77 0.73 (- 4.5%) commons-lang 0.76 0.67 (-10.7%) commons-io 0.41 0.44 ( 8.3%) numerics4j 0.69 0.65 (- 5.9%) Mutation Score up to 10% overestimated Low mutation coverage may lead to underestimation Redundant mutants tend to overestimate the mutation score Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
65. Introduction Operator Empirical Study Conclusion Conclusion and Future Work Conclusion: Operator for conditional expressions without redundancy Decreased number of mutants and improved runtime Increased accuracy of the mutation score Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
66. Introduction Operator Empirical Study Conclusion Conclusion and Future Work Conclusion: Operator for conditional expressions without redundancy Decreased number of mutants and improved runtime Increased accuracy of the mutation score Future Work: Investigate redundancies in other mutation operators Analyze whether sufficient mutants tend to be equivalent Apply constraint solver to identify equivalent mutants Just & Kapfhammer & Schweiggert Ulm University, Allegheny College Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?
67. Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? Thank you for your attention! Questions? http://www.mathematik.uni-ulm.de/sai/major