MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Just2011b/

Transcript

1. MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler Ren´ e Just1, Franz Schweiggert1, and Gregory M. Kapfhammer2 1Ulm University, Germany 2Allegheny College, USA 26th International Conference on Automated Software Engineering Lawrence, Kansas, USA November 6 - 12, 2011
2. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
3. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Compiler- Integrated Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
4. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Compiler- Integrated Fast and Scalable Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
5. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Fast and Scalable Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
6. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Fast and Scalable Configurable and Extensible Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
7. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Mutation Coverage Information Fast and Scalable Configurable and Extensible Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
8. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation Analysis Compiler- Integrated Domain Specific Language Mutation Coverage Information Fast and Scalable Configurable and Extensible Enables Optimized Workflow Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
9. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
10. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Methodically inject small syntactical faults into the program under test Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
11. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Methodically inject small syntactical faults into the program under test Unbiased and powerful method for assessing test oracles and input values Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
12. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Methodically inject small syntactical faults into the program under test Unbiased and powerful method for assessing test oracles and input values Useful method for fault seeding during the empirical study of testing techniques Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
13. Introduction MAJOR Conclusion Overview of Mutation Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
14. Introduction MAJOR Conclusion Overview of Mutation Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
15. Introduction MAJOR Conclusion Overview of Mutation Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } =⇒ =⇒ y = a - x; y = a + x; y = a / x; if(b < a) if(b != a) if(b == a) Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
16. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
17. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java Compiler Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
18. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
19. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Common Compiler Options Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
20. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Common Compiler Options Domain Specific Language Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
21. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java Compiler Source Files Common Compiler Options Domain Specific Language Bytecode with Embedded Mutants Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
22. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
23. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
24. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Define own mutation operator groups Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
25. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Define own mutation operator groups Enable operators for a specific package, class, or method Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
26. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
27. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
28. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
29. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
30. Introduction MAJOR Conclusion Conclusion Key Concepts and Features: Compiler-integrated solution Furnishes its own domain specific language Provides mutation coverage information Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
31. Introduction MAJOR Conclusion Conclusion Key Concepts and Features: Compiler-integrated solution Furnishes its own domain specific language Provides mutation coverage information Characteristics of MAJOR: Fast and scalable technique Configurable and extensible mutation tool Enables an optimized workflow for mutation analysis Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
32. Do you want to learn more details about MAJOR? MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler Ren´ e Just1 and Franz Schweiggert1 and Gregory M. Kapfhammer2 1Department of Applied Information Processing, Ulm University 2Department of Computer Science, Allegheny College IMPORTANT CONTRIBUTIONS Enhances the Java 6 Standard Edition compiler Provides its own domain specific language (DSL) Easily applicable in all Java development environments Effectively reduces mutant generation time to a minimum CONDITIONAL MUTATION Transforms the program’s abstract syntax tree (AST) Encapsulates the mutations within conditional statements ASSIGN IDENT y BINARY * a x ⇒ ASSIGN IDENT y COND-EXPR THEN BINARY - a x COND (M NO ==1) ELSE COND-EXPR THEN BINARY + a x COND (M NO ==2) ELSE BINARY * a x Figure: Multiple mutated binary expression as the right hand side of an assignment statement. SUPPORTED FEATURES Simple compiler options enable the mutation analysis Configurable mutation operators by means of a DSL Determination of mutation coverage by running the original code MUTATION COVERAGE public int eval(int x){ int a = 3, b = 1, y; y = (M_NO==1)? a - x: (M_NO==2)? a + x: (M_NO==3)? a % x: (M_NO==0 && COVERED(1,3))? a * x : a * x; // original if(M_NO==4){ y -= b; }else if(M_NO==0 && COVERED(4,4)){ y += b; }else{ y += b; // original } return y; } Figure: Collecting coverage information. It is impossible to kill a mutant if it is not reached and executed Additional instrumentation determines the covered mutations Mutation coverage is only examined if the tests execute the original code An external driver efficiently records the covered mutations as ranges Only those mutants covered by a test case are executed IMPLEMENTATION DETAILS Figure: Integration of the conditional mutation approach into the compilation process. listCOR={&&, ||, ==, !=}; method ="java.lang.System@println"; myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<method>; Figure: DSL script to define the mutation process. package major.mutation; public class Driver{ public static int MAX_NO = 100000; public static int[] COV = new int[MAX_NO]; public static int M_NO=0; public static boolean COVERED (int from, int to){ for(int i=from; i<=to; ++i){ COV[i]++; } return false; } } Figure: Simple driver class implementation. OPTIMIZED MUTATION ANALYSIS PROCESS Figure: Minimizing the runtime of mutation analysis by means of test prioritization and mutation coverage. 1. Embed and compile all mutants into the original program 2. Run tests on original program to gather runtime and coverage 3. Sort tests in ascending order according to their runtime 4. Perform mutation analysis while employing the reordered test suite and the mutation coverage information RUNTIME OF MAJOR’S COMPILER 1 2 3 4 5 6 7 8 9 10 11 12 0 20000 40000 60000 80000 100000 120000 140000 Compiler runtime in seconds Number of mutants apache ant jfreechart itext java pathfinder commons math commons lang numerics4j Figure: Compiler runtime to generate and compile the mutants for all of the projects. Negligible overhead for generating and compiling the mutants Applicable on commodity workstations, even for large projects EVALUATION OF MUTATION ANALYSIS PROCESSES 0 20 40 60 80 100 120 140 160 180 0 20 40 60 80 100 Number of killed mutants Runtime in seconds optimized order (using coverage information) random order (using coverage information) original order (using coverage information) optimized order (without coverage information) random order (without coverage information) original order (without coverage information) Figure: Runtime of the mutation analysis processes. FUTURE WORK Implementing new mutation operators at the semantic level Extending the domain specific language to support new operators Integrating conditional mutation into the new Java 7 compiler rene.just@uni-ulm.de 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011) gkapfham@allegheny.edu See you tomorrow for a live demonstration!