Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler

Gregory Kapfhammer
November 06, 2011
190

MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Just2011b/

Gregory Kapfhammer

November 06, 2011
Tweet

More Decks by Gregory Kapfhammer

Transcript

  1. MAJOR: An Efficient and Extensible Tool for Mutation Analysis in

    a Java Compiler Ren´ e Just1, Franz Schweiggert1, and Gregory M. Kapfhammer2 1Ulm University, Germany 2Allegheny College, USA 26th International Conference on Automated Software Engineering Lawrence, Kansas, USA November 6 - 12, 2011
  2. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  3. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Compiler- Integrated Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  4. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Compiler- Integrated Fast and Scalable Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  5. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Compiler- Integrated Domain Specific Language Fast and Scalable Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  6. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Compiler- Integrated Domain Specific Language Fast and Scalable Configurable and Extensible Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  7. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Compiler- Integrated Domain Specific Language Mutation Coverage Information Fast and Scalable Configurable and Extensible Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  8. Introduction MAJOR Conclusion Overview of MAJOR A Tool for Mutation

    Analysis Compiler- Integrated Domain Specific Language Mutation Coverage Information Fast and Scalable Configurable and Extensible Enables Optimized Workflow Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  9. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Just,

    Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  10. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Methodically

    inject small syntactical faults into the program under test Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  11. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Methodically

    inject small syntactical faults into the program under test Unbiased and powerful method for assessing test oracles and input values Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  12. Introduction MAJOR Conclusion Overview of Mutation Analysis Mutation Analysis Methodically

    inject small syntactical faults into the program under test Unbiased and powerful method for assessing test oracles and input values Useful method for fault seeding during the empirical study of testing techniques Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  13. Introduction MAJOR Conclusion Overview of Mutation Analysis public int eval(int

    x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  14. Introduction MAJOR Conclusion Overview of Mutation Analysis public int eval(int

    x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  15. Introduction MAJOR Conclusion Overview of Mutation Analysis public int eval(int

    x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } =⇒ =⇒ y = a - x; y = a + x; y = a / x; if(b < a) if(b != a) if(b == a) Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  16. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Just, Kapfhammer, and

    Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  17. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java

    Compiler Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  18. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java

    Compiler Source Files Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  19. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java

    Compiler Source Files Common Compiler Options Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  20. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java

    Compiler Source Files Common Compiler Options Domain Specific Language Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  21. Introduction MAJOR Conclusion MAJOR’s Compiler MAJOR’s Compiler Enhanced Standard Java

    Compiler Source Files Common Compiler Options Domain Specific Language Bytecode with Embedded Mutants Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  22. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration

    listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  23. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration

    listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  24. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration

    listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Define own mutation operator groups Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  25. Introduction MAJOR Conclusion MAJOR’s Domain Specific Language // variable declaration

    listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Define own mutation operator groups Enable operators for a specific package, class, or method Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  26. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and

    compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  27. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and

    compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  28. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and

    compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  29. Introduction MAJOR Conclusion Optimized Mutation Analysis Process 1 Embed and

    compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  30. Introduction MAJOR Conclusion Conclusion Key Concepts and Features: Compiler-integrated solution

    Furnishes its own domain specific language Provides mutation coverage information Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  31. Introduction MAJOR Conclusion Conclusion Key Concepts and Features: Compiler-integrated solution

    Furnishes its own domain specific language Provides mutation coverage information Characteristics of MAJOR: Fast and scalable technique Configurable and extensible mutation tool Enables an optimized workflow for mutation analysis Just, Kapfhammer, and Schweiggert Ulm University, Allegheny College MAJOR: An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler
  32. Do you want to learn more details about MAJOR? MAJOR:

    An Efficient and Extensible Tool for Mutation Analysis in a Java Compiler Ren´ e Just1 and Franz Schweiggert1 and Gregory M. Kapfhammer2 1Department of Applied Information Processing, Ulm University 2Department of Computer Science, Allegheny College IMPORTANT CONTRIBUTIONS Enhances the Java 6 Standard Edition compiler Provides its own domain specific language (DSL) Easily applicable in all Java development environments Effectively reduces mutant generation time to a minimum CONDITIONAL MUTATION Transforms the program’s abstract syntax tree (AST) Encapsulates the mutations within conditional statements ASSIGN IDENT y BINARY * a x ⇒ ASSIGN IDENT y COND-EXPR THEN BINARY - a x COND (M NO ==1) ELSE COND-EXPR THEN BINARY + a x COND (M NO ==2) ELSE BINARY * a x Figure: Multiple mutated binary expression as the right hand side of an assignment statement. SUPPORTED FEATURES Simple compiler options enable the mutation analysis Configurable mutation operators by means of a DSL Determination of mutation coverage by running the original code MUTATION COVERAGE public int eval(int x){ int a = 3, b = 1, y; y = (M_NO==1)? a - x: (M_NO==2)? a + x: (M_NO==3)? a % x: (M_NO==0 && COVERED(1,3))? a * x : a * x; // original if(M_NO==4){ y -= b; }else if(M_NO==0 && COVERED(4,4)){ y += b; }else{ y += b; // original } return y; } Figure: Collecting coverage information. It is impossible to kill a mutant if it is not reached and executed Additional instrumentation determines the covered mutations Mutation coverage is only examined if the tests execute the original code An external driver efficiently records the covered mutations as ranges Only those mutants covered by a test case are executed IMPLEMENTATION DETAILS Figure: Integration of the conditional mutation approach into the compilation process. listCOR={&&, ||, ==, !=}; method ="java.lang.System@println"; myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<method>; Figure: DSL script to define the mutation process. package major.mutation; public class Driver{ public static int MAX_NO = 100000; public static int[] COV = new int[MAX_NO]; public static int M_NO=0; public static boolean COVERED (int from, int to){ for(int i=from; i<=to; ++i){ COV[i]++; } return false; } } Figure: Simple driver class implementation. OPTIMIZED MUTATION ANALYSIS PROCESS Figure: Minimizing the runtime of mutation analysis by means of test prioritization and mutation coverage. 1. Embed and compile all mutants into the original program 2. Run tests on original program to gather runtime and coverage 3. Sort tests in ascending order according to their runtime 4. Perform mutation analysis while employing the reordered test suite and the mutation coverage information RUNTIME OF MAJOR’S COMPILER 1 2 3 4 5 6 7 8 9 10 11 12 0 20000 40000 60000 80000 100000 120000 140000 Compiler runtime in seconds Number of mutants apache ant jfreechart itext java pathfinder commons math commons lang numerics4j Figure: Compiler runtime to generate and compile the mutants for all of the projects. Negligible overhead for generating and compiling the mutants Applicable on commodity workstations, even for large projects EVALUATION OF MUTATION ANALYSIS PROCESSES 0 20 40 60 80 100 120 140 160 180 0 20 40 60 80 100 Number of killed mutants Runtime in seconds optimized order (using coverage information) random order (using coverage information) original order (using coverage information) optimized order (without coverage information) random order (without coverage information) original order (without coverage information) Figure: Runtime of the mutation analysis processes. FUTURE WORK Implementing new mutation operators at the semantic level Extending the domain specific language to support new operators Integrating conditional mutation into the new Java 7 compiler [email protected] 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011) [email protected] See you tomorrow for a live demonstration!