Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects

Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects

Interested in learning more about this topic? Visit this web site to read a related post: https://www.gregorykapfhammer.com/testing/mutation/research/2014/10/25/Major-Mutation/

4ae30d49c8cc07e42d5a871efb9bcfba?s=128

Gregory Kapfhammer

May 02, 2012
Tweet

Transcript

  1. 1.

    Efficient and Effective Mutation Testing: Supporting the Implementation of Quality

    Software by Purposefully Inserting Defects Gregory M. Kapfhammer† Department of Computer Science Allegheny College http://www.cs.allegheny.edu/∼gkapfham/ University of Delhi – May 2, 2012 †Joint with Ren´ e Just and Franz Schweiggert (University of Ulm) and Jonathan Miller Kauffman (Allegheny College)
  2. 2.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Important Points Presenter Introduction:

    Gregory M. Kapfhammer test testing software suites prioritization components coverage empirically regression suite applications evaluating algorithm analysis data effectiveness empirical genetic performance understanding using approach commercialofftheshelf comparison comprehensive constrained creation databasecentric environments execution finding framework identifying interactive java methods multiplots mutation party prioritized reduction relational study third timeaware towards 105 adequacy approaches array automatically building call challenges chapter communication compare compressing computer conditional cost cots covering creating criteria database databaseaware databasedriven databases declarative dependable detection devices distributed distributing duringsearchbased dynamic efficiency efficient engineering environment evaluate examination executing experimental family flow forward frameworks free generation greedy gui hamiltonian handbook heaps implementation improve incorporating increase information initial intranode invariant javaspace javaspacebased kernel knapsack linux measurement memory method monitoring operators paths poster potential preliminary primitives prioritizations prioritizers prioritizing problematic receive remote reports resource resourceconstrained results role runtime science searchbased selection solutions solvers space studies studying supported synthetic techniques transmission transparently trees tuple unstructured wrappers Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  3. 3.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Important Points Inspiration and

    Motivation The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be. Frederick P. Brooks, Jr. Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  4. 4.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Important Points Inspiration and

    Motivation The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be. Frederick P. Brooks, Jr. In reference to software! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  5. 5.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Important Points Inspiration and

    Motivation I believe the hard part of building software to be the specification, design, and testing of this con- ceptual construct, not the labor of representing it and testing the fidelity of the representation. Frederick P. Brooks, Jr. Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  6. 6.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Important Points Inspiration and

    Motivation I believe the hard part of building software to be the specification, design, and testing of this con- ceptual construct, not the labor of representing it and testing the fidelity of the representation. Frederick P. Brooks, Jr. What happens if the “incantation” is incorrect? Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  7. 7.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Important Points Inspiration and

    Motivation I believe the hard part of building software to be the specification, design, and testing of this con- ceptual construct, not the labor of representing it and testing the fidelity of the representation. Frederick P. Brooks, Jr. How do we efficiently and effectively test software? Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  8. 8.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  9. 9.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Input Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  10. 10.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Input Output Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  11. 11.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  12. 12.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  13. 13.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  14. 14.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  15. 15.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  16. 16.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  17. 17.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  18. 18.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  19. 19.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case passes and the code is correct! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  20. 20.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  21. 21.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Case? Method Under Test Test Set Up Input Output Test Clean Up Test Oracle Expected Output Test Verdict Expected Output Output Test Verdict The test case fails and a defect is found! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  22. 22.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  23. 23.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  24. 24.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  25. 25.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  26. 26.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  27. 27.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  28. 28.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  29. 29.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  30. 30.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  31. 31.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  32. 32.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  33. 33.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  34. 34.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  35. 35.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } B2 B2 B2 B2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  36. 36.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } B2 B2 B2 B2 B2 B2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  37. 37.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } B2 B2 B2 B2 B2 B2 B2 B2 B2 B2 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  38. 38.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  39. 39.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } How Good is Test Suite T? Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  40. 40.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } How Good is Test Suite T? Coverage Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  41. 41.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Software Testing What is

    a Test Suite? T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 Test Suite T = T1, T2, . . . , T9, T10 R1 R2 R3 R4 R5 R6 F1 F2 F3 F4 B1 B2 Requirements R = {R1 , . . . , R6 }, Features F = {F1 , . . . , F4 }, Bug Fixes B = {B1 , B2 } How Good is Test Suite T? Coverage Analysis Mutation Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  42. 42.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  43. 43.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  44. 44.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) Implemented Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  45. 45.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) Implemented Potential Fault Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  46. 46.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  47. 47.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  48. 48.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 0 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  49. 49.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 0 false false Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  50. 50.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 15 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  51. 51.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 15 true true Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  52. 52.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 10 Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  53. 53.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 10 false true Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  54. 54.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 10 false true Can the tests differentiate between implemented and potential fault? Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  55. 55.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 10 false true If yes, then the tests are adequate! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  56. 56.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 10 false true If no, then the tests must be improved! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  57. 57.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Conceptual Faults

    if(a > 10) if(a >= 10) a true false a true false a = 10 false true Purposefully insert faults in order to implement quality software! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  58. 58.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  59. 59.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  60. 60.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  61. 61.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  62. 62.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Methodically inject small syntactical faults into the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  63. 63.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Methodically inject small syntactical faults into the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  64. 64.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Methodically inject small syntactical faults into the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  65. 65.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Methodically inject small syntactical faults into the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  66. 66.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Mutation Operator Mutation Operator Mutation Operator Mutation Operator Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  67. 67.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  68. 68.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  69. 69.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  70. 70.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  71. 71.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  72. 72.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  73. 73.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  74. 74.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  75. 75.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  76. 76.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test The test suite cannot kill the mutant – either a test suite weakness or an equivalent mutant! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  77. 77.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Overview of

    Mutation Analysis Test Case T1 Test Case T2 Test Case T3 Test Case T4 Execute the test suite after enabling a single mutant in the program under test Repeat this process for all of the test cases and mutants – calculate mutation score when finished Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  78. 78.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  79. 79.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  80. 80.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  81. 81.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Conditional Mutation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  82. 82.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Conditional Mutation Syntax Tree Transformation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  83. 83.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Conditional Mutation Syntax Tree Transformation Expressions and Statements Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  84. 84.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Conditional Mutation Syntax Tree Transformation Expressions and Statements Compiler Integrated Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  85. 85.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Conditional Mutation Syntax Tree Transformation Expressions and Statements Compiler Integrated Comprehensive Empirical Study Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  86. 86.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Contributions of

    this Presentation Efficient Mutation Analysis Challenges Solutions Conditional Mutation Syntax Tree Transformation Expressions and Statements Compiler Integrated Comprehensive Empirical Study Efficient Technique - Fully Integrated into the Java 6 SE Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  87. 87.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Understanding Mutation

    Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  88. 88.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Understanding Mutation

    Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Methodically inject small syntactical faults into the program under test Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  89. 89.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Understanding Mutation

    Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  90. 90.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Understanding Mutation

    Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } =⇒ =⇒ • y = a - x; • y = a + x; • y = a / x; • if(b < a) • if(b != a) • if(b == a) Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  91. 91.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Understanding Mutation

    Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Unbiased and powerful method for assessing oracles and input values Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  92. 92.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Understanding Mutation

    Analysis public int eval(int x){ int a=3, b=1, y; y = a * x; y += b; return y; } public int max(int a, int b){ int max = a; if(b>a){ max=b; } return max; } Unbiased and powerful method for assessing oracles and input values Useful method for fault seeding during the empirical study of testing techniques Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  93. 93.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  94. 94.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  95. 95.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  96. 96.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  97. 97.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  98. 98.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants High Time Over- head for Generation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  99. 99.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants High Time Over- head for Generation Mutation Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  100. 100.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants High Time Over- head for Generation Mutation Analysis Tests Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  101. 101.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants High Time Over- head for Generation Mutation Analysis Tests Results Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  102. 102.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants High Time Over- head for Generation Mutation Analysis Tests Results Individually Executing the Mutants is Too Expensive Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  103. 103.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Mutation Analysis

    Challenges Mutant Generation Mutation Operators Program Mutants Often Yields a Substantial Num- ber of Mutants High Time Over- head for Generation Mutation Analysis Tests Results Individually Executing the Mutants is Too Expensive Prior Solutions? Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  104. 104.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  105. 105.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  106. 106.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  107. 107.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Sampling Selection Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  108. 108.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Do Smarter Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  109. 109.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Do Smarter Distributed Weak Mutation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  110. 110.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Do Smarter Do Faster Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  111. 111.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Do Smarter Do Faster Compiler Integrated Bytecode Transformation Mutant Schemata Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  112. 112.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Fundamental Concepts Prior Work

    in Mutation Analysis Improving Mutation Analysis Offutt and Untch Do Fewer Do Smarter Do Faster Higher Order Mutation Jia and Harman Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  113. 113.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Conditional Mutation Conditional Mutation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  114. 114.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Conditional Mutation Conditional Mutation Encapsulates all mutants within the same block Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  115. 115.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Conditional Mutation Conditional Mutation Encapsulates all mutants within the same block Can be inte- grated within the compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  116. 116.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Conditional Mutation Conditional Mutation Encapsulates all mutants within the same block Transforms the abstract syntax tree (AST) Can be inte- grated within the compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  117. 117.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Conditional Mutation Conditional Mutation Encapsulates all mutants within the same block Transforms the abstract syntax tree (AST) Stmt → Conditional Stmt (if-then-else, switch) Expr → Conditional Expr (conditional operator ?:) Can be inte- grated within the compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  118. 118.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Transforming the AST public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  119. 119.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Transforming the AST public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } ⇓ ASSIGN IDENT y BINARY ∗ a x Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  120. 120.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Transforming the AST public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } ⇓ ASSIGN IDENT y BINARY ∗ a x Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  121. 121.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Transforming the AST public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } ⇓ ASSIGN IDENT y BINARY ∗ a x =⇒ =⇒ ASSIGN IDENT y COND-EXPR THEN BINARY + a x COND (M NO ==2) ELSE COND-EXPR THEN BINARY - a x COND (M NO ==1) ELSE BINARY ∗ a x Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  122. 122.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  123. 123.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  124. 124.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = a * x ; y += b; return y; } 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  125. 125.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = (M_NO==1)? a - x : a * x ; y += b; return y; } 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  126. 126.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = (M_NO==2)? a + x : (M_NO==1)? a - x : a * x ; y += b; return y; } 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  127. 127.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = (M_NO==3)? a / x : (M_NO==2)? a + x : (M_NO==1)? a - x : a * x ; y += b; return y; } 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  128. 128.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Source Code View of Inserting Mutants public int eval(int x){ int a=3, b=1, y; y = (M_NO==3)? a / x : (M_NO==2)? a + x : (M_NO==1)? a - x : a * x ; y += b; return y; } Mutants that are not ex- ecuted cannot be killed 1 Define mutation operators MOP(x ∗ y) = {x − y, x + y, x/y} 2 Determine whether current expression or statement is affected by mutation 3 Apply mutation operators Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  129. 129.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Collecting and Using Mutation Coverage public int eval(int x){ int a=3, b=1, y; y = (M_NO==3)? a / x : (M_NO==2)? a + x : (M_NO==1)? a - x : a * x ; y += b; return y; } Mutants that are not ex- ecuted cannot be killed Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  130. 130.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Collecting and Using Mutation Coverage public int eval(int x){ int a=3, b=1, y; y = (M_NO==3)? a / x : (M_NO==2)? a + x : (M_NO==1)? a - x : (M_NO==0 && COVERED(1,3))? a * x : a * x ; y += b; return y; } Mutants that are not ex- ecuted cannot be killed Determine covered mutants with addi- tional instrumentation Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  131. 131.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Collecting and Using Mutation Coverage public int eval(int x){ int a=3, b=1, y; y = (M_NO==3)? a / x : (M_NO==2)? a + x : (M_NO==1)? a - x : (M_NO==0 && COVERED(1,3))? a * x : a * x ; y += b; return y; } Mutants that are not ex- ecuted cannot be killed Determine covered mutants with addi- tional instrumentation Only execute and investi- gate the covered mutants Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  132. 132.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Compiler MAJOR’s Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  133. 133.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Compiler MAJOR’s Compiler Enhanced the Standard Java Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  134. 134.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Compiler MAJOR’s Compiler Enhanced the Standard Java Compiler Source Files Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  135. 135.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Compiler MAJOR’s Compiler Enhanced the Standard Java Compiler Source Files Common Compiler Options Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  136. 136.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Compiler MAJOR’s Compiler Enhanced the Standard Java Compiler Source Files Common Compiler Options Domain Specific Language Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  137. 137.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Compiler MAJOR’s Compiler Enhanced the Standard Java Compiler Source Files Common Compiler Options Domain Specific Language Bytecode with Embedded Mutants Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  138. 138.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Integration into the Java Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  139. 139.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Integration into the Java Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  140. 140.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Integration into the Java Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  141. 141.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Integration into the Java Compiler Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  142. 142.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  143. 143.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  144. 144.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Define own mutation operator groups Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  145. 145.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    MAJOR’s Domain Specific Language // variable declaration listCOR={&&, ||, ==, !=}; // Define replacement list BIN(+)<"org"> -> {-,*}; BIN(*)<"org"> -> {/,%}; // Define own operator myOp{ BIN(&&) -> listCOR; BIN(||) -> listCOR; COR; LVR; } // Enable built-in operator AOR AOR<"org">; // Enable operator myOp myOp<"java.lang.System@println">; Specify mutation operators in detail Define own mutation operator groups Enable operators for a specific package, class, or method Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  146. 146.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  147. 147.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  148. 148.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  149. 149.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis with MAJOR

    Optimized Mutation Analysis Process 1 Embed and compile all mutants 2 Run test suite on instrumented program 3 Sort tests according to their runtime 4 Perform mutation analysis with reordered test suite Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  150. 150.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Mutant Generation

    and Compilation 1 2 3 4 5 6 7 8 9 10 11 12 0 20000 40000 60000 80000 100000 120000 140000 Compiler runtime in seconds Number of mutants apache ant jfreechart itext java pathfinder commons math commons lang numerics4j Overhead for generating and compiling mutants is negligible Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  151. 151.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Mutant Generation

    and Compilation 1 2 3 4 5 6 7 8 9 10 11 12 0 20000 40000 60000 80000 100000 120000 140000 Compiler runtime in seconds Number of mutants apache ant jfreechart itext java pathfinder commons math commons lang numerics4j Overhead for generating and compiling mutants is negligible Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  152. 152.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Time and

    Space Overhead Application Mutants Runtime of test suite Memory consumption original instrumented original instrumented wcs wcs+cov aspectj 406,382 4.3 4.8 5.0 559 813 apache ant 60,258 331.0 335.0 346.0 237 293 jfreechart 68,782 15.0 18.0 23.0 220 303 itext 124,184 5.1 5.6 6.3 217 325 java pathfinder 37,331 17.0 22.0 29.0 182 217 commons math 67,895 67.0 83.0 98.0 153 225 commons lang 25,783 10.3 11.8 14.8 104 149 numerics4j 5,869 1.2 1.3 1.6 73 90 • Runtime overhead is application dependent • Larger for CPU-bound applications • Small for I/O-bound applications • Even for large projects, applicable on commodity workstations Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  153. 153.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Time and

    Space Overhead Application Mutants Runtime of test suite Memory consumption original instrumented original instrumented wcs wcs+cov aspectj 406,382 4.3 4.8 5.0 559 813 apache ant 60,258 331.0 335.0 346.0 237 293 jfreechart 68,782 15.0 18.0 23.0 220 303 itext 124,184 5.1 5.6 6.3 217 325 java pathfinder 37,331 17.0 22.0 29.0 182 217 commons math 67,895 67.0 83.0 98.0 153 225 commons lang 25,783 10.3 11.8 14.8 104 149 numerics4j 5,869 1.2 1.3 1.6 73 90 • Runtime overhead is application dependent • Larger for CPU-bound applications • Small for I/O-bound applications • Even for large projects, applicable on commodity workstations Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  154. 154.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Time and

    Space Overhead Application Mutants Runtime of test suite Memory consumption original instrumented original instrumented wcs wcs+cov aspectj 406,382 4.3 4.8 5.0 559 813 apache ant 60,258 331.0 335.0 346.0 237 293 jfreechart 68,782 15.0 18.0 23.0 220 303 itext 124,184 5.1 5.6 6.3 217 325 java pathfinder 37,331 17.0 22.0 29.0 182 217 commons math 67,895 67.0 83.0 98.0 153 225 commons lang 25,783 10.3 11.8 14.8 104 149 numerics4j 5,869 1.2 1.3 1.6 73 90 • Runtime overhead is application dependent • Larger for CPU-bound applications • Small for I/O-bound applications • Even for large projects, applicable on commodity workstations Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  155. 155.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Time and

    Space Overhead Application Mutants Runtime of test suite Memory consumption original instrumented original instrumented wcs wcs+cov aspectj 406,382 4.3 4.8 5.0 559 813 apache ant 60,258 331.0 335.0 346.0 237 293 jfreechart 68,782 15.0 18.0 23.0 220 303 itext 124,184 5.1 5.6 6.3 217 325 java pathfinder 37,331 17.0 22.0 29.0 182 217 commons math 67,895 67.0 83.0 98.0 153 225 commons lang 25,783 10.3 11.8 14.8 104 149 numerics4j 5,869 1.2 1.3 1.6 73 90 • Runtime overhead is application dependent • Larger for CPU-bound applications • Small for I/O-bound applications • Even for large projects, applicable on commodity workstations Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  156. 156.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Compilation Efficiency Time and

    Space Overhead Application Mutants Runtime of test suite Memory consumption original instrumented original instrumented wcs wcs+cov aspectj 406,382 4.3 4.8 5.0 559 813 apache ant 60,258 331.0 335.0 346.0 237 293 jfreechart 68,782 15.0 18.0 23.0 220 303 itext 124,184 5.1 5.6 6.3 217 325 java pathfinder 37,331 17.0 22.0 29.0 182 217 commons math 67,895 67.0 83.0 98.0 153 225 commons lang 25,783 10.3 11.8 14.8 104 149 numerics4j 5,869 1.2 1.3 1.6 73 90 • Runtime overhead is application dependent • Larger for CPU-bound applications • Small for I/O-bound applications • Even for large projects, applicable on commodity workstations Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  157. 157.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis Efficiency Evaluating

    and Improving Mutation Analysis 0 20 40 60 80 100 120 140 160 180 0 20 40 60 80 100 Number of killed mutants Runtime in seconds optimized order (using coverage information) random order (using coverage information) original order (using coverage information) optimized order (without coverage information) random order (without coverage information) original order (without coverage information) • Mutation analysis is not feasible without coverage information • Reordering the test suite significantly speeds up the process, especially if runtimes of tests differ by orders of magnitude Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  158. 158.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis Efficiency Evaluating

    and Improving Mutation Analysis 0 20 40 60 80 100 120 140 160 180 0 20 40 60 80 100 Number of killed mutants Runtime in seconds optimized order (using coverage information) random order (using coverage information) original order (using coverage information) optimized order (without coverage information) random order (without coverage information) original order (without coverage information) • Mutation analysis is not feasible without coverage information • Reordering the test suite significantly speeds up the process, especially if runtimes of tests differ by orders of magnitude Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  159. 159.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Mutation Analysis Efficiency Evaluating

    and Improving Mutation Analysis 0 20 40 60 80 100 120 140 160 180 0 20 40 60 80 100 Number of killed mutants Runtime in seconds optimized order (using coverage information) random order (using coverage information) original order (using coverage information) optimized order (without coverage information) random order (without coverage information) original order (without coverage information) • Mutation analysis is not feasible without coverage information • Reordering the test suite significantly speeds up the process, especially if runtimes of tests differ by orders of magnitude Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  160. 160.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  161. 161.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Program Test Suite Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  162. 162.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Program Test Suite Mutation Score Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  163. 163.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Program Test Suite Mutation Score Improve Tests Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  164. 164.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Program Test Suite Mutation Score Improve Tests Automated Manual Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  165. 165.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Program Test Suite Mutation Score Improve Tests Use Tests Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  166. 166.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Improving Test Suite

    Quality Mutation Analysis Program Test Suite Mutation Score Improve Tests Use Tests Test improvement is only effective if mutation analysis is efficient! Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  167. 167.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Reviewing MAJOR’s Contributions

    Mutation Analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  168. 168.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Reviewing MAJOR’s Contributions

    Mutation Analysis Efficiency: MAJOR has ac- ceptable time and space over- heads and scales to large, real-world programs Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  169. 169.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Reviewing MAJOR’s Contributions

    Mutation Analysis Efficiency: MAJOR has ac- ceptable time and space over- heads and scales to large, real-world programs Usability: MAJOR’s inte- gration into the Java SE compiler makes it a no- hassle, drop-in tool Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  170. 170.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Retrospective Reviewing MAJOR’s Contributions

    Mutation Analysis Efficiency: MAJOR has ac- ceptable time and space over- heads and scales to large, real-world programs Usability: MAJOR’s inte- gration into the Java SE compiler makes it a no- hassle, drop-in tool We will release MAJOR as free and open source software Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  171. 171.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Conclusions and Future Work

    Conclusion Key Concepts and Features: • Compiler-integrated solution • Conditional mutation with the abstract syntax tree • Furnishes its own domain specific language • Collects and leverages mutation coverage information Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  172. 172.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Conclusions and Future Work

    Conclusion Key Concepts and Features: • Compiler-integrated solution • Conditional mutation with the abstract syntax tree • Furnishes its own domain specific language • Collects and leverages mutation coverage information Characteristics of MAJOR: • Fast and scalable technique • Configurable and extensible mutation tool • Enables an optimized workflow for mutation analysis Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  173. 173.

    Introduction Mutation Analysis Empirical Evaluation Conclusion Conclusions and Future Work

    Recently Published Papers • Ren´ e Just, Gregory M. Kapfhammer, and Franz Schweiggert. Using conditional mutation to increase the efficiency of mutation analysis. In Proceedings of the 6th International Workshop on the Automation of Software Test, Honolulu, Hawaii, May 2011. • Ren´ e Just, Franz Schweiggert, and Gregory M. Kapfhammer. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (Tool Paper), Lawrence, Kansas, November 2011. Kapfhammer Allegheny College Efficient and Effective Mutation Testing: Supporting the Implementation of Quality Software by Purposefully Inserting Defects
  174. 174.

    Efficient and Effective Mutation Testing: Supporting the Implementation of Quality

    Software by Purposefully Inserting Defects Gregory M. Kapfhammer Department of Computer Science Allegheny College http://www.cs.allegheny.edu/∼gkapfham/ Thank you for your attention! I welcome your questions and comments.