Mutation testing in continuous delivery pipelines

Mutation testing in continuous delivery pipelines

Mutation testing is a method of evaluating and improving the quality of our test suites. It introduces subtle bugs to source code of our projects and checks that our tests catch those synthetic faults. High computational cost of this technique have prevented it from being widely used in our standard programming toolchain.

Can we try to sneak it in through the backdoor of our continuous delivery pipelines? How much of it can we automate? How can our version control system help us in the process?

Let's try to answer those questions together.

As seen at Continuous Lifecycle 2017 in Mannheim, Germany.

Ae7a42fb716793697b1d222f3cc753b8?s=128

Jan Stępień

November 16, 2017
Tweet

Transcript

  1. 3.
  2. 4.
  3. 5.
  4. 13.
  5. 14.
  6. 16.

    1. Read all source files in a directory 2. Generate

    mutants from your code 3. Run the test suite for every mutant 4. Report all survivors
  7. 18.

    x and y x or y x < 1 x

    <= 1 if (a) b else c if (a) c else b f (a, b) { a + b } f (a, b) { a + b + 1 }
  8. 19.

    x and y x or y x < 1 x

    <= 1 if (a) b else c if (a) c else b f (a, b) { a + b } f (a, b) { null }
  9. 31.

    (…) our 3D engine has a lot of unit tests


    trying to cover as many features as possible over more than 100k lines of code. — Llorens Marti Garcia, IMVU, 2015
  10. 32.

    We find mutation testing to be highly valuable, and I

    hope this idea can help other engineers deliver solid, well-tested code. — Llorens Marti Garcia, IMVU, 2015