Mutation testing in continuous delivery pipelines

Mutation testing is a method of evaluating and improving the quality of our test suites. It introduces subtle bugs to source code of our projects and checks that our tests catch those synthetic faults. High computational cost of this technique have prevented it from being widely used in our standard programming toolchain.

Can we try to sneak it in through the backdoor of our continuous delivery pipelines? How much of it can we automate? How can our version control system help us in the process?

Let's try to answer those questions together.

As seen at Continuous Lifecycle 2017 in Mannheim, Germany.

Transcript

1. Mutation testing with Jan Stępień in continuous delivery pipelines janstepien.com @janstepien
2. @innoQ @cljmuc
6. We write tests
7. We write tests
8. Who is testing our tests? @janstepien is not
9. Mutation testing
10. Competent programmer hypothesis
11. Coupling effect hypothesis
12. Mutants which
 don’t get killed
 become survivors
14. Off the shelf …and many more Mutant
 for Ruby PIT
 for Java Stryker
 for JavaScript
15. Mutation testing for Clojure github.com/jstepien/mutant servus! Mutant.
16. 1. Read all source files in a directory 2. Generate mutants from your code 3. Run the test suite for every mutant 4. Report all survivors
17. Generating mutants
18. x and y x or y x < 1 x <= 1 if (a) b else c if (a) c else b f (a, b) { a + b } f (a, b) { a + b + 1 }
19. x and y x or y x < 1 x <= 1 if (a) b else c if (a) c else b f (a, b) { a + b } f (a, b) { null }
20. Recompiling the code
21. Running tests
22. Reporting survivors
23. Let’s talk about problems
24. It’s slow Do fewer, do faster,
 or do smarter. — Offutt and Untch, 2001
25. 1. Select what to mutate 2. Select your mutation operators 3. Sample your mutants Do fewer
26. Do faster 1. Don’t restart the virtual machine 2. Run tests in parallel
27. 1. Reorder the test suite 2. Execute only relevant tests Do smarter
28. Mutate continuously @janstepien
29. git diff master~..master
30. Introduce gradually @janstepien
31. (…) our 3D engine has a lot of unit tests
 trying to cover as many features as possible over more than 100k lines of code. — Llorens Marti Garcia, IMVU, 2015
32. We find mutation testing to be highly valuable, and I hope this idea can help other engineers deliver solid, well-tested code. — Llorens Marti Garcia, IMVU, 2015
33. Mutation testing Yours continuously, Jan Stępień in continuous delivery pipelines jan.stepien@innoq.com @janstepien