Mutation testing
with Jan Stępień
in continuous delivery pipelines
janstepien.com @janstepien
Slide 2
Slide 2 text
@innoQ @cljmuc
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
We write tests
Slide 7
Slide 7 text
We write tests
Slide 8
Slide 8 text
Who is testing our tests?
@janstepien is not
Slide 9
Slide 9 text
Mutation testing
Slide 10
Slide 10 text
Competent programmer
hypothesis
Slide 11
Slide 11 text
Coupling effect
hypothesis
Slide 12
Slide 12 text
Mutants which
don’t get killed
become survivors
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Off the shelf
…and many more
Mutant
for Ruby
PIT
for Java
Stryker
for JavaScript
Slide 15
Slide 15 text
Mutation testing for Clojure
github.com/jstepien/mutant
servus!
Mutant.
Slide 16
Slide 16 text
1. Read all source files in a directory
2. Generate mutants from your code
3. Run the test suite for every mutant
4. Report all survivors
Slide 17
Slide 17 text
Generating mutants
Slide 18
Slide 18 text
x and y x or y
x < 1 x <= 1
if (a) b else c if (a) c else b
f (a, b) {
a + b
}
f (a, b) {
a + b + 1
}
Slide 19
Slide 19 text
x and y x or y
x < 1 x <= 1
if (a) b else c if (a) c else b
f (a, b) {
a + b
}
f (a, b) {
null
}
Slide 20
Slide 20 text
Recompiling the code
Slide 21
Slide 21 text
Running tests
Slide 22
Slide 22 text
Reporting survivors
Slide 23
Slide 23 text
Let’s talk about
problems
Slide 24
Slide 24 text
It’s slow
Do fewer, do faster,
or do smarter.
— Offutt and Untch, 2001
Slide 25
Slide 25 text
1. Select what to mutate
2. Select your mutation operators
3. Sample your mutants
Do fewer
Slide 26
Slide 26 text
Do faster
1. Don’t restart the virtual machine
2. Run tests in parallel
Slide 27
Slide 27 text
1. Reorder the test suite
2. Execute only relevant tests
Do smarter
Slide 28
Slide 28 text
Mutate continuously
@janstepien
Slide 29
Slide 29 text
git diff master~..master
Slide 30
Slide 30 text
Introduce gradually
@janstepien
Slide 31
Slide 31 text
(…) our 3D engine has a lot of unit tests
trying to cover as many features as possible
over more than 100k lines of code.
— Llorens Marti Garcia, IMVU, 2015
Slide 32
Slide 32 text
We find mutation testing to be highly valuable,
and I hope this idea can help other engineers
deliver solid, well-tested code.
— Llorens Marti Garcia, IMVU, 2015
Slide 33
Slide 33 text
Mutation testing
Yours continuously,
Jan Stępień
in continuous delivery pipelines
[email protected]
@janstepien