GENETICALLY MODIFIED TESTS
AN INTRODUCTION TO MUTATION TESTING
Droidcon Berlin 2017
1
Slide 2
Slide 2 text
ABOUT… XAVIER F. GOUCHET
ANDROID ARCHITECT AT DEEZER
#### ‘Mr Tools’ / CI Admin / UT Advocate #### Fluent in Android since Cupcake
ON ALL SOCIAL NETWORKS …
@XGOUCHET
2
Slide 3
Slide 3 text
♫ Put your hands in the air ♪ — Placebo
3
Slide 4
Slide 4 text
Unit tests ?
Integration tests ?
Functional tests ?
Test Driven Development ?
Code Coverage ?
Mutation Testing ?
4
Slide 5
Slide 5 text
CODE HAS BUGS
TESTS ARE CODE
∴
TESTS HAVE BUGS
5
Slide 6
Slide 6 text
TESTS ENSURE THE QUALITY OF YOUR CODE,
BUT WHAT ENSURES THE QUALITY OF YOUR TESTS?
6
Slide 7
Slide 7 text
♫ Let's start at the very beginning
(A very good place to start) ♪
— Julie Andrews
7
“When a measure becomes a target, it ceases to be a good measure.”
— Goodhart’s Law
14
Slide 15
Slide 15 text
“The more any indicator is used for decision-making, the more subject it will be
to corruption pressures.”
— Campbell's Law
15
Slide 16
Slide 16 text
THE GOAL OF TESTS ?
“Verify that the code works”
Make the contract explicit
Guide the development (TDD)
Prevent regression caused by
Other devs
Ourselves in the future
Ensure retrocompatibility
16
Slide 17
Slide 17 text
BAD TESTS CAN GIVE
A FALSE SENSE OF SECURITY
“We need to be as confident in the tests we code as we are in the code we test.”
— Me
17
Slide 18
Slide 18 text
♫ We're mutants and we just don't care ♪ — Oingo Boingo
18
Slide 19
Slide 19 text
MUTATION TESTING 101
1. Write tests
2. Mutate the code¹
3. Watch the tests fail
4. ???
5. Profit
19
Slide 20
Slide 20 text
STEP 1
Write the tests
Make sure all the tests are green ✓
20
Slide 21
Slide 21 text
STEP 2
Mutate the code¹
Make one or more modification to the source code
Mutation should break the behavior of the system
21
Slide 22
Slide 22 text
¹ WHAT'S A MUTATION ?
A MODIFICATION OF A STATEMENT IN THE CODE
Math operations switch
Condition boundaries
Constant values
Return values
…
22
Slide 23
Slide 23 text
STEP 3
Watch the tests fail
At least one test should break for each mutation ✗
… or not
23
Slide 24
Slide 24 text
WHY WOULD A MUTATION DIE ?
Test Condition Fails
Unexpected Exception
Non viable code
System error
Timeout
24
Slide 25
Slide 25 text
WHY WOULD A MUTATION SURVIVE ?
Uncovered
Silent mutation
Incomplete / bad test suite
25
Slide 26
Slide 26 text
♫ Ladies and gentlemen,
those magnificent examples of … ♪
— Paul McCartney
26
Slide 27
Slide 27 text
EXAMPLE
int check(boolean a, boolean b) {
if (a && b) {
return 42;
} else {
return 0;
}
}
void testCheck() {
assertEquals(check(true, true), 42);
assertEquals(check(false, false), 0);
}
27
Slide 28
Slide 28 text
EXAMPLE (MUTATED)
int check(boolean a, boolean b) {
if (a || b) { // {a && b} → {a || b}
return 42;
} else {
return 0;
}
}
void testCheck() {
assertEquals(check(true, true), 42);
assertEquals(check(false, false), 0);
}
28
♫ There’s somethin’ we can use,
so don't say no ♪
— Tony Basil
36
Slide 37
Slide 37 text
FREQUENTLY ASKED QUESTIONS
It does work with Kotlin…
…and other JVM languages too
…maybe
Configurable
Extensible
37
Slide 38
Slide 38 text
THINGS TO KEEP IN MIND
Mutants won’t find bugs in the code, just reveal test issues
Not bulletproof
Not a viable metric
Only simulate atomic faults
Costly
38
Slide 39
Slide 39 text
MY OWN RECOMMENDATIONS
Only used locally in the TDD process
Automatically triggered per PR
Not ran on CI server
Coverage value is not shared with management
Always take results with a grain of salt
39