Introduction to mutation testing

Slide 1

Slide 1 text

Introduction to Mutation Testing Tests ensure the quality of your code, but what ensures the quality of your tests? CodeMobile UK 2017 - Chester

Slide 2

Slide 2 text

Android Architect Jenkins/Sonar Admin Bad Puns Advocate @xgouchet on Github, StackOverflow, Twitter, … About... Xavier F. Gouchet

Slide 3

Slide 3 text

“ “Put your hands in the air” Placebo @xgouchet

Slide 4

Slide 4 text

◎ Unit tests ? ◎ Integration tests ? ◎ Functional tests ? ◎ Test Driven Development ? ◎ Code Coverage ? ◎ Mutation Testing ? @xgouchet

Slide 5

Slide 5 text

“ Code has bugs Tests are Code ∴ Tests have bugs @xgouchet

Slide 6

Slide 6 text

“ “Let's start at the very beginning (A very good place to start)” Julie Andrews @xgouchet

Slide 7

Slide 7 text

Unit testing @xgouchet

Slide 8

Slide 8 text

Integration testing @xgouchet

Slide 9

Slide 9 text

Functional testing @xgouchet

Slide 10

Slide 10 text

Test Driven Development @xgouchet

Slide 11

Slide 11 text

Code coverage @xgouchet

Slide 12

Slide 12 text

“ The edge cases are where the fun happens — Kevlin Henney @xgouchet

Slide 13

Slide 13 text

SUCCESS: 26/26 (100%) Tests passed — @bloerwald @xgouchet

Slide 14

Slide 14 text

“ When a measure becomes a target, it ceases to be a good measure. Goodhart’s Law @xgouchet

Slide 15

Slide 15 text

“ The more any quantitative indicator is used for decision-making, the more subject it will be to corruption pressures Campbell’s Law @xgouchet

Slide 16

Slide 16 text

“ “I'm always testing (the sh*t around me)” Korn @xgouchet

Slide 17

Slide 17 text

◎ “Verify that the code works” ◎ Make the contract explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ○ Other devs ○ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Bad tests can give you a false sense of security “We need to be as confident in the tests we code as we are in the code we test.” — Me @xgouchet

Slide 20

Slide 20 text

“ “We're mutants and we just don't care ” Oingo Boingo @xgouchet

Slide 21

Slide 21 text

The basic idea behind Mutation Testing 1. Write tests 2. Mutate the code 3. Watch the tests fail 4. ??? 5. Profit @xgouchet

Slide 22

Slide 22 text

Step 1 : Write tests @xgouchet Make sure that all tests are green here !

Slide 23

Slide 23 text

Step 2 : Mutate the code @xgouchet

Slide 24

Slide 24 text

Step 3 : Watch the tests fail @xgouchet Mutation is detected by unit tests Well done !

Slide 25

Slide 25 text

Step 3 : … or not @xgouchet Mutation survives unit tests bad tests or good mutation ?

Slide 26

Slide 26 text

Why would a mutation die ? ◎ Test Condition Fails ◎ Unexpected Exception ◎ Non viable code ◎ System error ◎ Timeout @xgouchet

Slide 27

Slide 27 text

Why would a mutation survive ? ◎ Uncovered (dead code ?) ◎ Silent mutation (doesn’t affect the logic) ◎ Incomplete or bad test suite @xgouchet

Slide 28

Slide 28 text

“ “Ladies and gentlemen, those magnificent examples of …” Paul McCartney @xgouchet

Slide 29

Slide 29 text

int add(int a, int b) { return a + b; } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet

Slide 30

Slide 30 text

int add(int a, int b) { return a - b; } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet

Slide 31

Slide 31 text

int add(int a, int b) { return a - b; } void testAdd() { assertEquals(add(5, 3), 8); } Mutation example ¹ @xgouchet

Slide 32

Slide 32 text

int check(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet

Slide 33

Slide 33 text

int check(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet

Slide 34

Slide 34 text

int check(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); assertEquals(check(true, false), 0); assertEquals(check(false, true), 0); } Mutation example ² @xgouchet

Slide 35

Slide 35 text

void foo() { int i = 0; while (true) { doSomething(); i++; if (i == 100) break; } } Mutation example ³ @xgouchet

Slide 36

Slide 36 text

void foo() { int i = 0; while (true) { doSomething(); i++; if (i >= 100) break; } } Mutation example ³ @xgouchet

Slide 37

Slide 37 text

“ “You may think you're normal, but you are all product of mutation” Rob Zombie @xgouchet

Slide 38

Slide 38 text

x + y ←→ x - y x * y ←→ x / y x | y ←→ x & y x % y → x * y x ^ y → x & y x >> y ←→ x << y x >>> y → x << y Math operators @xgouchet

Slide 39

Slide 39 text

x ≤ y ←→ x < y x ≥ y ←→ x > y x == y ←→ x != y if (x ≥ y) ←→ if (x > y) if (x ≤ y) ←→ if (x < y) Conditons @xgouchet

Slide 40

Slide 40 text

1 → 0 Byte.MAX_VALUE → Byte.MIN_VALUE Short.MAX_VALUE → Short.MIN_VALUE n → n + 1 Inline Constants (int / byte / short / long) @xgouchet

Slide 41

Slide 41 text

1.0 → 2.0 n (not 1.0) → 1.0 Inline Constants (float / double) @xgouchet

Slide 42

Slide 42 text

Return value return i; ←→ return i == 0 ? 1 : 0; return l; ←→ return l + 1; return d; ←→ return -(d + 1); return f; ←→ return -(f + 1); return o; → if (o ≠ null) return null; else throw … @xgouchet

Slide 43

Slide 43 text

Remove void methods @xgouchet void foo (…) { f(this.z); } void test (…) { foo(…); } → void foo (…) { f(this.z); } void test (…) { // nope }

Slide 44

Slide 44 text

Remove non void methods @xgouchet T foo (…) { return f(this.z); } void test (…) { return foo(…); } → T foo (…) { return f(this.z); } void test (…) { return null; }

Slide 45

Slide 45 text

Force if conditions @xgouchet if (…) foo(); else bar(); → foo(); if (…) foo(); else bar(); → bar();

Slide 46

Slide 46 text

Misc @xgouchet -x → x ++i ←→ --i T foo (T t) { return f(t, this.z); } → T foo (T t) { return t; } Foo f = new Foo(…); → Foo f = null;

Slide 47

Slide 47 text

“ “It's time to get our hands dirty” Natalie Grant @xgouchet

Slide 48

Slide 48 text

buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet

Slide 49

Slide 49 text

buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet

Slide 50

Slide 50 text

buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Mutation coverage @xgouchet

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

“ “There’s somethin’ we can use, so don't say no” Toni Basil @xgouchet

Slide 55

Slide 55 text

◎ It does work with Kotlin… ◎ …and other JVM languages too ◎ …maybe ◎ Configurable ◎ Extensible Frequently asked questions @xgouchet

Slide 56

Slide 56 text

◎ Mutants won’t find bugs in the code, just reveal test issues ◎ Not bulletproof ◎ Not a viable metric ◎ Only simulate atomic faults ◎ Costly Keep in mind... @xgouchet

Slide 57

Slide 57 text

◎ Only used locally in the TDD process ◎ Automatically triggered in a pre-commit hook ◎ Not ran on CI server (yet) ◎ Coverage value is not shared with management Personnal reccomendations @xgouchet

Slide 58

Slide 58 text

“ “Yeah, get out your toolbox boys” Iggy Pop @xgouchet

Slide 59

Slide 59 text

Mutation testing Framework ◎ Java Framework : Pitest pitest.org/ ◎ IntelliJ / Android Studio plugin github.com/zalando/zester ◎ Android gradle plugin github.com/koral--/gradle-pitest-plugin ◎ Objective C mutation tests github.com/scjurgen/objective-C-mutationtests @xgouchet

Slide 60

Slide 60 text

Thanks! Any questions? @xgouchet