Introduction to mutation testing

Tests ensure the quality of your code,
but what ensures the quality of your tests?

Transcript

1. Introduction to Mutation Testing Tests ensure the quality of your code, but what ensures the quality of your tests? CodeMobile UK 2017 - Chester
2. Android Architect Jenkins/Sonar Admin Bad Puns Advocate @xgouchet on Github, StackOverflow, Twitter, … About... Xavier F. Gouchet
3. “ “Put your hands in the air” Placebo @xgouchet
4. ◎ Unit tests ? ◎ Integration tests ? ◎ Functional tests ? ◎ Test Driven Development ? ◎ Code Coverage ? ◎ Mutation Testing ? @xgouchet
5. “ Code has bugs Tests are Code ∴ Tests have bugs @xgouchet
6. “ “Let's start at the very beginning (A very good place to start)” Julie Andrews @xgouchet
7. Unit testing @xgouchet
8. Integration testing @xgouchet
9. Functional testing @xgouchet
10. Test Driven Development @xgouchet
11. Code coverage @xgouchet
12. “ The edge cases are where the fun happens — Kevlin Henney @xgouchet
13. SUCCESS: 26/26 (100%) Tests passed — @bloerwald @xgouchet
14. “ When a measure becomes a target, it ceases to be a good measure. Goodhart’s Law @xgouchet
15. “ The more any quantitative indicator is used for decision-making, the more subject it will be to corruption pressures Campbell’s Law @xgouchet
16. “ “I'm always testing (the sh*t around me)” Korn @xgouchet
17. ◎ “Verify that the code works” ◎ Make the contract explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ◦ Other devs ◦ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet
18. ◎ “Verify that the code works” ◎ Make the contract explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ◦ Other devs ◦ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet
19. Bad tests can give you a false sense of security “We need to be as confident in the tests we code as we are in the code we test.” — Me @xgouchet
20. “ “We're mutants and we just don't care ” Oingo Boingo @xgouchet
21. The basic idea behind Mutation Testing 1. Write tests 2. Mutate the code 3. Watch the tests fail 4. ??? 5. Profit @xgouchet
22. Step 1 : Write tests @xgouchet Make sure that all tests are green here !
23. Step 2 : Mutate the code @xgouchet
24. Step 3 : Watch the tests fail @xgouchet Mutation is detected by unit tests Well done !
25. Step 3 : … or not @xgouchet Mutation survives unit tests bad tests or good mutation ?
26. Why would a mutation die ? ◎ Test Condition Fails ◎ Unexpected Exception ◎ Non viable code ◎ System error ◎ Timeout @xgouchet
27. Why would a mutation survive ? ◎ Uncovered (dead code ?) ◎ Silent mutation (doesn’t affect the logic) ◎ Incomplete or bad test suite @xgouchet
28. “ “Ladies and gentlemen, those magnificent examples of …” Paul McCartney @xgouchet
29. int add(int a, int b) { return a + b; } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet
30. int add(int a, int b) { return a - b; } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet
31. int add(int a, int b) { return a - b; } void testAdd() { assertEquals(add(5, 3), 8); } Mutation example ¹ @xgouchet
32. int check(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet
33. int check(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet
34. int check(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); assertEquals(check(true, false), 0); assertEquals(check(false, true), 0); } Mutation example ² @xgouchet
35. void foo() { int i = 0; while (true) { doSomething(); i++; if (i == 100) break; } } Mutation example ³ @xgouchet
36. void foo() { int i = 0; while (true) { doSomething(); i++; if (i >= 100) break; } } Mutation example ³ @xgouchet
37. “ “You may think you're normal, but you are all product of mutation” Rob Zombie @xgouchet
38. x + y ←→ x - y x * y ←→ x / y x | y ←→ x & y x % y → x * y x ^ y → x & y x >> y ←→ x << y x >>> y → x << y Math operators @xgouchet
39. x ≤ y ←→ x < y x ≥ y ←→ x > y x == y ←→ x != y if (x ≥ y) ←→ if (x > y) if (x ≤ y) ←→ if (x < y) Conditons @xgouchet
40. 1 → 0 Byte.MAX_VALUE → Byte.MIN_VALUE Short.MAX_VALUE → Short.MIN_VALUE n → n + 1 Inline Constants (int / byte / short / long) @xgouchet
41. 1.0 → 2.0 n (not 1.0) → 1.0 Inline Constants (float / double) @xgouchet
42. Return value return i; ←→ return i == 0 ? 1 : 0; return l; ←→ return l + 1; return d; ←→ return -(d + 1); return f; ←→ return -(f + 1); return o; → if (o ≠ null) return null; else throw … @xgouchet
43. Remove void methods @xgouchet void foo (…) { f(this.z); } void test (…) { foo(…); } → void foo (…) { f(this.z); } void test (…) { // nope }
44. Remove non void methods @xgouchet T foo (…) { return f(this.z); } void test (…) { return foo(…); } → T foo (…) { return f(this.z); } void test (…) { return null; }
45. Force if conditions @xgouchet if (…) foo(); else bar(); → foo(); if (…) foo(); else bar(); → bar();
46. Misc @xgouchet -x → x ++i ←→ --i T foo (T t) { return f(t, this.z); } → T foo (T t) { return t; } Foo f = new Foo(…); → Foo f = null;
47. “ “It's time to get our hands dirty” Natalie Grant @xgouchet
48. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
49. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
50. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
51. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4' } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
52. Mutation coverage @xgouchet
54. “ “There’s somethin’ we can use, so don't say no” Toni Basil @xgouchet
55. ◎ It does work with Kotlin… ◎ …and other JVM languages too ◎ …maybe ◎ Configurable ◎ Extensible Frequently asked questions @xgouchet
56. ◎ Mutants won’t find bugs in the code, just reveal test issues ◎ Not bulletproof ◎ Not a viable metric ◎ Only simulate atomic faults ◎ Costly Keep in mind... @xgouchet
57. ◎ Only used locally in the TDD process ◎ Automatically triggered in a pre-commit hook ◎ Not ran on CI server (yet) ◎ Coverage value is not shared with management Personnal reccomendations @xgouchet
58. “ “Yeah, get out your toolbox boys” Iggy Pop @xgouchet
59. Mutation testing Framework ◎ Java Framework : Pitest pitest.org/ ◎ IntelliJ / Android Studio plugin github.com/zalando/zester ◎ Android gradle plugin github.com/koral--/gradle-pitest-plugin ◎ Objective C mutation tests github.com/scjurgen/objective-C-mutationtests @xgouchet
60. Thanks! Any questions? @xgouchet