Genetically Modified Tests

Tests ensure the quality of your code, but what ensures the quality of your tests?
An introduction to mutation testing.

Transcript

1. GENETICALLY MODIFIED TESTS AN INTRODUCTION TO MUTATION TESTING Droidcon Berlin 2017 1
2. ABOUT… XAVIER F. GOUCHET ANDROID ARCHITECT AT DEEZER #### ‘Mr Tools’ / CI Admin / UT Advocate #### Fluent in Android since Cupcake ON ALL SOCIAL NETWORKS … @XGOUCHET 2
3. ♫ Put your hands in the air ♪ — Placebo 3
4. Unit tests ? Integration tests ? Functional tests ? Test Driven Development ? Code Coverage ? Mutation Testing ? 4
5. CODE HAS BUGS TESTS ARE CODE ∴ TESTS HAVE BUGS 5
6. TESTS ENSURE THE QUALITY OF YOUR CODE, BUT WHAT ENSURES THE QUALITY OF YOUR TESTS? 6
7. ♫ Let's start at the very beginning (A very good place to start) ♪ — Julie Andrews 7
8. UNIT TESTS 8
9. INTEGRATION TESTS 9
10. FUNCTIONNAL TESTS 10
11. TEST DRIVEN DEVELOPMENT 11
12. CODE COVERAGE 12
13. “SUCCESS: 26/26 (100%) Tests passed” — @bloerwald 13
14. “When a measure becomes a target, it ceases to be a good measure.” — Goodhart’s Law 14
15. “The more any indicator is used for decision-making, the more subject it will be to corruption pressures.” — Campbell's Law 15
16. THE GOAL OF TESTS ? “Verify that the code works” Make the contract explicit Guide the development (TDD) Prevent regression caused by Other devs Ourselves in the future Ensure retrocompatibility 16
17. BAD TESTS CAN GIVE A FALSE SENSE OF SECURITY “We need to be as confident in the tests we code as we are in the code we test.” — Me 17
18. ♫ We're mutants and we just don't care ♪ — Oingo Boingo 18
19. MUTATION TESTING 101 1. Write tests 2. Mutate the code¹ 3. Watch the tests fail 4. ??? 5. Profit 19
20. STEP 1 Write the tests Make sure all the tests are green ✓ 20
21. STEP 2 Mutate the code¹ Make one or more modification to the source code Mutation should break the behavior of the system 21
22. ¹ WHAT'S A MUTATION ? A MODIFICATION OF A STATEMENT IN THE CODE Math operations switch Condition boundaries Constant values Return values … 22
23. STEP 3 Watch the tests fail At least one test should break for each mutation ✗ … or not 23
24. WHY WOULD A MUTATION DIE ? Test Condition Fails Unexpected Exception Non viable code System error Timeout 24
25. WHY WOULD A MUTATION SURVIVE ? Uncovered Silent mutation Incomplete / bad test suite 25
26. ♫ Ladies and gentlemen, those magnificent examples of … ♪ — Paul McCartney 26
27. EXAMPLE int check(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } 27
28. EXAMPLE (MUTATED) int check(boolean a, boolean b) { if (a || b) { // {a && b} → {a || b} return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } 28
29. EXAMPLE (FIXED) void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(true, false), 0); assertEquals(check(false, true), 0); assertEquals(check(false, false), 0); } 29
30. ♫ It's time to get our hands dirty ♪ — Natalie Grant 30
31. GRADLE PLUGIN buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.9' } } apply plugin: 'pl.droidsonroids.pitest' 31
32. GRADLE PLUGIN (2.1+) plugins { id "pl.droidsonroids.pitest" version "0.0.9" } 32
33. CONFIGURATION pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } 33
35. HANDS ON 35
36. ♫ There’s somethin’ we can use, so don't say no ♪ — Tony Basil 36
37. FREQUENTLY ASKED QUESTIONS It does work with Kotlin… …and other JVM languages too …maybe Configurable Extensible 37
38. THINGS TO KEEP IN MIND Mutants won’t find bugs in the code, just reveal test issues Not bulletproof Not a viable metric Only simulate atomic faults Costly 38
39. MY OWN RECOMMENDATIONS Only used locally in the TDD process Automatically triggered per PR Not ran on CI server Coverage value is not shared with management Always take results with a grain of salt 39
40. Java Framework : Pitest IntelliJ / Android Studio plugin : Android gradle plugin : pitest.org github.com/zalando/zester github.com/koral--/gradle-pitest-plugin 40
41. THANKS FOR YOUR ATTENTION ANY QUESTION ? 41