Introduction to mutation testing

Introduction to mutation testing

Tests ensure the quality of your code,
but what ensures the quality of your tests?

Cf95f93e78f6d6dd0630049396f723c6?s=128

Xavier Gouchet

April 20, 2017
Tweet

Transcript

  1. Introduction to Mutation Testing Tests ensure the quality of your

    code, but what ensures the quality of your tests? CodeMobile UK 2017 - Chester
  2. Android Architect Jenkins/Sonar Admin Bad Puns Advocate @xgouchet on Github,

    StackOverflow, Twitter, … About... Xavier F. Gouchet
  3. “ “Put your hands in the air” Placebo @xgouchet

  4. ◎ Unit tests ? ◎ Integration tests ? ◎ Functional

    tests ? ◎ Test Driven Development ? ◎ Code Coverage ? ◎ Mutation Testing ? @xgouchet
  5. “ Code has bugs Tests are Code ∴ Tests have

    bugs @xgouchet
  6. “ “Let's start at the very beginning (A very good

    place to start)” Julie Andrews @xgouchet
  7. Unit testing @xgouchet

  8. Integration testing @xgouchet

  9. Functional testing @xgouchet

  10. Test Driven Development @xgouchet

  11. Code coverage @xgouchet

  12. “ The edge cases are where the fun happens —

    Kevlin Henney @xgouchet
  13. SUCCESS: 26/26 (100%) Tests passed — @bloerwald @xgouchet

  14. “ When a measure becomes a target, it ceases to

    be a good measure. Goodhart’s Law @xgouchet
  15. “ The more any quantitative indicator is used for decision-making,

    the more subject it will be to corruption pressures Campbell’s Law @xgouchet
  16. “ “I'm always testing (the sh*t around me)” Korn @xgouchet

  17. ◎ “Verify that the code works” ◎ Make the contract

    explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ◦ Other devs ◦ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet
  18. ◎ “Verify that the code works” ◎ Make the contract

    explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ◦ Other devs ◦ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet
  19. Bad tests can give you a false sense of security

    “We need to be as confident in the tests we code as we are in the code we test.” — Me @xgouchet
  20. “ “We're mutants and we just don't care ” Oingo

    Boingo @xgouchet
  21. The basic idea behind Mutation Testing 1. Write tests 2.

    Mutate the code 3. Watch the tests fail 4. ??? 5. Profit @xgouchet
  22. Step 1 : Write tests @xgouchet Make sure that all

    tests are green here !
  23. Step 2 : Mutate the code @xgouchet

  24. Step 3 : Watch the tests fail @xgouchet Mutation is

    detected by unit tests Well done !
  25. Step 3 : … or not @xgouchet Mutation survives unit

    tests bad tests or good mutation ?
  26. Why would a mutation die ? ◎ Test Condition Fails

    ◎ Unexpected Exception ◎ Non viable code ◎ System error ◎ Timeout @xgouchet
  27. Why would a mutation survive ? ◎ Uncovered (dead code

    ?) ◎ Silent mutation (doesn’t affect the logic) ◎ Incomplete or bad test suite @xgouchet
  28. “ “Ladies and gentlemen, those magnificent examples of …” Paul

    McCartney @xgouchet
  29. int add(int a, int b) { return a + b;

    } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet
  30. int add(int a, int b) { return a - b;

    } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet
  31. int add(int a, int b) { return a - b;

    } void testAdd() { assertEquals(add(5, 3), 8); } Mutation example ¹ @xgouchet
  32. int check(boolean a, boolean b) { if (a && b)

    { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet
  33. int check(boolean a, boolean b) { if (a || b)

    { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet
  34. int check(boolean a, boolean b) { if (a || b)

    { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); assertEquals(check(true, false), 0); assertEquals(check(false, true), 0); } Mutation example ² @xgouchet
  35. void foo() { int i = 0; while (true) {

    doSomething(); i++; if (i == 100) break; } } Mutation example ³ @xgouchet
  36. void foo() { int i = 0; while (true) {

    doSomething(); i++; if (i >= 100) break; } } Mutation example ³ @xgouchet
  37. “ “You may think you're normal, but you are all

    product of mutation” Rob Zombie @xgouchet
  38. x + y ←→ x - y x * y

    ←→ x / y x | y ←→ x & y x % y → x * y x ^ y → x & y x >> y ←→ x << y x >>> y → x << y Math operators @xgouchet
  39. x ≤ y ←→ x < y x ≥ y

    ←→ x > y x == y ←→ x != y if (x ≥ y) ←→ if (x > y) if (x ≤ y) ←→ if (x < y) Conditons @xgouchet
  40. 1 → 0 Byte.MAX_VALUE → Byte.MIN_VALUE Short.MAX_VALUE → Short.MIN_VALUE n

    → n + 1 Inline Constants (int / byte / short / long) @xgouchet
  41. 1.0 → 2.0 n (not 1.0) → 1.0 Inline Constants

    (float / double) @xgouchet
  42. Return value return i; ←→ return i == 0 ?

    1 : 0; return l; ←→ return l + 1; return d; ←→ return -(d + 1); return f; ←→ return -(f + 1); return o; → if (o ≠ null) return null; else throw … @xgouchet
  43. Remove void methods @xgouchet void foo (…) { f(this.z); }

    void test (…) { foo(…); } → void foo (…) { f(this.z); } void test (…) { // nope }
  44. Remove non void methods @xgouchet T foo (…) { return

    f(this.z); } void test (…) { return foo(…); } → T foo (…) { return f(this.z); } void test (…) { return null; }
  45. Force if conditions @xgouchet if (…) foo(); else bar(); →

    foo(); if (…) foo(); else bar(); → bar();
  46. Misc @xgouchet -x → x ++i ←→ --i T foo

    (T t) { return f(t, this.z); } → T foo (T t) { return t; } Foo f = new Foo(…); → Foo f = null;
  47. “ “It's time to get our hands dirty” Natalie Grant

    @xgouchet
  48. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  49. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  50. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  51. buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  52. Mutation coverage @xgouchet

  53. None
  54. “ “There’s somethin’ we can use, so don't say no”

    Toni Basil @xgouchet
  55. ◎ It does work with Kotlin… ◎ …and other JVM

    languages too ◎ …maybe ◎ Configurable ◎ Extensible Frequently asked questions @xgouchet
  56. ◎ Mutants won’t find bugs in the code, just reveal

    test issues ◎ Not bulletproof ◎ Not a viable metric ◎ Only simulate atomic faults ◎ Costly Keep in mind... @xgouchet
  57. ◎ Only used locally in the TDD process ◎ Automatically

    triggered in a pre-commit hook ◎ Not ran on CI server (yet) ◎ Coverage value is not shared with management Personnal reccomendations @xgouchet
  58. “ “Yeah, get out your toolbox boys” Iggy Pop @xgouchet

  59. Mutation testing Framework ◎ Java Framework : Pitest pitest.org/ ◎

    IntelliJ / Android Studio plugin github.com/zalando/zester ◎ Android gradle plugin github.com/koral--/gradle-pitest-plugin ◎ Objective C mutation tests github.com/scjurgen/objective-C-mutationtests @xgouchet
  60. Thanks! Any questions? @xgouchet