Introduction to mutation testing

Introduction to mutation testing

Tests ensure the quality of your code,
but what ensures the quality of your tests?

Cf95f93e78f6d6dd0630049396f723c6?s=128

Xavier Gouchet

April 20, 2017
Tweet

Transcript

  1. 1.

    Introduction to Mutation Testing Tests ensure the quality of your

    code, but what ensures the quality of your tests? CodeMobile UK 2017 - Chester
  2. 2.

    Android Architect Jenkins/Sonar Admin Bad Puns Advocate @xgouchet on Github,

    StackOverflow, Twitter, … About... Xavier F. Gouchet
  3. 4.

    ◎ Unit tests ? ◎ Integration tests ? ◎ Functional

    tests ? ◎ Test Driven Development ? ◎ Code Coverage ? ◎ Mutation Testing ? @xgouchet
  4. 6.

    “ “Let's start at the very beginning (A very good

    place to start)” Julie Andrews @xgouchet
  5. 14.

    “ When a measure becomes a target, it ceases to

    be a good measure. Goodhart’s Law @xgouchet
  6. 15.

    “ The more any quantitative indicator is used for decision-making,

    the more subject it will be to corruption pressures Campbell’s Law @xgouchet
  7. 17.

    ◎ “Verify that the code works” ◎ Make the contract

    explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ◦ Other devs ◦ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet
  8. 18.

    ◎ “Verify that the code works” ◎ Make the contract

    explicit ◎ Guide the development (TDD) ◎ Prevent regression caused by ◦ Other devs ◦ Ourselves in the future ◎ Ensure retrocompatibility What are the goals of tests ? @xgouchet
  9. 19.

    Bad tests can give you a false sense of security

    “We need to be as confident in the tests we code as we are in the code we test.” — Me @xgouchet
  10. 21.

    The basic idea behind Mutation Testing 1. Write tests 2.

    Mutate the code 3. Watch the tests fail 4. ??? 5. Profit @xgouchet
  11. 24.

    Step 3 : Watch the tests fail @xgouchet Mutation is

    detected by unit tests Well done !
  12. 25.

    Step 3 : … or not @xgouchet Mutation survives unit

    tests bad tests or good mutation ?
  13. 26.

    Why would a mutation die ? ◎ Test Condition Fails

    ◎ Unexpected Exception ◎ Non viable code ◎ System error ◎ Timeout @xgouchet
  14. 27.

    Why would a mutation survive ? ◎ Uncovered (dead code

    ?) ◎ Silent mutation (doesn’t affect the logic) ◎ Incomplete or bad test suite @xgouchet
  15. 29.

    int add(int a, int b) { return a + b;

    } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet
  16. 30.

    int add(int a, int b) { return a - b;

    } void testAdd() { assertThat(add(5, 3), greaterThan(0)); } Mutation example ¹ @xgouchet
  17. 31.

    int add(int a, int b) { return a - b;

    } void testAdd() { assertEquals(add(5, 3), 8); } Mutation example ¹ @xgouchet
  18. 32.

    int check(boolean a, boolean b) { if (a && b)

    { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet
  19. 33.

    int check(boolean a, boolean b) { if (a || b)

    { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); } Mutation example ² @xgouchet
  20. 34.

    int check(boolean a, boolean b) { if (a || b)

    { return 42; } else { return 0; } } void testCheck() { assertEquals(check(true, true), 42); assertEquals(check(false, false), 0); assertEquals(check(true, false), 0); assertEquals(check(false, true), 0); } Mutation example ² @xgouchet
  21. 35.

    void foo() { int i = 0; while (true) {

    doSomething(); i++; if (i == 100) break; } } Mutation example ³ @xgouchet
  22. 36.

    void foo() { int i = 0; while (true) {

    doSomething(); i++; if (i >= 100) break; } } Mutation example ³ @xgouchet
  23. 37.

    “ “You may think you're normal, but you are all

    product of mutation” Rob Zombie @xgouchet
  24. 38.

    x + y ←→ x - y x * y

    ←→ x / y x | y ←→ x & y x % y → x * y x ^ y → x & y x >> y ←→ x << y x >>> y → x << y Math operators @xgouchet
  25. 39.

    x ≤ y ←→ x < y x ≥ y

    ←→ x > y x == y ←→ x != y if (x ≥ y) ←→ if (x > y) if (x ≤ y) ←→ if (x < y) Conditons @xgouchet
  26. 40.

    1 → 0 Byte.MAX_VALUE → Byte.MIN_VALUE Short.MAX_VALUE → Short.MIN_VALUE n

    → n + 1 Inline Constants (int / byte / short / long) @xgouchet
  27. 41.
  28. 42.

    Return value return i; ←→ return i == 0 ?

    1 : 0; return l; ←→ return l + 1; return d; ←→ return -(d + 1); return f; ←→ return -(f + 1); return o; → if (o ≠ null) return null; else throw … @xgouchet
  29. 43.

    Remove void methods @xgouchet void foo (…) { f(this.z); }

    void test (…) { foo(…); } → void foo (…) { f(this.z); } void test (…) { // nope }
  30. 44.

    Remove non void methods @xgouchet T foo (…) { return

    f(this.z); } void test (…) { return foo(…); } → T foo (…) { return f(this.z); } void test (…) { return null; }
  31. 45.

    Force if conditions @xgouchet if (…) foo(); else bar(); →

    foo(); if (…) foo(); else bar(); → bar();
  32. 46.

    Misc @xgouchet -x → x ++i ←→ --i T foo

    (T t) { return f(t, this.z); } → T foo (T t) { return t; } Foo f = new Foo(…); → Foo f = null;
  33. 48.

    buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  34. 49.

    buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  35. 50.

    buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.android.architecture.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  36. 51.

    buildscript { repositories { mavenCentral() } dependencies { classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'

    } } apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } Gradle configuration @xgouchet
  37. 53.
  38. 55.

    ◎ It does work with Kotlin… ◎ …and other JVM

    languages too ◎ …maybe ◎ Configurable ◎ Extensible Frequently asked questions @xgouchet
  39. 56.

    ◎ Mutants won’t find bugs in the code, just reveal

    test issues ◎ Not bulletproof ◎ Not a viable metric ◎ Only simulate atomic faults ◎ Costly Keep in mind... @xgouchet
  40. 57.

    ◎ Only used locally in the TDD process ◎ Automatically

    triggered in a pre-commit hook ◎ Not ran on CI server (yet) ◎ Coverage value is not shared with management Personnal reccomendations @xgouchet
  41. 59.

    Mutation testing Framework ◎ Java Framework : Pitest pitest.org/ ◎

    IntelliJ / Android Studio plugin github.com/zalando/zester ◎ Android gradle plugin github.com/koral--/gradle-pitest-plugin ◎ Objective C mutation tests github.com/scjurgen/objective-C-mutationtests @xgouchet