Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to mutation testing

Introduction to mutation testing

Tests ensure the quality of your code,
but what ensures the quality of your tests?

Xavier Gouchet

April 20, 2017
Tweet

More Decks by Xavier Gouchet

Other Decks in Technology

Transcript

  1. Introduction to
    Mutation Testing
    Tests ensure the quality of your code,
    but what ensures the quality of your tests?
    CodeMobile UK 2017 - Chester

    View Slide

  2. Android Architect
    Jenkins/Sonar Admin
    Bad Puns Advocate
    @xgouchet on Github,
    StackOverflow, Twitter, …
    About...
    Xavier F. Gouchet

    View Slide


  3. “Put your hands in the air”
    Placebo
    @xgouchet

    View Slide

  4. ◎ Unit tests ?
    ◎ Integration tests ?
    ◎ Functional tests ?
    ◎ Test Driven Development ?
    ◎ Code Coverage ?
    ◎ Mutation Testing ?
    @xgouchet

    View Slide


  5. Code has bugs
    Tests are Code

    Tests have bugs
    @xgouchet

    View Slide


  6. “Let's start at the very beginning
    (A very good place to start)”
    Julie Andrews
    @xgouchet

    View Slide

  7. Unit testing
    @xgouchet

    View Slide

  8. Integration testing
    @xgouchet

    View Slide

  9. Functional testing
    @xgouchet

    View Slide

  10. Test Driven Development
    @xgouchet

    View Slide

  11. Code coverage
    @xgouchet

    View Slide


  12. The edge cases are where the fun happens
    — Kevlin Henney
    @xgouchet

    View Slide

  13. SUCCESS: 26/26 (100%) Tests passed — @bloerwald
    @xgouchet

    View Slide


  14. When a measure becomes a target,
    it ceases to be a good measure.
    Goodhart’s Law
    @xgouchet

    View Slide


  15. The more any quantitative indicator
    is used for decision-making,
    the more subject it will be
    to corruption pressures
    Campbell’s Law
    @xgouchet

    View Slide


  16. “I'm always testing
    (the sh*t around me)”
    Korn
    @xgouchet

    View Slide

  17. ◎ “Verify that the code works”
    ◎ Make the contract explicit
    ◎ Guide the development (TDD)
    ◎ Prevent regression caused by
    ○ Other devs
    ○ Ourselves in the future
    ◎ Ensure retrocompatibility
    What are the goals of tests ?
    @xgouchet

    View Slide

  18. ◎ “Verify that the code works”
    ◎ Make the contract explicit
    ◎ Guide the development (TDD)
    ◎ Prevent regression caused by
    ○ Other devs
    ○ Ourselves in the future
    ◎ Ensure retrocompatibility
    What are the goals of tests ?
    @xgouchet

    View Slide

  19. Bad tests can give you a false sense of security
    “We need to be as
    confident in the tests
    we code as we are in
    the code we test.”
    — Me
    @xgouchet

    View Slide


  20. “We're mutants and we just don't care ”
    Oingo Boingo
    @xgouchet

    View Slide

  21. The basic idea behind Mutation Testing
    1. Write tests
    2. Mutate the code
    3. Watch the tests fail
    4. ???
    5. Profit
    @xgouchet

    View Slide

  22. Step 1 : Write tests
    @xgouchet
    Make sure that all tests are green here !

    View Slide

  23. Step 2 : Mutate the code
    @xgouchet

    View Slide

  24. Step 3 : Watch the tests fail
    @xgouchet
    Mutation is detected by unit tests
    Well done !

    View Slide

  25. Step 3 : … or not
    @xgouchet
    Mutation survives unit tests
    bad tests or good mutation ?

    View Slide

  26. Why would a mutation die ?
    ◎ Test Condition
    Fails
    ◎ Unexpected
    Exception
    ◎ Non viable code
    ◎ System error
    ◎ Timeout
    @xgouchet

    View Slide

  27. Why would a mutation survive ?
    ◎ Uncovered (dead
    code ?)
    ◎ Silent mutation
    (doesn’t affect the
    logic)
    ◎ Incomplete or bad
    test suite
    @xgouchet

    View Slide


  28. “Ladies and gentlemen,
    those magnificent examples of …”
    Paul McCartney
    @xgouchet

    View Slide

  29. int add(int a, int b) {
    return a + b;
    }
    void testAdd() {
    assertThat(add(5, 3), greaterThan(0));
    }
    Mutation example ¹
    @xgouchet

    View Slide

  30. int add(int a, int b) {
    return a - b;
    }
    void testAdd() {
    assertThat(add(5, 3), greaterThan(0));
    }
    Mutation example ¹
    @xgouchet

    View Slide

  31. int add(int a, int b) {
    return a - b;
    }
    void testAdd() {
    assertEquals(add(5, 3), 8);
    }
    Mutation example ¹
    @xgouchet

    View Slide

  32. int check(boolean a, boolean b) {
    if (a && b) {
    return 42;
    } else {
    return 0;
    }
    }
    void testCheck() {
    assertEquals(check(true, true), 42);
    assertEquals(check(false, false), 0);
    }
    Mutation example ²
    @xgouchet

    View Slide

  33. int check(boolean a, boolean b) {
    if (a || b) {
    return 42;
    } else {
    return 0;
    }
    }
    void testCheck() {
    assertEquals(check(true, true), 42);
    assertEquals(check(false, false), 0);
    }
    Mutation example ²
    @xgouchet

    View Slide

  34. int check(boolean a, boolean b) {
    if (a || b) {
    return 42;
    } else {
    return 0;
    }
    }
    void testCheck() {
    assertEquals(check(true, true), 42);
    assertEquals(check(false, false), 0);
    assertEquals(check(true, false), 0);
    assertEquals(check(false, true), 0);
    }
    Mutation example ²
    @xgouchet

    View Slide

  35. void foo() {
    int i = 0;
    while (true) {
    doSomething();
    i++;
    if (i == 100) break;
    }
    }
    Mutation example ³
    @xgouchet

    View Slide

  36. void foo() {
    int i = 0;
    while (true) {
    doSomething();
    i++;
    if (i >= 100) break;
    }
    }
    Mutation example ³
    @xgouchet

    View Slide


  37. “You may think you're normal,
    but you are all product of mutation”
    Rob Zombie
    @xgouchet

    View Slide

  38. x + y ←→ x - y
    x * y ←→ x / y
    x | y ←→ x & y
    x % y → x * y
    x ^ y → x & y
    x >> y ←→ x << y
    x >>> y → x << y
    Math operators
    @xgouchet

    View Slide

  39. x ≤ y ←→ x < y
    x ≥ y ←→ x > y
    x == y ←→ x != y
    if (x ≥ y) ←→ if (x > y)
    if (x ≤ y) ←→ if (x < y)
    Conditons
    @xgouchet

    View Slide

  40. 1 → 0
    Byte.MAX_VALUE → Byte.MIN_VALUE
    Short.MAX_VALUE → Short.MIN_VALUE
    n → n + 1
    Inline Constants (int / byte / short / long)
    @xgouchet

    View Slide

  41. 1.0 → 2.0
    n (not 1.0) → 1.0
    Inline Constants (float / double)
    @xgouchet

    View Slide

  42. Return value
    return i; ←→ return i == 0 ? 1 : 0;
    return l; ←→ return l + 1;
    return d; ←→ return -(d + 1);
    return f; ←→ return -(f + 1);
    return o; →
    if (o ≠ null)
    return null;
    else
    throw …
    @xgouchet

    View Slide

  43. Remove void methods
    @xgouchet
    void foo (…) {
    f(this.z);
    }
    void test (…) {
    foo(…);
    }

    void foo (…) {
    f(this.z);
    }
    void test (…) {
    // nope
    }

    View Slide

  44. Remove non void methods
    @xgouchet
    T foo (…) {
    return f(this.z);
    }
    void test (…) {
    return foo(…);
    }

    T foo (…) {
    return f(this.z);
    }
    void test (…) {
    return null;
    }

    View Slide

  45. Force if conditions
    @xgouchet
    if (…)
    foo();
    else
    bar();
    → foo();
    if (…)
    foo();
    else
    bar();
    → bar();

    View Slide

  46. Misc
    @xgouchet
    -x → x
    ++i ←→ --i
    T foo (T t) {
    return f(t,
    this.z);
    }

    T foo (T t) {
    return t;
    }
    Foo f = new Foo(…); → Foo f = null;

    View Slide


  47. “It's time to get our hands dirty”
    Natalie Grant
    @xgouchet

    View Slide

  48. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet

    View Slide

  49. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.android.architecture.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet

    View Slide

  50. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.android.architecture.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet

    View Slide

  51. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet

    View Slide

  52. Mutation coverage
    @xgouchet

    View Slide

  53. View Slide


  54. “There’s somethin’ we can use,
    so don't say no”
    Toni Basil
    @xgouchet

    View Slide

  55. ◎ It does work with Kotlin…
    ◎ …and other JVM languages too
    ◎ …maybe
    ◎ Configurable
    ◎ Extensible
    Frequently asked questions
    @xgouchet

    View Slide

  56. ◎ Mutants won’t find bugs in the code,
    just reveal test issues
    ◎ Not bulletproof
    ◎ Not a viable metric
    ◎ Only simulate atomic faults
    ◎ Costly
    Keep in mind...
    @xgouchet

    View Slide

  57. ◎ Only used locally in the TDD process
    ◎ Automatically triggered in a pre-commit
    hook
    ◎ Not ran on CI server (yet)
    ◎ Coverage value is not shared with
    management
    Personnal reccomendations
    @xgouchet

    View Slide


  58. “Yeah, get out your toolbox boys”
    Iggy Pop
    @xgouchet

    View Slide

  59. Mutation testing Framework
    ◎ Java Framework : Pitest
    pitest.org/
    ◎ IntelliJ / Android Studio plugin
    github.com/zalando/zester
    ◎ Android gradle plugin
    github.com/koral--/gradle-pitest-plugin
    ◎ Objective C mutation tests
    github.com/scjurgen/objective-C-mutationtests
    @xgouchet

    View Slide

  60. Thanks!
    Any questions?
    @xgouchet

    View Slide