Introduction to mutation testing

Tests ensure the quality of your code,
but what ensures the quality of your tests?

Transcript

1. Introduction to
   Mutation Testing
   Tests ensure the quality of your code,
   but what ensures the quality of your tests?
   CodeMobile UK 2017 - Chester
   

   View Slide

2. Android Architect
   Jenkins/Sonar Admin
   Bad Puns Advocate
   @xgouchet on Github,
   StackOverflow, Twitter, …
   About...
   Xavier F. Gouchet
   

   View Slide

3. “
   “Put your hands in the air”
   Placebo
   @xgouchet
   

   View Slide

4. ◎ Unit tests ?
   ◎ Integration tests ?
   ◎ Functional tests ?
   ◎ Test Driven Development ?
   ◎ Code Coverage ?
   ◎ Mutation Testing ?
   @xgouchet
   

   View Slide

5. “
   Code has bugs
   Tests are Code
   ∴
   Tests have bugs
   @xgouchet
   

   View Slide

6. “
   “Let's start at the very beginning
   (A very good place to start)”
   Julie Andrews
   @xgouchet
   

   View Slide

7. Unit testing
   @xgouchet
   

   View Slide

8. Integration testing
   @xgouchet
   

   View Slide

9. Functional testing
   @xgouchet
   

   View Slide

10. Test Driven Development
    @xgouchet
    

    View Slide

11. Code coverage
    @xgouchet
    

    View Slide

12. “
    The edge cases are where the fun happens
    — Kevlin Henney
    @xgouchet
    

    View Slide

13. SUCCESS: 26/26 (100%) Tests passed — @bloerwald
    @xgouchet
    

    View Slide

14. “
    When a measure becomes a target,
    it ceases to be a good measure.
    Goodhart’s Law
    @xgouchet
    

    View Slide

15. “
    The more any quantitative indicator
    is used for decision-making,
    the more subject it will be
    to corruption pressures
    Campbell’s Law
    @xgouchet
    

    View Slide

16. “
    “I'm always testing
    (the sh*t around me)”
    Korn
    @xgouchet
    

    View Slide

17. ◎ “Verify that the code works”
    ◎ Make the contract explicit
    ◎ Guide the development (TDD)
    ◎ Prevent regression caused by
    ○ Other devs
    ○ Ourselves in the future
    ◎ Ensure retrocompatibility
    What are the goals of tests ?
    @xgouchet
    

    View Slide

18. ◎ “Verify that the code works”
    ◎ Make the contract explicit
    ◎ Guide the development (TDD)
    ◎ Prevent regression caused by
    ○ Other devs
    ○ Ourselves in the future
    ◎ Ensure retrocompatibility
    What are the goals of tests ?
    @xgouchet
    

    View Slide

19. Bad tests can give you a false sense of security
    “We need to be as
    confident in the tests
    we code as we are in
    the code we test.”
    — Me
    @xgouchet
    

    View Slide

20. “
    “We're mutants and we just don't care ”
    Oingo Boingo
    @xgouchet
    

    View Slide

21. The basic idea behind Mutation Testing
    1. Write tests
    2. Mutate the code
    3. Watch the tests fail
    4. ???
    5. Profit
    @xgouchet
    

    View Slide

22. Step 1 : Write tests
    @xgouchet
    Make sure that all tests are green here !
    

    View Slide

23. Step 2 : Mutate the code
    @xgouchet
    

    View Slide

24. Step 3 : Watch the tests fail
    @xgouchet
    Mutation is detected by unit tests
    Well done !
    

    View Slide

25. Step 3 : … or not
    @xgouchet
    Mutation survives unit tests
    bad tests or good mutation ?
    

    View Slide

26. Why would a mutation die ?
    ◎ Test Condition
    Fails
    ◎ Unexpected
    Exception
    ◎ Non viable code
    ◎ System error
    ◎ Timeout
    @xgouchet
    

    View Slide

27. Why would a mutation survive ?
    ◎ Uncovered (dead
    code ?)
    ◎ Silent mutation
    (doesn’t affect the
    logic)
    ◎ Incomplete or bad
    test suite
    @xgouchet
    

    View Slide

28. “
    “Ladies and gentlemen,
    those magnificent examples of …”
    Paul McCartney
    @xgouchet
    

    View Slide

29. int add(int a, int b) {
    return a + b;
    }
    void testAdd() {
    assertThat(add(5, 3), greaterThan(0));
    }
    Mutation example ¹
    @xgouchet
    

    View Slide

30. int add(int a, int b) {
    return a - b;
    }
    void testAdd() {
    assertThat(add(5, 3), greaterThan(0));
    }
    Mutation example ¹
    @xgouchet
    

    View Slide

31. int add(int a, int b) {
    return a - b;
    }
    void testAdd() {
    assertEquals(add(5, 3), 8);
    }
    Mutation example ¹
    @xgouchet
    

    View Slide

32. int check(boolean a, boolean b) {
    if (a && b) {
    return 42;
    } else {
    return 0;
    }
    }
    void testCheck() {
    assertEquals(check(true, true), 42);
    assertEquals(check(false, false), 0);
    }
    Mutation example ²
    @xgouchet
    

    View Slide

33. int check(boolean a, boolean b) {
    if (a || b) {
    return 42;
    } else {
    return 0;
    }
    }
    void testCheck() {
    assertEquals(check(true, true), 42);
    assertEquals(check(false, false), 0);
    }
    Mutation example ²
    @xgouchet
    

    View Slide

34. int check(boolean a, boolean b) {
    if (a || b) {
    return 42;
    } else {
    return 0;
    }
    }
    void testCheck() {
    assertEquals(check(true, true), 42);
    assertEquals(check(false, false), 0);
    assertEquals(check(true, false), 0);
    assertEquals(check(false, true), 0);
    }
    Mutation example ²
    @xgouchet
    

    View Slide

35. void foo() {
    int i = 0;
    while (true) {
    doSomething();
    i++;
    if (i == 100) break;
    }
    }
    Mutation example ³
    @xgouchet
    

    View Slide

36. void foo() {
    int i = 0;
    while (true) {
    doSomething();
    i++;
    if (i >= 100) break;
    }
    }
    Mutation example ³
    @xgouchet
    

    View Slide

37. “
    “You may think you're normal,
    but you are all product of mutation”
    Rob Zombie
    @xgouchet
    

    View Slide

38. x + y ←→ x - y
    x * y ←→ x / y
    x | y ←→ x & y
    x % y → x * y
    x ^ y → x & y
    x >> y ←→ x << y
    x >>> y → x << y
    Math operators
    @xgouchet
    

    View Slide

39. x ≤ y ←→ x < y
    x ≥ y ←→ x > y
    x == y ←→ x != y
    if (x ≥ y) ←→ if (x > y)
    if (x ≤ y) ←→ if (x < y)
    Conditons
    @xgouchet
    

    View Slide

40. 1 → 0
    Byte.MAX_VALUE → Byte.MIN_VALUE
    Short.MAX_VALUE → Short.MIN_VALUE
    n → n + 1
    Inline Constants (int / byte / short / long)
    @xgouchet
    

    View Slide

41. 1.0 → 2.0
    n (not 1.0) → 1.0
    Inline Constants (float / double)
    @xgouchet
    

    View Slide

42. Return value
    return i; ←→ return i == 0 ? 1 : 0;
    return l; ←→ return l + 1;
    return d; ←→ return -(d + 1);
    return f; ←→ return -(f + 1);
    return o; →
    if (o ≠ null)
    return null;
    else
    throw …
    @xgouchet
    

    View Slide

43. Remove void methods
    @xgouchet
    void foo (…) {
    f(this.z);
    }
    void test (…) {
    foo(…);
    }
    →
    void foo (…) {
    f(this.z);
    }
    void test (…) {
    // nope
    }
    

    View Slide

44. Remove non void methods
    @xgouchet
    T foo (…) {
    return f(this.z);
    }
    void test (…) {
    return foo(…);
    }
    →
    T foo (…) {
    return f(this.z);
    }
    void test (…) {
    return null;
    }
    

    View Slide

45. Force if conditions
    @xgouchet
    if (…)
    foo();
    else
    bar();
    → foo();
    if (…)
    foo();
    else
    bar();
    → bar();
    

    View Slide

46. Misc
    @xgouchet
    -x → x
    ++i ←→ --i
    T foo (T t) {
    return f(t,
    this.z);
    }
    →
    T foo (T t) {
    return t;
    }
    Foo f = new Foo(…); → Foo f = null;
    

    View Slide

47. “
    “It's time to get our hands dirty”
    Natalie Grant
    @xgouchet
    

    View Slide

48. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet
    

    View Slide

49. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.android.architecture.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet
    

    View Slide

50. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.android.architecture.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet
    

    View Slide

51. buildscript {
    repositories {
    mavenCentral()
    }
    dependencies {
    classpath 'pl.droidsonroids.gradle:gradle-pitest-plugin:0.0.4'
    }
    }
    apply plugin: 'pl.droidsonroids.pitest'
    pitest {
    targetClasses = ['com.example.*']
    outputFormats = ['XML', 'HTML']
    }
    Gradle configuration
    @xgouchet
    

    View Slide

52. Mutation coverage
    @xgouchet
    

    View Slide

53. View Slide

54. “
    “There’s somethin’ we can use,
    so don't say no”
    Toni Basil
    @xgouchet
    

    View Slide

55. ◎ It does work with Kotlin…
    ◎ …and other JVM languages too
    ◎ …maybe
    ◎ Configurable
    ◎ Extensible
    Frequently asked questions
    @xgouchet
    

    View Slide

56. ◎ Mutants won’t find bugs in the code,
    just reveal test issues
    ◎ Not bulletproof
    ◎ Not a viable metric
    ◎ Only simulate atomic faults
    ◎ Costly
    Keep in mind...
    @xgouchet
    

    View Slide

57. ◎ Only used locally in the TDD process
    ◎ Automatically triggered in a pre-commit
    hook
    ◎ Not ran on CI server (yet)
    ◎ Coverage value is not shared with
    management
    Personnal reccomendations
    @xgouchet
    

    View Slide

58. “
    “Yeah, get out your toolbox boys”
    Iggy Pop
    @xgouchet
    

    View Slide

59. Mutation testing Framework
    ◎ Java Framework : Pitest
    pitest.org/
    ◎ IntelliJ / Android Studio plugin
    github.com/zalando/zester
    ◎ Android gradle plugin
    github.com/koral--/gradle-pitest-plugin
    ◎ Objective C mutation tests
    github.com/scjurgen/objective-C-mutationtests
    @xgouchet
    

    View Slide

60. Thanks!
    Any questions?
    @xgouchet
    

    View Slide