Mutation Testing

Brown Bag lightning talk

Transcript

1. Mutation Testing
   Chris Sinjakli
   

   View Slide

2. Testing is a good thing
   But how do we know our tests are
   good?
   

   View Slide

3. Code coverage is a start
   But it can give a “good” score with
   really dreadful tests
   

   View Slide

4. Really dreadful tests
   public int addTwoNumbers(int a, int b) {
   return a – b;
   }
   ...
   @Test
   public void shouldAddTwoNumbers() {
   int result = addTwoNumbers(1, 1);
   assertTrue(true);
   }
   Coverage: 100%
   Usefulness: 0
   

   View Slide

5. View Slide

6. A contrived example
   But how could we detect it?
   

   View Slide

7. Mutation Testing!
   “Who watches the watchmen?”
   

   View Slide

8. If you can change the code, and a
   test doesn’t fail, either the code is
   never run or the tests are wrong.
   

   View Slide

9. Going with our previous example
   public int addTwoNumbers(int a, int b) {
   return a – b;
   }
   ...
   @Test
   public void shouldAddTwoNumbers() {
   int result = addTwoNumbers(1, 1);
   assertTrue(true);
   }
   Let’s change something
   

   View Slide

10. Going with our previous example
    public int addTwoNumbers(int a, int b) {
    return a + b;
    }
    ...
    @Test
    public void shouldAddTwoNumbers() {
    int result = addTwoNumbers(1, 1);
    assertTrue(true);
    }
    This still passes
    

    View Slide

11. So it caught a really rubbish test
    How about something slightly less
    obvious?
    

    View Slide

12. Slightly less obvious (and I mean slightly)
    public int checkConditions(boolean a, boolean b) {
    if (a && b) {
    return 42;
    }
    else {
    return 0;
    }
    }
    @Test
    public void testBothFalse() {
    int result = checkConditions(false, false);
    assertEquals(0, result);
    }
    @Test
    public void testBothTrue () {
    int result = checkConditions(true, true);
    assertEquals(42, result);
    }
    Coverage: 100%
    Usefulness: >0
    But still wrong
    

    View Slide

13. Slightly less obvious (and I mean slightly)
    public int checkConditions(boolean a, boolean b) {
    if (a && b) {
    return 42;
    }
    else {
    return 0;
    }
    }
    @Test
    public void testBothFalse() {
    int result = checkConditions(false, false);
    assertEquals(0, result);
    }
    @Test
    public void testBothTrue () {
    int result = checkConditions(true, true);
    assertEquals(42, result);
    }
    Mutate
    

    View Slide

14. Slightly less obvious (and I mean slightly)
    public int checkConditions(boolean a, boolean b) {
    if (a || b) {
    return 42;
    }
    else {
    return 0;
    }
    }
    @Test
    public void testBothFalse() {
    int result = checkConditions(false, false);
    assertEquals(0, result);
    }
    @Test
    public void testBothTrue () {
    int result = checkConditions(true, true);
    assertEquals(42, result);
    }
    Passing tests
    

    View Slide

15. Mutation testing caught our
    mistake
    :D
    

    View Slide

16. Useful technique
    But still has its flaws
    

    View Slide

17. The downfall of mutation
    (Equivalent Mutants)
    int index = 0
    while (someCondition) {
    doStuff();
    index++;
    if (index == 100) {
    break;
    }
    }
    int index = 0
    while (someCondition) {
    doStuff();
    index++;
    if (index >= 100) {
    break;
    }
    }
    Mutates to
    But the programs are equivalent, so no test will fail
    

    View Slide

18. Tools
    Some Java, then some Ruby
    

    View Slide

19. Java
    • Loads of tools to choose from
    • Bytecode vs source mutation
    • Will look at PIT (seems like one of the better
    ones)
    

    View Slide

20. PIT - pitest.org
    • Works with “everything”
    – Command line
    – Ant
    – Maven
    • Bytecode level mutations (faster)
    • Very customisable
    – Exclude classes/packages from mutation
    – Choose which mutations you want
    – Timeouts
    • Makes pretty HTML reports (line/mutation coverage)
    

    View Slide

21. Ruby
    

    View Slide

22. Ruby
    • Mutant seems to be the new favourite
    • Runs in Rubinius (1.8 or 1.9 mode)
    • Only supports RSpec
    • Easy to set up
    rvm install rbx-head
    rvm use rbx-head
    gem install mutant
    • And easy to use
    mutate “ClassName#method_to_test” spec
    

    View Slide

23. Summary
    • Seems like it could identify areas of weakness
    in our tests
    • At the same time, could be very noisy
    • Might be worth just trying it against an
    existing project and seeing what happens
    

    View Slide

24. Questions?
    

    View Slide