$30 off During Our Annual Pro Sale. View Details »

Mutation Testing

Mutation Testing

Brown Bag lightning talk

Chris Sinjakli

May 31, 2012
Tweet

More Decks by Chris Sinjakli

Other Decks in Programming

Transcript

  1. Mutation Testing
    Chris Sinjakli

    View Slide

  2. Testing is a good thing
    But how do we know our tests are
    good?

    View Slide

  3. Code coverage is a start
    But it can give a “good” score with
    really dreadful tests

    View Slide

  4. Really dreadful tests
    public int addTwoNumbers(int a, int b) {
    return a – b;
    }
    ...
    @Test
    public void shouldAddTwoNumbers() {
    int result = addTwoNumbers(1, 1);
    assertTrue(true);
    }
    Coverage: 100%
    Usefulness: 0

    View Slide

  5. View Slide

  6. A contrived example
    But how could we detect it?

    View Slide

  7. Mutation Testing!
    “Who watches the watchmen?”

    View Slide

  8. If you can change the code, and a
    test doesn’t fail, either the code is
    never run or the tests are wrong.

    View Slide

  9. Going with our previous example
    public int addTwoNumbers(int a, int b) {
    return a – b;
    }
    ...
    @Test
    public void shouldAddTwoNumbers() {
    int result = addTwoNumbers(1, 1);
    assertTrue(true);
    }
    Let’s change something

    View Slide

  10. Going with our previous example
    public int addTwoNumbers(int a, int b) {
    return a + b;
    }
    ...
    @Test
    public void shouldAddTwoNumbers() {
    int result = addTwoNumbers(1, 1);
    assertTrue(true);
    }
    This still passes

    View Slide

  11. So it caught a really rubbish test
    How about something slightly less
    obvious?

    View Slide

  12. Slightly less obvious (and I mean slightly)
    public int checkConditions(boolean a, boolean b) {
    if (a && b) {
    return 42;
    }
    else {
    return 0;
    }
    }
    @Test
    public void testBothFalse() {
    int result = checkConditions(false, false);
    assertEquals(0, result);
    }
    @Test
    public void testBothTrue () {
    int result = checkConditions(true, true);
    assertEquals(42, result);
    }
    Coverage: 100%
    Usefulness: >0
    But still wrong

    View Slide

  13. Slightly less obvious (and I mean slightly)
    public int checkConditions(boolean a, boolean b) {
    if (a && b) {
    return 42;
    }
    else {
    return 0;
    }
    }
    @Test
    public void testBothFalse() {
    int result = checkConditions(false, false);
    assertEquals(0, result);
    }
    @Test
    public void testBothTrue () {
    int result = checkConditions(true, true);
    assertEquals(42, result);
    }
    Mutate

    View Slide

  14. Slightly less obvious (and I mean slightly)
    public int checkConditions(boolean a, boolean b) {
    if (a || b) {
    return 42;
    }
    else {
    return 0;
    }
    }
    @Test
    public void testBothFalse() {
    int result = checkConditions(false, false);
    assertEquals(0, result);
    }
    @Test
    public void testBothTrue () {
    int result = checkConditions(true, true);
    assertEquals(42, result);
    }
    Passing tests

    View Slide

  15. Mutation testing caught our
    mistake
    :D

    View Slide

  16. Useful technique
    But still has its flaws

    View Slide

  17. The downfall of mutation
    (Equivalent Mutants)
    int index = 0
    while (someCondition) {
    doStuff();
    index++;
    if (index == 100) {
    break;
    }
    }
    int index = 0
    while (someCondition) {
    doStuff();
    index++;
    if (index >= 100) {
    break;
    }
    }
    Mutates to
    But the programs are equivalent, so no test will fail

    View Slide

  18. Tools
    Some Java, then some Ruby

    View Slide

  19. Java
    • Loads of tools to choose from
    • Bytecode vs source mutation
    • Will look at PIT (seems like one of the better
    ones)

    View Slide

  20. PIT - pitest.org
    • Works with “everything”
    – Command line
    – Ant
    – Maven
    • Bytecode level mutations (faster)
    • Very customisable
    – Exclude classes/packages from mutation
    – Choose which mutations you want
    – Timeouts
    • Makes pretty HTML reports (line/mutation coverage)

    View Slide

  21. Ruby

    View Slide

  22. Ruby
    • Mutant seems to be the new favourite
    • Runs in Rubinius (1.8 or 1.9 mode)
    • Only supports RSpec
    • Easy to set up
    rvm install rbx-head
    rvm use rbx-head
    gem install mutant
    • And easy to use
    mutate “ClassName#method_to_test” spec

    View Slide

  23. Summary
    • Seems like it could identify areas of weakness
    in our tests
    • At the same time, could be very noisy
    • Might be worth just trying it against an
    existing project and seeing what happens

    View Slide

  24. Questions?

    View Slide