Mutation Testing Chris Sinjakli 

Testing is a good thing But how do we know our tests are good? 

Code coverage is a start But it can give a “good” score with really dreadful tests 

Really dreadful tests public int addTwoNumbers(int a, int b) { return a – b; } ... @Test public void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true); } Coverage: 100% Usefulness: 0 

No content

A contrived example But how could we detect it? 

Mutation Testing! “Who watches the watchmen?” 

If you can change the code, and a test doesn’t fail, either the code is never run or the tests are wrong. 

Going with our previous example public int addTwoNumbers(int a, int b) { return a – b; } ... @Test public void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true); } Let’s change something 

Going with our previous example public int addTwoNumbers(int a, int b) { return a + b; } ... @Test public void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true); } This still passes 

So it caught a really rubbish test How about something slightly less obvious? 

Slightly less obvious (and I mean slightly) public int checkConditions(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } @Test public void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result); } @Test public void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result); } Coverage: 100% Usefulness: >0 But still wrong 

Slightly less obvious (and I mean slightly) public int checkConditions(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } @Test public void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result); } @Test public void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result); } Mutate 

Slightly less obvious (and I mean slightly) public int checkConditions(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; } } @Test public void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result); } @Test public void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result); } Passing tests 

Mutation testing caught our mistake :D 

Useful technique But still has its flaws 

The downfall of mutation (Equivalent Mutants) int index = 0 while (someCondition) { doStuff(); index++; if (index == 100) { break; } } int index = 0 while (someCondition) { doStuff(); index++; if (index >= 100) { break; } } Mutates to But the programs are equivalent, so no test will fail 

Tools Some Java, then some Ruby 

Java • Loads of tools to choose from • Bytecode vs source mutation • Will look at PIT (seems like one of the better ones) 

PIT - pitest.org • Works with “everything” – Command line – Ant – Maven • Bytecode level mutations (faster) • Very customisable – Exclude classes/packages from mutation – Choose which mutations you want – Timeouts • Makes pretty HTML reports (line/mutation coverage) 

Ruby 

Ruby • Mutant seems to be the new favourite • Runs in Rubinius (1.8 or 1.9 mode) • Only supports RSpec • Easy to set up rvm install rbx-head rvm use rbx-head gem install mutant • And easy to use mutate “ClassName#method_to_test” spec 

Summary • Seems like it could identify areas of weakness in our tests • At the same time, could be very noisy • Might be worth just trying it against an existing project and seeing what happens 

Questions?