Chris Sinjakli
May 31, 2012
53

# Mutation Testing

Brown Bag lightning talk

May 31, 2012

## Transcript

1. Mutation Testing
Chris Sinjakli

2. Testing is a good thing
But how do we know our tests are
good?

3. Code coverage is a start
But it can give a “good” score with

public int addTwoNumbers(int a, int b) {
return a – b;
}
...
@Test
assertTrue(true);
}
Coverage: 100%
Usefulness: 0

5. A contrived example
But how could we detect it?

6. Mutation Testing!
“Who watches the watchmen?”

7. If you can change the code, and a
test doesn’t fail, either the code is
never run or the tests are wrong.

8. Going with our previous example
public int addTwoNumbers(int a, int b) {
return a – b;
}
...
@Test
assertTrue(true);
}
Let’s change something

9. Going with our previous example
public int addTwoNumbers(int a, int b) {
return a + b;
}
...
@Test
assertTrue(true);
}
This still passes

10. So it caught a really rubbish test
obvious?

11. Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a && b) {
return 42;
}
else {
return 0;
}
}
@Test
public void testBothFalse() {
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);
assertEquals(42, result);
}
Coverage: 100%
Usefulness: >0
But still wrong

12. Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a && b) {
return 42;
}
else {
return 0;
}
}
@Test
public void testBothFalse() {
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);
assertEquals(42, result);
}
Mutate

13. Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a || b) {
return 42;
}
else {
return 0;
}
}
@Test
public void testBothFalse() {
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);
assertEquals(42, result);
}
Passing tests

14. Mutation testing caught our
mistake
:D

15. Useful technique
But still has its flaws

16. The downfall of mutation
(Equivalent Mutants)
int index = 0
while (someCondition) {
doStuff();
index++;
if (index == 100) {
break;
}
}
int index = 0
while (someCondition) {
doStuff();
index++;
if (index >= 100) {
break;
}
}
Mutates to
But the programs are equivalent, so no test will fail

17. Tools
Some Java, then some Ruby

18. Java
• Loads of tools to choose from
• Bytecode vs source mutation
• Will look at PIT (seems like one of the better
ones)

19. PIT - pitest.org
• Works with “everything”
– Command line
– Ant
– Maven
• Bytecode level mutations (faster)
• Very customisable
– Exclude classes/packages from mutation
– Choose which mutations you want
– Timeouts
• Makes pretty HTML reports (line/mutation coverage)

20. Ruby

21. Ruby
• Mutant seems to be the new favourite
• Runs in Rubinius (1.8 or 1.9 mode)
• Only supports RSpec
• Easy to set up