Mutation Testing Chris Sinjakli

Testing is a good thing But how do we know our tests are good?

Code coverage is a start But it can give a “good” score with really dreadful tests

Really dreadful tests class Adder def self.add (x, y) return x - y end end describe Adder do it "should add the two arguments" do Adder.add(1, 1) end end Coverage: 100% Usefulness: 0

No content

A contrived example But how could we detect it?

Mutation Testing! “Who watches the watchmen?”

If you can change the code, and a test doesn’t fail, either the code is never run or the tests are wrong.

How? 1. Run test suite 2. Change code (mutate) 3. Run test suite again If tests now fail, mutant dies. Otherwise it survives.

Going with our previous example class Adder def self.add (x, y) return x - y end end describe Adder do it "should add the two arguments" do Adder.add(1, 1) end end Let’s change something

Going with our previous example class Adder def self.add (x, y) return x + y end end describe Adder do it "should add the two arguments" do Adder.add(1, 1) end end This still passes

Success We know something is wrong

So what? It caught a really rubbish test How about something slightly less obvious?

Slightly less obvious (and I mean slightly) class ConditionChecker def self.check(a, b) if a && b return 42 else return 0 end end end describe ConditionChecker do it "should return 42 when both arguments are true" do ConditionChecker.check(true, true).should == 42 end it "should return 0 when both arguments are false" do ConditionChecker.check(false, false).should == 0 end end Coverage: 100% Usefulness: >0 But still wrong

Slightly less obvious (and I mean slightly) class ConditionChecker def self.check(a, b) if a && b return 42 else return 0 end end end describe ConditionChecker do it "should return 42 when both arguments are true" do ConditionChecker.check(true, true).should == 42 end it "should return 0 when both arguments are false" do ConditionChecker.check(false, false).should == 0 end end Mutate

Slightly less obvious (and I mean slightly) class ConditionChecker def self.check(a, b) if a || b return 42 else return 0 end end end describe ConditionChecker do it "should return 42 when both arguments are true" do ConditionChecker.check(true, true).should == 42 end it "should return 0 when both arguments are false" do ConditionChecker.check(false, false).should == 0 end end Passing tests

Mutation testing caught our mistake :D

Useful technique But still has its flaws

The downfall of mutation (Equivalent Mutants) index = 0 while index != 100 do doStuff() index += 1 end index = 0 while index < 100 do doStuff() index += 1 end Mutates to But the programs are equivalent, so no test will fail

There is no possible test which can “kill” the mutant The programs are equivalent

Also (potentially) • Infinite loops • More memory used • Compile/run time errors – tools should minimise these

How bad is it? • Good paper assessing the problem [SZ10] • Took 7 widely used, “large” projects • Found: – 15 mins to assess one mutation – 45% uncaught mutations are equivalent – Better tested project -> worse signal-to-noise ratio

Can we detect the equivalents? • Not in the general case [BA82] • Some specific cases can be detected – Using compiler optimisation techniques [BS79] – Using mathematical constraints [DO91] – Line coverage changes [SZ10] • All heuristic algorithms – not seen any claiming to kill all equivalent mutants

Tools Some Ruby, then a Java one I liked

Ruby • Looked into Heckle • Seemed unmaintained (nothing since 2009) • Then I saw...

Ruby

Ruby • Mutant seems to be the new favourite • Runs in Rubinius (1.8 or 1.9 mode) • Only supports RSpec • Easy to set up rvm install rbx-head rvm use rbx-head gem install mutant • And easy to use mutate “ClassName#method_to_test” spec

Java • Loads of tools to choose from • Bytecode vs source mutation • Will look at PIT (seems like one of the better ones)

PIT - pitest.org • Works with “everything” – Command line – Ant – Maven • Bytecode level mutations (faster) • Very customisable – Exclude classes/packages from mutation – Choose which mutations you want – Timeouts • Makes pretty HTML reports (line/mutation coverage)

Summary • Can point at weak areas in your tests • At the same time, can be prohibitively noisy • Try it and see

Questions?

References • [BA82] - T. A. Budd and D. Angluin. Two notions of correctness and their relation to testing. Acta Informatica, 18(1):31-45, November 1982. • [BS79] - D. Baldwin and F. Sayward. Heuristics for determining equivalence of program mutations. Research report 276, Department of Computer Science, Yale University, 1979. • [DO91] - R. A. DeMillo and A. J. O utt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900-910, September 1991. • [SZ10] - D. Schuler and A. Zeller. (Un-)Covering Equivalent Mutants. Third International Conference on Software Testing, Verification and Validation (ICST), pages 45-54. April 2010.

Also interesting • [AHH04] – K. Adamopoulos, M. Harman and R. M. Hierons. How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. Genetic and Evolutionary Computation -- GECCO 2004, pages 1338-1349. 2004.