ABOUT ME
• Roy van Rijn
• Mutants:
• Nora
• Lucas
• Works for
Slide 3
Slide 3 text
SHOW OF HANDS
let's do a
Slide 4
Slide 4 text
WHO DOES
• Unit testing
• Test-driven development (TDD)
• Continuous integration
• Measure code coverage
• Mutation testing
Slide 5
Slide 5 text
UNIT TESTING
• Prove your code works
• Instant regression tests
• Improve code design
• Has become a mainstream practise over the last 10 years
Slide 6
Slide 6 text
CONTINUOUS INTEGRATION
• Automate testing
• Maintain a single source repository
• Collect statistics
Slide 7
Slide 7 text
CODE COVERAGE
• Measure the lines (or branches) that are executed during testing
Slide 8
Slide 8 text
CODE COVERAGE
• How did they test your car?
Slide 9
Slide 9 text
CODE COVERAGE
• Who has seen (or written?) tests
• without verifications or assertions?
• just to fake and boost coverage?
• 100% branch coverage proves nothing
Slide 10
Slide 10 text
QUIS CUSTODIET IPSOS CUSTODES?
Who watches the watchmen?
Slide 11
Slide 11 text
MUTATION TESTING
• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize)
• A better way to measure the quality of your tests
• Surge of interest in the 1980s
• Time to revive this interest!
Slide 12
Slide 12 text
TERMINOLOGY: MUTATION
• A mutation is a (small) change in your codebase, for example:
Slide 13
Slide 13 text
TERMINOLOGY: MUTANT
• A mutant is a mutated version of your class
Slide 14
Slide 14 text
MUTATION TESTING
• Generate (a lot of) mutants of your codebase
• Run (some of) your unit tests
• Check the outcome!
Slide 15
Slide 15 text
OUTCOME #1: KILLED
• A mutant is killed if a test fails (detecting the mutated code)
• This proves the mutated code is properly tested
Slide 16
Slide 16 text
OUTCOME #2: LIVED
• A mutant didn’t trigger a failing test…
Slide 17
Slide 17 text
OUTCOME #3: TIMED OUT
• The mutant caused the program loop, get stuck
Slide 18
Slide 18 text
OTHER OUTCOMES
• NON-VIABLE
• JVM could not load the mutant bytecode
• MEMORY ERROR
• JVM ran out of memory during test
• RUN ERROR
• An error but none of the above.
Slide 19
Slide 19 text
FAULT INJECTION?
• With fault injection you test code
• Inject faults/mutations and see how the system reacts
• With mutation testing you test your tests
• Inject faults/mutations and see how the tests react
USING PIT
• PIT uses configurable ‘mutators'
• ASM (bytecode manipulation) is used to mutate your code
• No mutated code is stored, it can't interfere with your code
• Generates reports with test results
Slide 22
Slide 22 text
MUTATORS: CONDITION BOUNDARY
> into >=
< into <=
>= into >
<= into <
Slide 23
Slide 23 text
MUTATORS: NEGATE CONDITIONALS
== into !=
!= into ==
<= into >
>= into <
< into >=
> into <=
Slide 24
Slide 24 text
MUTATORS: REMOVE CONDITIONALS
into
if(true) {
//something
}
if(a == b) {
//something
}
Slide 25
Slide 25 text
MUTATORS: MATH
+ into -
- into +
* into /
/ into *
% into *
& into |
<< into >>
>> into <<
>>> into <<<
a++ into a--
a-- into a++
Slide 26
Slide 26 text
MUTATORS: MANY MORE
• Replacing return values (return a; becomes return 0;)
• Removal of void invocations (doSomething(); is removed)
• Some enabled by default, others are optional/configurable
Slide 27
Slide 27 text
MUTATION TESTING IS SLOW?
• Speed was unacceptable in the 80's
• Mutation testing is still CPU intensive
• But PIT has a lot of methods to speed it up!
Slide 28
Slide 28 text
WHICH TESTS TO RUN?
• PIT uses code coverage to decide which tests to run:
• A mutation is on a line covered by 3 tests? Only run those.
Slide 29
Slide 29 text
SIMPLE EXAMPLE
• 100 classes
• 10 unit tests per class
• 2 ms per unit test
• Total time (all tests): 100 x 10 x 2ms = 2s
Slide 30
Slide 30 text
SIMPLE EXAMPLE
• Total time (all tests): 100 x 10 x 2ms = 2s
• 8 mutants per class, 100 classes x 8 = 800 mutants
• Brute force: 800 x 2s = 26m40s
• Smart testing: 800 x 10 x 2ms = 16s
Slide 31
Slide 31 text
LONGER EXAMPLE
• Total time (all tests): 1000 x 10 x 2ms = 20s
• 8 mutants per class, 1000 classes x 8 = 8000 mutants
• Brute force: 8000 x 20s = 1d20h26m40s…!!!
• Smart testing: 8000 x 10 x 2ms = 2m40s
Slide 32
Slide 32 text
PERFORMANCE TIPS
• Write fast tests
• Good separation or concerns
• Use small classes, keep amount of unit tests per class low
Slide 33
Slide 33 text
INCREMENTAL ANALYSIS
• Experimental feature
• Incremental analysis keeps track of:
• Changes in the codebase
• Previous results
Slide 34
Slide 34 text
HOW ABOUT MOCKING?
• PIT has support for:
• Mockito, EasyMock, JMock, PowerMock and JMockit
Slide 35
Slide 35 text
HOW TO USE PIT?
• Standalone Java process
• Build: Ant task, Maven plugin
• CI: Sonarqube plugin, Gradle plugin
• IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin
USE CASE
The price of an item is 17 euro
If you buy 20 or more, all items cost 15 euro
If you have a coupon, all items cost 15 euro
Slide 40
Slide 40 text
CODE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
Slide 41
Slide 41 text
TEST #1
@Test
public void testNormalPricing() {
//Not enough for discount:
int amount = 1;
Assert.assertEquals(17, businessLogic.getPrice(amount, false));
}
Slide 42
Slide 42 text
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
Slide 43
Slide 43 text
TEST #2
@Test
public void testDiscountPricingByAmount() {
//Enough for discount:
int amount = 100;
Assert.assertEquals(1500, businessLogic.getPrice(amount, false));
}
Slide 44
Slide 44 text
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
Slide 45
Slide 45 text
TEST #3
@Test
public void testDiscountWithCoupon() {
//Not enough for discount, but coupon:
int amount = 1;
Assert.assertEquals(15, businessLogic.getPrice(amount, true));
}
Slide 46
Slide 46 text
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
Slide 47
Slide 47 text
PIT RESULT
Slide 48
Slide 48 text
PIT RESULT
> org.pitest.mutationtest…ConditionalsBoundaryMutator
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
PIT tells us: Changing >= into > doesn’t trigger a failing test
Slide 49
Slide 49 text
TEST #4
@Test
public void testDiscountAmountCornerCase() {
//Just enough for discount, mutation into > should fail this test
int amount = 20;
Assert.assertEquals(300, businessLogic.getPrice(amount, true));
}
Slide 50
Slide 50 text
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {
if (amountOfThings >= 20 || coupon) {
return amountOfThings * 15;
}
return amountOfThings * 17;
}
DID YOU SPOT THE BUG?
@Test
public void testDiscountAmountCornerCase() {
//Just enough for discount, mutation into > should fail this test
int amount = 20;
Assert.assertEquals(300, businessLogic.getPrice(amount, true));
}
Slide 54
Slide 54 text
SUMMARY
• Mutation testing automatically tests your tests
• Mutation testing can find bugs in your tests
• Code coverage is wrong, gives a false sense of security
• Mutation testing with PIT is easy to implement