Kill the mutants, test your tests

Kill the mutants, test your tests

A846fc46522b396026adcb62e162b7dc?s=128

Joy of Coding

May 29, 2015
Tweet

Transcript

  1. KILL THE MUTANTS a better way to test your tests

  2. ABOUT ME • Roy van Rijn • Mutants: • Nora

    • Lucas • Works for
  3. SHOW OF HANDS let's do a

  4. WHO DOES • Unit testing • Test-driven development (TDD) •

    Continuous integration • Measure code coverage • Mutation testing
  5. UNIT TESTING • Prove your code works • Instant regression

    tests • Improve code design • Has become a mainstream practise over the last 10 years
  6. CONTINUOUS INTEGRATION • Automate testing • Maintain a single source

    repository • Collect statistics
  7. CODE COVERAGE • Measure the lines (or branches) that are

    executed during testing
  8. CODE COVERAGE • How did they test your car?

  9. CODE COVERAGE • Who has seen (or written?) tests •

    without verifications or assertions? • just to fake and boost coverage? • 100% branch coverage proves nothing
  10. QUIS CUSTODIET IPSOS CUSTODES? Who watches the watchmen?

  11. MUTATION TESTING • Proposed by Richard J. Lipton in 1971

    (winner of 2014 Knuth Prize) • A better way to measure the quality of your tests • Surge of interest in the 1980s • Time to revive this interest!
  12. TERMINOLOGY: MUTATION • A mutation is a (small) change in

    your codebase, for example:
  13. TERMINOLOGY: MUTANT • A mutant is a mutated version of

    your class
  14. MUTATION TESTING • Generate (a lot of) mutants of your

    codebase • Run (some of) your unit tests • Check the outcome!
  15. OUTCOME #1: KILLED • A mutant is killed if a

    test fails (detecting the mutated code) • This proves the mutated code is properly tested
  16. OUTCOME #2: LIVED • A mutant didn’t trigger a failing

    test…
  17. OUTCOME #3: TIMED OUT • The mutant caused the program

    loop, get stuck
  18. OTHER OUTCOMES • NON-VIABLE • JVM could not load the

    mutant bytecode • MEMORY ERROR • JVM ran out of memory during test • RUN ERROR • An error but none of the above.
  19. FAULT INJECTION? • With fault injection you test code •

    Inject faults/mutations and see how the system reacts • With mutation testing you test your tests • Inject faults/mutations and see how the tests react
  20. TOOLING • µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive) • Jester: http://jester.sourceforge.net/ (inactive) •

    Jumble: http://jumble.sourceforge.net/ (inactive) • javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive) • PIT: http://pitest.org/
  21. USING PIT • PIT uses configurable ‘mutators' • ASM (bytecode

    manipulation) is used to mutate your code • No mutated code is stored, it can't interfere with your code • Generates reports with test results
  22. MUTATORS: CONDITION BOUNDARY > into >=
 < into <=
 >=

    into >
 <= into <
  23. MUTATORS: NEGATE CONDITIONALS == into !=
 != into ==
 <=

    into >
 >= into <
 < into >=
 > into <=
  24. MUTATORS: REMOVE CONDITIONALS into if(true) {
 //something
 } if(a ==

    b) {
 //something
 }
  25. MUTATORS: MATH + into -
 - into +
 * into

    /
 / into *
 % into *
 & into | << into >>
 >> into <<
 >>> into <<<
 a++ into a--
 a-- into a++
  26. MUTATORS: MANY MORE • Replacing return values (return a; becomes

    return 0;) • Removal of void invocations (doSomething(); is removed) • Some enabled by default, others are optional/configurable
  27. MUTATION TESTING IS SLOW? • Speed was unacceptable in the

    80's • Mutation testing is still CPU intensive • But PIT has a lot of methods to speed it up!
  28. WHICH TESTS TO RUN? • PIT uses code coverage to

    decide which tests to run: • A mutation is on a line covered by 3 tests? Only run those.
  29. SIMPLE EXAMPLE • 100 classes • 10 unit tests per

    class • 2 ms per unit test • Total time (all tests): 100 x 10 x 2ms = 2s
  30. SIMPLE EXAMPLE • Total time (all tests): 100 x 10

    x 2ms = 2s • 8 mutants per class, 100 classes x 8 = 800 mutants • Brute force: 800 x 2s = 26m40s • Smart testing: 800 x 10 x 2ms = 16s
  31. LONGER EXAMPLE • Total time (all tests): 1000 x 10

    x 2ms = 20s • 8 mutants per class, 1000 classes x 8 = 8000 mutants • Brute force: 8000 x 20s = 1d20h26m40s…!!! • Smart testing: 8000 x 10 x 2ms = 2m40s
  32. PERFORMANCE TIPS • Write fast tests • Good separation or

    concerns • Use small classes, keep amount of unit tests per class low
  33. INCREMENTAL ANALYSIS • Experimental feature • Incremental analysis keeps track

    of: • Changes in the codebase • Previous results
  34. HOW ABOUT MOCKING? • PIT has support for: • Mockito,

    EasyMock, JMock, PowerMock and JMockit
  35. HOW TO USE PIT? • Standalone Java process • Build:

    Ant task, Maven plugin • CI: Sonarqube plugin, Gradle plugin • IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin
  36. STANDALONE JAVA java -cp <your classpath including pit jar and

    dependencies>
 org.pitest.mutationtest.commandline.MutationCoverageReport
 --reportDir /somePath/
 --targetClasses com.your.package.tobemutated*
 --targetTests com.your.package.*
 --sourceDirs /sourcePath/
  37. MAVEN PLUGIN Run as: mvn clean package org.pitest:pitest-maven:mutationCoverage <plugin> <groupId>org.pitest</groupId>

    <artifactId>pitest-maven</artifactId> <version>1.0.0</version> <configuration> <targetClasses> <param>com.your.package.tobemutated*</param> </targetClasses> <jvmArgs>…</jvmArgs> </configuration> </plugin>
  38. EXAMPLE Let’s kill some mutants… or be killed.

  39. USE CASE The price of an item is 17 euro

    If you buy 20 or more, all items cost 15 euro If you have a coupon, all items cost 15 euro
  40. CODE public int getPrice(int amountOfThings, boolean coupon) { if (amountOfThings

    >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  41. TEST #1 @Test public void testNormalPricing() { //Not enough for

    discount: int amount = 1; Assert.assertEquals(17, businessLogic.getPrice(amount, false)); }
  42. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if

    (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  43. TEST #2 @Test public void testDiscountPricingByAmount() { //Enough for discount:

    int amount = 100; Assert.assertEquals(1500, businessLogic.getPrice(amount, false)); }
  44. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if

    (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  45. TEST #3 @Test public void testDiscountWithCoupon() { //Not enough for

    discount, but coupon: int amount = 1; Assert.assertEquals(15, businessLogic.getPrice(amount, true)); }
  46. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if

    (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  47. PIT RESULT

  48. PIT RESULT > org.pitest.mutationtest…ConditionalsBoundaryMutator >> Generated 1 Killed 0 (0%)

    > KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 PIT tells us: Changing >= into > doesn’t trigger a failing test
  49. TEST #4 @Test public void testDiscountAmountCornerCase() { //Just enough for

    discount, mutation into > should fail this test int amount = 20; Assert.assertEquals(300, businessLogic.getPrice(amount, true)); }
  50. BRANCH COVERAGE public int getPrice(int amountOfThings, boolean coupon) { if

    (amountOfThings >= 20 || coupon) { return amountOfThings * 15; } return amountOfThings * 17; }
  51. PIT RESULT

  52. PIT RESULT > org.pitest.mutationtest…ConditionalsBoundaryMutator >> Generated 1 Killed 0 (0%)

    > KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 STILL WRONG!?
  53. DID YOU SPOT THE BUG? @Test public void testDiscountAmountCornerCase() {

    //Just enough for discount, mutation into > should fail this test int amount = 20; Assert.assertEquals(300, businessLogic.getPrice(amount, true)); }
  54. SUMMARY • Mutation testing automatically tests your tests • Mutation

    testing can find bugs in your tests • Code coverage is wrong, gives a false sense of security • Mutation testing with PIT is easy to implement
  55. QUESTIONS?