Getting Started with Mutation Testing

A quick introduction to mutation testing in PHP with infection. This talk covers how to set up infection in a project, how to read the reports it generates and why mutation coverage might be more helpful than line coverage provided by phpunit. The talk closes with tips on how to improve performance as mutation testing will produce a test matrix that inevitably will be slower than the test suite its based upon.

Transcript

1. Getting
   Started With
   Mutation
   Testing
   Denis Brumann
   denis.brumann@sensiolabs.de
   

   View Slide

2. Denis Brumann
   @dbrumann
   denis.brumann@sensiolabs.de
   

   View Slide

3. Mutation testing involves modifying a program in small
   ways.
   If mutated program produces failing tests, this is called a
   killed mutant. If tests are green with mutated code, then
   we have an escaped mutant.
   

   View Slide

4. Installation
   Phar, Phive, Composer, Git or Homebrew
   Infection requires a recent version of PHP, and
   XDebug, phpdbg, or pcov enabled.
   

   View Slide

5. View Slide

6. Demo
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. Mutators
    Boolean Substitution
    Name Original Mutated
    TrueValue true false
    FalseValue - +
    LogicalAnd && ||
    LogicalOr || &&
    

    View Slide

12. Mutators
    Literal Numbers
    Name Original Mutated
    OneZeroInteger 0 1
    ZeroOneInteger 1 0
    DecrementInteger 7 6
    IncrementInteger 7 8
    

    View Slide

13. Mutators
    Binary Arithmetic
    Name Original Mutated
    Plus + -
    Minus - +
    AssignmentEqual == =
    

    View Slide

14. Mutators
    Exceptions
    Name Original Mutated
    Throw_ throw new \Exception new \Exception
    Finally_ try {} catch {} finally {} try {} catch {}
    

    View Slide

15. Mutators
    Equal or Identical Checks
    Name Original Mutated
    EqualIdentical == ===
    NotEqualNotIdentical != !==
    IdenticalEqual === ==
    NotIdenticalNotEqal !== !=
    

    View Slide

16. Profiles
    {
    "source": {
    "directories": [
    "src"
    ]
    },
    "timeout": 10,
    "logs": {
    "text": "infection.log"
    },
    "mutators": {
    "@default": true,
    "@identical": true,
    }
    }
    

    View Slide

17. More Demo
    

    View Slide

18. Performance
    Only Changed Files
    CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ |
    paste -sd "," -);
    INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";
    infection --threads=4 $INFECTION_FILTER
    

    View Slide

19. vendor/bin/infection --only-covered
    Performance
    Only Covered
    

    View Slide

20. Performance
    Disable Filters
    {
    "mutators": {
    "@default": true,
    "Plus": false,
    }
    }
    

    View Slide

21. Summary
    Cons Pros
    easy setup
    false positives mutation shows what to test for
    "better" coverage
    slow runs in CI + provides coverage report
    

    View Slide