Getting Started with Mutation Testing

A quick introduction to mutation testing in PHP with infection. This talk covers how to set up infection in a project, how to read the reports it generates and why mutation coverage might be more helpful than line coverage provided by phpunit. The talk closes with tips on how to improve performance as mutation testing will produce a test matrix that inevitably will be slower than the test suite its based upon.

Transcript

1. Getting Started With Mutation Testing Denis Brumann denis.brumann@sensiolabs.de
2. Denis Brumann @dbrumann denis.brumann@sensiolabs.de
3. Mutation testing involves modifying a program in small ways. If mutated program produces failing tests, this is called a killed mutant. If tests are green with mutated code, then we have an escaped mutant.
4. Installation Phar, Phive, Composer, Git or Homebrew Infection requires a recent version of PHP, and XDebug, phpdbg, or pcov enabled.
6. Demo
11. Mutators Boolean Substitution Name Original Mutated TrueValue true false FalseValue - + LogicalAnd && || LogicalOr || &&
12. Mutators Literal Numbers Name Original Mutated OneZeroInteger 0 1 ZeroOneInteger 1 0 DecrementInteger 7 6 IncrementInteger 7 8
13. Mutators Binary Arithmetic Name Original Mutated Plus + - Minus - + AssignmentEqual == =
14. Mutators Exceptions Name Original Mutated Throw_ throw new \Exception new \Exception Finally_ try {} catch {} finally {} try {} catch {}
15. Mutators Equal or Identical Checks Name Original Mutated EqualIdentical == === NotEqualNotIdentical != !== IdenticalEqual === == NotIdenticalNotEqal !== !=
16. Profiles { "source": { "directories": [ "src" ] }, "timeout": 10, "logs": { "text": "infection.log" }, "mutators": { "@default": true, "@identical": true, } }
17. More Demo
18. Performance Only Changed Files CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -); INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations"; infection --threads=4 $INFECTION_FILTER
19. vendor/bin/infection --only-covered Performance Only Covered
20. Performance Disable Filters { "mutators": { "@default": true, "Plus": false, } }
21. Summary Cons Pros easy setup false positives mutation shows what to test for "better" coverage slow runs in CI + provides coverage report