Getting Started with Mutation Testing

Getting Started with Mutation Testing

A quick introduction to mutation testing in PHP with infection. This talk covers how to set up infection in a project, how to read the reports it generates and why mutation coverage might be more helpful than line coverage provided by phpunit. The talk closes with tips on how to improve performance as mutation testing will produce a test matrix that inevitably will be slower than the test suite its based upon.

6a1345d8e6dd15b2c78eff0c331963b1?s=128

Denis Brumann

May 18, 2020
Tweet

Transcript

  1. Getting Started With Mutation Testing Denis Brumann denis.brumann@sensiolabs.de

  2. Denis Brumann @dbrumann denis.brumann@sensiolabs.de

  3. Mutation testing involves modifying a program in small ways. If

    mutated program produces failing tests, this is called a killed mutant. If tests are green with mutated code, then we have an escaped mutant.
  4. Installation Phar, Phive, Composer, Git or Homebrew Infection requires a

    recent version of PHP, and XDebug, phpdbg, or pcov enabled.
  5. None
  6. Demo

  7. None
  8. None
  9. None
  10. None
  11. Mutators Boolean Substitution Name Original Mutated TrueValue true false FalseValue

    - + LogicalAnd && || LogicalOr || &&
  12. Mutators Literal Numbers Name Original Mutated OneZeroInteger 0 1 ZeroOneInteger

    1 0 DecrementInteger 7 6 IncrementInteger 7 8
  13. Mutators Binary Arithmetic Name Original Mutated Plus + - Minus

    - + AssignmentEqual == =
  14. Mutators Exceptions Name Original Mutated Throw_ throw new \Exception new

    \Exception Finally_ try {} catch {} finally {} try {} catch {}
  15. Mutators Equal or Identical Checks Name Original Mutated EqualIdentical ==

    === NotEqualNotIdentical != !== IdenticalEqual === == NotIdenticalNotEqal !== !=
  16. Profiles { "source": { "directories": [ "src" ] }, "timeout":

    10, "logs": { "text": "infection.log" }, "mutators": { "@default": true, "@identical": true, } }
  17. More Demo

  18. Performance Only Changed Files CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only |

    grep src/ | paste -sd "," -); INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations"; infection --threads=4 $INFECTION_FILTER
  19. vendor/bin/infection --only-covered Performance Only Covered

  20. Performance Disable Filters { "mutators": { "@default": true, "Plus": false,

    } }
  21. Summary Cons Pros easy setup false positives mutation shows what

    to test for "better" coverage slow runs in CI + provides coverage report