Getting Started with Mutation Testing

Getting Started with Mutation Testing

A quick introduction to mutation testing in PHP with infection. This talk covers how to set up infection in a project, how to read the reports it generates and why mutation coverage might be more helpful than line coverage provided by phpunit. The talk closes with tips on how to improve performance as mutation testing will produce a test matrix that inevitably will be slower than the test suite its based upon.

6a1345d8e6dd15b2c78eff0c331963b1?s=128

Denis Brumann

May 18, 2020
Tweet

Transcript

  1. 3.

    Mutation testing involves modifying a program in small ways. If

    mutated program produces failing tests, this is called a killed mutant. If tests are green with mutated code, then we have an escaped mutant.
  2. 4.

    Installation Phar, Phive, Composer, Git or Homebrew Infection requires a

    recent version of PHP, and XDebug, phpdbg, or pcov enabled.
  3. 5.
  4. 6.
  5. 7.
  6. 8.
  7. 9.
  8. 10.
  9. 14.

    Mutators Exceptions Name Original Mutated Throw_ throw new \Exception new

    \Exception Finally_ try {} catch {} finally {} try {} catch {}
  10. 15.

    Mutators Equal or Identical Checks Name Original Mutated EqualIdentical ==

    === NotEqualNotIdentical != !== IdenticalEqual === == NotIdenticalNotEqal !== !=
  11. 16.

    Profiles { "source": { "directories": [ "src" ] }, "timeout":

    10, "logs": { "text": "infection.log" }, "mutators": { "@default": true, "@identical": true, } }
  12. 17.
  13. 18.

    Performance Only Changed Files CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only |

    grep src/ | paste -sd "," -); INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations"; infection --threads=4 $INFECTION_FILTER
  14. 21.

    Summary Cons Pros easy setup false positives mutation shows what

    to test for "better" coverage slow runs in CI + provides coverage report