Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Started with Mutation Testing

Getting Started with Mutation Testing

A quick introduction to mutation testing in PHP with infection. This talk covers how to set up infection in a project, how to read the reports it generates and why mutation coverage might be more helpful than line coverage provided by phpunit. The talk closes with tips on how to improve performance as mutation testing will produce a test matrix that inevitably will be slower than the test suite its based upon.

Denis Brumann

May 18, 2020
Tweet

More Decks by Denis Brumann

Other Decks in Programming

Transcript

  1. Getting
    Started With
    Mutation
    Testing
    Denis Brumann
    [email protected]

    View Slide

  2. Denis Brumann
    @dbrumann
    [email protected]

    View Slide

  3. Mutation testing involves modifying a program in small
    ways.
    If mutated program produces failing tests, this is called a
    killed mutant. If tests are green with mutated code, then
    we have an escaped mutant.

    View Slide

  4. Installation
    Phar, Phive, Composer, Git or Homebrew
    Infection requires a recent version of PHP, and
    XDebug, phpdbg, or pcov enabled.

    View Slide

  5. View Slide

  6. Demo

    View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. Mutators
    Boolean Substitution
    Name Original Mutated
    TrueValue true false
    FalseValue - +
    LogicalAnd && ||
    LogicalOr || &&

    View Slide

  12. Mutators
    Literal Numbers
    Name Original Mutated
    OneZeroInteger 0 1
    ZeroOneInteger 1 0
    DecrementInteger 7 6
    IncrementInteger 7 8

    View Slide

  13. Mutators
    Binary Arithmetic
    Name Original Mutated
    Plus + -
    Minus - +
    AssignmentEqual == =

    View Slide

  14. Mutators
    Exceptions
    Name Original Mutated
    Throw_ throw new \Exception new \Exception
    Finally_ try {} catch {} finally {} try {} catch {}

    View Slide

  15. Mutators
    Equal or Identical Checks
    Name Original Mutated
    EqualIdentical == ===
    NotEqualNotIdentical != !==
    IdenticalEqual === ==
    NotIdenticalNotEqal !== !=

    View Slide

  16. Profiles
    {
    "source": {
    "directories": [
    "src"
    ]
    },
    "timeout": 10,
    "logs": {
    "text": "infection.log"
    },
    "mutators": {
    "@default": true,
    "@identical": true,
    }
    }

    View Slide

  17. More Demo

    View Slide

  18. Performance
    Only Changed Files
    CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ |
    paste -sd "," -);
    INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";
    infection --threads=4 $INFECTION_FILTER

    View Slide

  19. vendor/bin/infection --only-covered
    Performance
    Only Covered

    View Slide

  20. Performance
    Disable Filters
    {
    "mutators": {
    "@default": true,
    "Plus": false,
    }
    }

    View Slide

  21. Summary
    Cons Pros
    easy setup
    false positives mutation shows what to test for
    "better" coverage
    slow runs in CI + provides coverage report

    View Slide