$30 off During Our Annual Pro Sale. View Details »

Mutation Testing PHPDeveloperDay

Théo FIDRY
September 22, 2018

Mutation Testing PHPDeveloperDay

Mutation testing: better code by making bugs Do you test your code? What about your tests? Your tests are code, you need to write, refactor and maintain them. This is not cheap so how do you make sure you are testing enough but not too much? Discover Mutation Testing, a fun tool to make your code better by introducing bugs.

Théo FIDRY

September 22, 2018
Tweet

More Decks by Théo FIDRY

Other Decks in Programming

Transcript

  1. MUTATION TESTING
    Better code by making bugs
    1

    View Slide

  2. ABOUT ME
    2
    @tfidry
    @theofidry
    Théo Fidry
    Web developer
    Barcelona (Spain)

    View Slide

  3. PROJECTS
    3
    Alice
    Humbug & Infection

    PHP-Scoper & Box
    API-Platform

    View Slide

  4. LET’S DO A
    SHOW OF HANDS
    4

    View Slide

  5. WHO DOES
    • UNIT TESTING
    • TEST-DRIVEN DEVELOPMENT
    • CONTINUOUS INTEGRATION
    • MEASURE CODE COVERAGE
    • MUTATION TESTING
    5

    View Slide

  6. HOW TO DEFINE
    SOFTWARE QUALITY?
    6

    View Slide

  7. 7
    Marcello Duarte
    Creator of PhpSpec
    ex-Head of Training @Inviqa
    The extent to what
    the software takes into account
    what matters most for the customer & the
    maintainability of the source code
    DEFINING SOFTWARE QUALITY
    Internal Quality
    External Quality
    https://speakerdeck.com/jakzal/building-in-quality

    View Slide

  8. 8
    (LACK OF) EXTERNAL QUALITY

    View Slide

  9. EXTERNAL QUALITY
    • Conformity to the user expectation
    • Reliability
    • Accuracy
    • Ergonomics
    • Design
    • …
    9

    View Slide

  10. THE QUALITY PERCEIVED
    BY THE USER
    10

    View Slide

  11. 11
    (LACK OF) INTERNAL QUALITY
    https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744

    View Slide

  12. INTERNAL QUALITY
    • Maintainability
    • Concision
    • Cohesion
    • Simplicity
    • Clarity
    • …
    12

    View Slide

  13. THE QUALITY PERCEIVED
    BY THE DEVELOPER
    (WHICH WE WILL FOCUS ON)
    13

    View Slide

  14. SHOULD WE CARE?
    14

    View Slide

  15. Cumulative functioncality
    Time
    15
    PROJECT STAMINA
    Payoff line
    https://martinfowler.com/bliki/DesignStaminaHypothesis.html

    View Slide

  16. THE LESS YOU CARE
    THE HARDER IT WILL BE
    TO ADD NEW FEATURES
    16

    View Slide

  17. COST OF BUG / PHASE
    17
    http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf
    Cost
    US$1,666.67
    US$3,333.33
    US$5,000.00
    US$6,666.67
    US$8,333.33
    US$10,000.00
    Development phase
    Dev CI/Review QA Prod
    US$100 US$500
    US$1,500
    US$10,000
    https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ

    View Slide

  18. ORIGIN OF THE COSTS
    • Implementation
    • Debug & Repair
    • Technical debt
    • Delay: the extra you are paying for not
    fixing bugs earlier on
    18

    View Slide

  19. ORIGIN OF THE COSTS
    19
    Technical debt
    Build cost
    Build cost
    Cost of delay
    Technical
    debt
    Cost of debug
    Cost of repair
    Right feature
    built wrong
    Right feature
    IDEA

    View Slide

  20. THE MORE YOU DELAY
    THE MORE EXPENSIVE IT GETS
    20

    View Slide

  21. YOU SHOULD CARE
    21

    View Slide

  22. HOW TO IMPROVE (INTERNAL) QUALITY?
    22
    (ONE) ANSWER: BY ADDING TESTS

    View Slide

  23. TESTS ARE CODE
    23

    View Slide

  24. TESTS ARE CODE
    • You need to write them
    • You need to make sure they work
    • You need to refactor them
    • You need to maintain them
    24

    View Slide

  25. TESTS ARE EXPENSIVE
    25
    NOT FREE

    View Slide

  26. PROBLEMS
    • How do I safely refactor my tests?
    • How do I know I can trust a test suite?
    • How do I ensure my team is writing
    effective tests
    • How do I know if I’ve retrofitted
    enough tests to safely refactor a piece
    of legacy code?
    26

    View Slide

  27. HOW DO I ASSESS THE QUALITY?
    27

    View Slide

  28. 28
    NO TEST MAX TEST
    SHORT-TERM
    HIGH VELOCITY
    SHORT-TERM
    LOW VELOCITY
    TESTS QUALITY
    Level of quality

    View Slide

  29. HOW DO I ASSESS THE QUALITY
    OF THE TEST SUITE?
    29

    View Slide

  30. COMMON ANSWERS
    • Don’t worry, it’ll be fine
    • I’m a ninja rockstar, I know my tests are good
    • I do TDD, I know my tests are good
    • What about the tests you didn’t write?
    • How do you test drive changes to tests?
    • Code review
    • Inconsistent + Labour intensive
    • Code coverage
    30

    View Slide

  31. LINE CODE COVERAGE MEASURE
    DOES NOT TELL YOU WHICH
    PART HAS BEEN TESTED
    31

    View Slide

  32. 32
    EXAMPLE
    https://github.com/theofidry/mutation-testing-demo
    namespace Acme;
    final class Counter
    {
    public $count = 0;
    public function count(int $nbr): void
    {
    if ($nbr >= 10) {
    $this->count++;
    }
    }
    }

    View Slide

  33. 33
    https://github.com/theofidry/mutation-testing-demo
    function test_it_counts_number_above_ten(): void
    {
    $initialCount = $this->counter->count;
    $this->counter->count(100);
    $newCount = $this->counter->count;
    Assert::assertGreaterThan($initialCount, $newCount);
    }
    function test_it_does_not_count_number_below_ten(): void
    {
    $initialCount = $this->counter->count;
    $this->counter->count(7);
    $newCount = $this->counter->count;
    Assert::assertSame($initialCount, $newCount);
    }

    View Slide

  34. 34
    RUN PHPUNIT WITH COVERAGE REPORT
    https://github.com/theofidry/mutation-testing-demo
    $ vendor/bin/phpunit --coverage-text \
    --coverage-html=dist/coverage
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    .. 2 / 2 (100%)
    Time: 113 ms, Memory: 6.00MB
    OK (2 tests, 2 assertions)
    Generating code coverage report in HTML format ... done
    Code Coverage Report:
    Summary:
    Classes: 100.00% (1/1)
    Methods: 100.00% (1/1)
    Lines: 100.00% (3/3)
    \Acme::Acme\Counter
    Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
    100% code coverage

    View Slide

  35. IS IT GOOD ENOUGH?
    35

    View Slide

  36. 36
    LET’S INTRODUCE A BUG
    https://github.com/theofidry/mutation-testing-demo
    namespace Acme;
    final class Counter
    {
    public $count = 0;
    public function count(int $nbr): void
    {
    if ($nbr > 10) {
    $this->count++;
    }
    }
    }

    View Slide

  37. 37
    RUN PHPUNIT WITH COVERAGE REPORT
    https://github.com/theofidry/mutation-testing-demo
    $ vendor/bin/phpunit --coverage-text \
    —-coverage-html=dist/coverage
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    .. 2 / 2 (100%)
    Time: 113 ms, Memory: 6.00MB
    OK (2 tests, 2 assertions)
    Generating code coverage report in HTML format ... done
    Code Coverage Report:
    Summary:
    Classes: 100.00% (1/1)
    Methods: 100.00% (1/1)
    Lines: 100.00% (3/3)
    \Acme::Acme\Counter
    Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
    DID NOT FAIL

    View Slide

  38. OUR TESTS STILL PASS.
    OUR TEST SUITE IS DEFICIENT
    38

    View Slide

  39. CODE COVERAGE MEASURE
    DOES NOT TELL YOU WHICH
    PART HAS BEEN TESTED
    39

    View Slide

  40. WHAT CODE COVERAGE DOES TELL YOU
    40

    View Slide

  41. EXECUTED ≠ TESTED
    Executed
    Tested
    41

    View Slide

  42. CODE COVERAGE TELLS YOU ONLY
    WHAT HAS NOT BEEN TESTED
    42

    View Slide

  43. 43
    A TEST CASE IS MISSING
    https://github.com/theofidry/mutation-testing-demo
    function test_it_counts_ten(): void
    {
    $initialCount = $this->counter->count;
    $this->counter->count(10);
    $newCount = $this->counter->count;
    Assert::assertGreaterThan($initialCount, $newCount);
    }

    View Slide

  44. 44
    $ vendor/bin/phpunit --coverage-text \
    --coverage-html=dist/coverage
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    .F. 3 / 3 (100%)
    Time: 70 ms, Memory: 6.00MB
    There was 1 failure:
    1) Acme\CounterTest::test_it_counts_ten
    Failed asserting that 0 is greater than 0.
    /path/to/tests/CounterTest.php:38
    FAILURES!
    Tests: 3, Assertions: 3, Failures: 1.
    Generating code coverage report in HTML format ... done
    Code Coverage Report:
    Summary:
    Classes: 100.00% (1/1)
    Methods: 100.00% (1/1)
    Lines: 100.00% (3/3)
    \Acme::Acme\Counter
    Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)

    View Slide

  45. HOW TO DETECT IF A TEST SUITE IS
    DEFICIENT?
    45

    View Slide

  46. INTRODUCE A BUG
    46

    View Slide

  47. MUTATION TESTING
    47

    View Slide

  48. CREATE A MUTANT
    SOURCE CODE
    MUTATOR
    MUTATION
    PROCESS
    MUTANT
    48

    View Slide

  49. EXAMPLE OF A MUTANT
    49
    namespace Acme;
    final class Counter
    {
    public $count = 0;
    public function count(int $nbr): void
    {
    if ($nbr >= 10) {
    $this->count++;
    }
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr > 10) {
    $this->count++;
    }
    }

    View Slide

  50. MUTATOR EXAMPLES
    50
    Name Original Mutated
    Plus + -
    GreaterThanOrEqualTo >= >
    Spaceship $a <=> $b $b <=> $a
    TrueValue return true; return false;
    https://infection.github.io/guide/mutators.html

    View Slide

  51. COLLECT THE SOURCE FILES
    51
    Counter.php
    Foo.php
    FILE
    COLLECTOR

    View Slide

  52. GENERATE MUTANTS
    52
    Counter.php
    Foo.php
    MUTATOR
    MUTATOR
    MUTATOR
    MUTATOR
    MUTATOR
    MUTATOR
    MUTANTS

    View Slide

  53. --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr > 9) {
    $this->count++;
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr >= 11) {
    $this->count++;
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr > 10) {
    $this->count++;
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    if ($nbr >= 10) {
    - $this->count++;
    + $this->count—-;
    }
    }
    GENERATED MUTANTS
    53
    … and more

    View Slide

  54. APPLY MUTANTS
    54
    PROCESS
    BUILDER
    MUTANT
    PROCESS
    WITH MUTATED
    CODE
    RESULT
    Runs tests
    TESTS
    RUNNER

    View Slide

  55. IF A MUTANT DOES NOT CAUSE THE
    TESTS TO FAIL, IT SURVIVED
    55

    View Slide

  56. IF A MUTANT DOES CAUSE THE
    TESTS TO FAIL, IT WAS KILLED
    56

    View Slide

  57. MUTATION SCORE
    Nbr of mutant killed
    Nbr of mutant generated
    Mutation score =
    57

    View Slide

  58. CODE COVERAGE HIGHLIGHTS CODE
    THAT IS DEFINITELY NOT TESTED
    MUTATION SCORE HIGHLIGHTS
    CODE THAT IS DEFINITELY TESTED
    58
    HOW TO DETECT IF A TEST SUITE IS
    DEFICIENT?

    View Slide

  59. DOES IT WORK?
    “Complex faults are coupled to simple
    faults in such a way that a test data set
    that detects all simple faults in a program
    will detect most complex faults”
    Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions”
    59

    View Slide

  60. MUTATION TESTING IN PHP
    infection/infection
    60
    humbug/humbug

    View Slide

  61. DEMO
    61

    View Slide

  62. MUTATION TESTING IN PHP
    infection/infection
    62
    symfony/dependency-injection

    View Slide

  63. SYMFONY/DEPENDENCY-INJECTION
    63

    View Slide

  64. 64
    INSTALLATION
    Infection installation guide: https://infection.github.io/guide/installation.html
    PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html
    $ git clone [email protected]:symfony/dependency-injection.git infection-demo
    $ cd infection-demo
    $ composer install
    $ wget -O infection https://github.com/infection/infection/releases/
    download/0.10.3/infection.phar
    $ chmod +x infection
    $ wget -O phpunit https://phar.phpunit.de/phpunit-7.phar
    $ chmod /+x phpunit

    View Slide

  65. PHPUNIT CONFIGURATION
    65
    phpunit.xml.dist

    xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/7.0/phpunit.xsd"
    bootstrap="vendor/autoload.php">





    ./Tests/




    ./

    ./Resources
    ./Tests
    ./vendor




    View Slide

  66. 66
    RUNNING TESTS
    $ ./phpunit
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    ............................................................... 63 / 672 ( 9%)
    ............................................................... 126 / 672 ( 18%)
    ............................................................... 189 / 672 ( 28%)
    ............................................................... 252 / 672 ( 37%)
    ............................................................... 315 / 672 ( 46%)
    ............................................................... 378 / 672 ( 56%)
    ............................................................... 441 / 672 ( 65%)
    ............................................................... 504 / 672 ( 75%)
    ........S.SSSSS................................................ 567 / 672 ( 84%)
    ............................................................... 630 / 672 ( 93%)
    .......................................... 672 / 672 (100%)
    Time: 1.89 seconds, Memory: 26.00MB
    OK, but incomplete, skipped, or risky tests!
    Tests: 672, Assertions: 1550, Skipped: 6.

    View Slide

  67. CONFIGURE INFECTION
    67
    $ ./infection init

    View Slide

  68. INFECTION CONFIGURATION
    68
    {
    "timeout": 10,
    "source": {
    "directories": [
    "."
    ],
    "excludes": [
    "Tests",
    "vendor"
    ]
    },
    "phpunit": {
    "customPath": "phpunit"
    },
    "logs": {
    "text": "infection-log.txt"
    }
    }
    infection.json.dist
    https://infection.github.io/guide/usage.html#Configuration

    View Slide

  69. 69
    RUNNING INFECTION
    $ ./infection
    You are running Infection with xdebug enabled.
    ____ ____ __ _
    / _/___ / __/__ _____/ /_(_)___ ____
    / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
    _/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
    /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
    Running initial test suite...
    PHPUnit version: 7.0.2
    678 [============================] 26 secs
    Generate mutants...
    Processing source code files: 144/144
    Creating mutated files and processes: 3573/3573
    .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
    .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573)
    E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573)
    ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)
    .M...........M...........................M.MM..... ( 200 / 3573)
    ....MM..M.......M.....M...SSSSSSSS...M......M.M... ( 250 / 3573)
    .M.M.MEM..MST.E.M.......MMMM............SMMM.....E ( 300 / 3573)
    .E.......TM.........SM....................EM...... ( 350 / 3573)
    ....EM.M...E..MM.MM.S.MS......M.....EM...E...EE... ( 400 / 3573)
    ...TME......S................MM....M........E...M. ( 450 / 3573)
    ...ESSSSSSSSS...MM.SM....ESSMMMM.MSS..MM.SM.M.MMMS ( 500 / 3573)
    SS.S......M........S.S....ME...M.......M...S.M..S. ( 550 / 3573)
    ...MS.......S.M....E.E..EET.M.M.....M..MM.S....... ( 600 / 3573)
    .E.S.T.E..S..MM..M.EM.E..E......S.MM...MM...M.M... ( 650 / 3573)
    .....M..M......EM..ESS.S.M.............M.....E..MS ( 700 / 3573)
    ....EM.........EE...........MMME.....E.M..MM..M... ( 750 / 3573)
    M.......EE...........M.......MMEMM..M...MM.M..M... ( 800 / 3573)
    ..M....................MMM...M...MM.........MM...M ( 850 / 3573)
    .SSM.......MM..M......E..SEMM....E.M..M.E..EM..ETM ( 900 / 3573)
    M.......EM.EM...SSSSSSSSSSSSSSSSSSSSSSSE.S.M...... ( 950 / 3573)
    .S..M.........MM........S......EE.E......E....EEE. (1000 / 3573)
    .S....SSM..E..S.E....TMSSSMM.........M.........E.S (1050 / 3573)
    EE..M....STM..M.......M.S......SM..........S.SM..S (1100 / 3573)
    ....MMMMM.MMM....M.....M.......M.SM...M.MMM.S..... (1150 / 3573)
    .E....E.M.E.....MM..MMMMMS.M........MMMMMM.......E (1200 / 3573)
    .EE....S.......................T......EEM.M.MM.... (1250 / 3573)
    ..SSMM.MM..MMMM.MMMMMMMM.......................... (1300 / 3573)
    ..E....E.......E..SSSSM..M.S.E..M............MMM.M (1350 / 3573)
    ......M.SMMM...............M...M.M......M.M.E..... (1400 / 3573)
    .MSM.........S.MMMMMMMMMM..........M.............. (1450 / 3573)
    ................M...............S..M.............. (1500 / 3573)
    ....M...............M.........M................M.. (1550 / 3573)
    ...........................M...................... (1600 / 3573)
    ..E......M...M...........M.MM.....SSSS..MMM..MM... (1650 / 3573)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMSMSE....M..MMM (1700 / 3573)
    M.M...M.MM........MM..M...M.SSSSSSSSSSSMM.MSSSSSSM (1750 / 3573)
    MMSSSSSSSSSSSSSSSSSSSSSSSS...MM............MMS.... (1800 / 3573)
    MMMMMM..S....M......MMM...SSSSSSSSSSSSSSSSSSSSSSSS (1850 / 3573)
    SSSSSSSM..S.....M........MS....................... (1900 / 3573)
    ............SSSS..MM.SSSSSSSSSSSSSSSSSSSSSSSS..... (1950 / 3573)
    .M.M..........SS.M.....SSSSS.....SSS.............. (2000 / 3573)
    .....M.S..SSSSSSS..SM...SSSSSSSS.MSM....MM.....M.. (2050 / 3573)
    ..SSSSSS..MMMSSSSSSSSSS..S.....M.MSSSS........SSSS (2100 / 3573)
    S....SSSSSSSS.MMM.SMSSSMM.M.......SSS...M....M..M. (2150 / 3573)
    MSM.MMS..MMM.M...SSS........M.......SS.S..SSSSSSSS (2200 / 3573)
    SSSSSSSSSSSSSSSSSS.S..MMMSSSSMS......MS....MM..SSS (2250 / 3573)
    SS...M........M.SSSSSSMM..MMM.MMM...M..MMTMTTSSSMM (2300 / 3573)
    S..S...........MSMSSS...SSSSSSSSSSSSSSSSSSSSSSSSSS (2350 / 3573)
    SSSSSSMMMMMSS.M.MMM.SSSSSSSSSSSSSSSSSSS.E......... (2400 / 3573)
    .................................................. (2450 / 3573)
    ......SS.....MS.......M.S.E...........M.M......... (2500 / 3573)
    ..M.M......M............SS.S.......M.......E.SSSSS (2550 / 3573)
    .................S....S.SS.SSS.S.SSSSS..M....SSSSS (2600 / 3573)
    E.SMMSMM....MMSMM......MSSSS.MMSS.....MMSSSS.SSSE. (2650 / 3573)
    ..S....EMMM.E......E.EEEM........SSSSSSEESSM..SSMS (2700 / 3573)
    ........SSSM.....MS.M.MM..SSM..E....M.SS.E...M..M. (2750 / 3573)
    .M.MM.......M..........MMMMMMM..S.....SSM..MM..... (2800 / 3573)
    .........M........SSS....EE....M........M...M...MS (2850 / 3573)
    .M.MM..MM..M.E...M....E...S.SE..M........S..MM.... (2900 / 3573)
    ...M...MS...S.....SS..E.EM......M.E.S.....ES...... (2950 / 3573)
    .....MMM......MM......M.......E.S.....EE.S.....E.. (3000 / 3573)
    M..M..........S.MM.....S............M........M.... (3050 / 3573)
    .........MMM.M.........M.....MM.....S.SMMS.S...... (3100 / 3573)
    ....M....M.M....S...M..M....SSSM...........M...... (3150 / 3573)
    .MME.S.M..M.E......M.........MSMMM.S.......MM.S..S (3200 / 3573)
    ..MSSSSMMM.SS..........................M.S........ (3250 / 3573)
    .....S............................S......SMMM..SSS (3300 / 3573)
    ..........M.MM.......M...MSS.S.S....MM.M...M...... (3350 / 3573)
    S.S...MSM........S......S.......MM........MM...... (3400 / 3573)
    ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573)
    .................MM.... (3573 / 3573)
    3573 mutations were generated:
    2322 mutants were killed
    569 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 69%
    Mutation Code Coverage: 84%
    Covered Code MSI: 82%
    Please note that some mutants will inevitably be harmless (i.e. fa
    14min 21s

    View Slide

  70. 70
    You are running Infection with xdebug enabled.
    $ ./infection
    ____ ____ __ _
    / _/___ / __/__ _____/ /_(_)___ ____
    / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
    _/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
    /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
    Running initial test suite...
    PHPUnit version: 7.0.2
    678 [============================] 26 secs
    Generate mutants...
    Processing source code files: 144/144
    Creating mutated files and processes: 3573/3573
    .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
    .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573)
    E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573)
    ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)

    View Slide

  71. 71
    ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573)
    .................MM.... (3573 / 3573)
    3573 mutations were generated:
    2322 mutants were killed
    569 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 69%
    Mutation Code Coverage: 84%
    Covered Code MSI: 82%
    Please note that some mutants will inevitably be harmless (i.e. false
    positives).

    View Slide

  72. 72
    Escaped mutants:
    ================
    1) /path/to/Alias.php:68 [M] This
    exec /path/to/php -c /tmp/dir/infectionyy4H8n /path/to/phpunit --configuration /tmp/dir/infection/
    phpunitConfiguration.bce05.infection.xml --stop-on-failure
    --- Original
    +++ New
    @@ @@
    $this->private = (bool) $boolean;
    - return $this;
    + return null;
    }
    /**
    * Whether this alias is private.
    *
    * @return bool
    */
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    ............................................................... 63 / 127 ( 49%)
    ............................................................... 126 / 127 ( 99%)
    . 127 / 127 (100%)
    Time: 374 ms, Memory: 14.00MB
    OK (127 tests, 438 assertions)
    REPORT

    View Slide

  73. TIPS
    73

    View Slide

  74. RESTRICT THE FILES IT’S RUNNING ON
    74
    $ infection —-filter=DependencyInjection
    INFECTION_FILTER=$(git diff ${GIT_PREVIOUS_SUCCESSFUL_COMMIT:-origin/develop} \
    $GIT_COMMIT --name-only \
    | grep /src/ \
    | paste -sd "," -)
    $ infection —-filter=$INFECTION_FILTER
    https://infection.github.io/guide/command-line-options.html
    $ infection —-min-covered-msi 70

    View Slide

  75. MAKE IT FASTER
    75
    $ infection —-threads=$(nproc)
    https://infection.github.io/guide/command-line-options.html
    $ infection —-only-covered
    Should this be the default?
    $(sysctl -n hw.ncpu)

    View Slide

  76. PROFILES
    76
    {
    "mutators": {
    "@default": true,
    "@function_signature": false,
    "TrueValue": {
    "ignore": [
    “NameSpace\\*\\SourceClass::create",
    "Full\\NameSpaced\\Class"
    ]
    }
    }
    }
    infection.json.dist
    https://infection.github.io/guide/profiles.html

    View Slide

  77. RUN IT AGAIN
    77
    $ infection --threads=$(sysctl -n hw.ncpu) --only-covered

    View Slide

  78. 78
    3573 mutations were generated:
    2322 mutants were killed
    569 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 69%
    Mutation Code Coverage: 84%
    Covered Code MSI: 82%
    Time: 14min 20s
    3004 mutations were generated:
    2322 mutants were killed
    0 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 82%
    Mutation Code Coverage: 100%
    Covered Code MSI: 82%
    Time: 6min 39s

    View Slide

  79. DEMO DONE
    79

    View Slide

  80. IT IS NOT NEW… - HISTORY
    • Begins in 1971, R. Lipton, “Fault
    Diagnosis of Computer Programs”
    • Generally accepted in 1978, R. Lipton
    and al, “Hints on test data selection:
    Help for the practicing programmer”
    80

    View Slide

  81. WHY IS IT NOT WIDELY USED?
    81
    http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this

    View Slide

  82. WHY IS IT NOT WIDELY USED?
    Maturity Problem:
    Because testing is not widely used yet
    (Although it is increasing)
    82

    View Slide

  83. WHY IS IT NOT WIDELY USED?
    Integration Problem:
    Inability to successfully integrated it into
    software development process
    (TDD plays a key role now)
    83

    View Slide

  84. WHY IS IT NOT WIDELY USED?
    Technical Problem:
    It is a brute force technique!
    84

    View Slide

  85. BRUTE FORCE TECHNIQUE
    N: Number of tests
    M: Number of mutants
    NxM
    85

    View Slide

  86. THEORETICAL RUN
    • 672 tests in 25.29 seconds
    • 3573 mutants
    • 2,401,056 tests
    • ~53h
    With basic Mutation Testing
    86

    View Slide

  87. 87

    View Slide

  88. OPTIMISATION STRATEGIES
    • Mutate only covered code
    • Incremental analysis
    • Parallelism
    • Equivalent mutant (~30%)
    • Different levels of requirements
    88

    View Slide

  89. OPTIMISATION STRATEGIES
    • 672 tests in 25.29 seconds
    • 3573 mutants
    • ~6min
    With Infection
    89

    View Slide

  90. INFECTION IS STILL YOUNG
    90

    View Slide

  91. FUTURE WORK
    91

    View Slide

  92. PHP FRAMEWORKS SUPPORT
    • PHPUnit
    • PhpSpec
    • Behat (#147)
    • Atoum (#381)
    • CodeCeption (#45)
    92


    View Slide

  93. EQUIVALENT MUTANT
    93
    - $x = $a + $b;
    + $x = $a - (-$b);
    - if ($nbr >= 10) {
    + if ($nbr > 10) {
    - if ($nbr >= 10) {
    + if ($nbr >= 9) {
    ~30% of
    mutants

    View Slide

  94. SMART MUTATORS
    94
    class X
    {
    function foo(string $a, string $b): boolean
    {
    - return $a === $b;
    + return $a == $b;
    }
    }

    View Slide

  95. {
    "mutators": {
    "@default": true,
    "@function_signature": false,
    "TrueValue": {
    "ignore": [
    “NameSpace\\*\\SourceClass::create",
    "Full\\NameSpaced\\Class"
    ]
    }
    }
    }
    infection.json.dist
    GRANULAR CONFIGURATION
    95

    View Slide

  96. FALSE POSITIVE TRACKING
    96

    View Slide

  97. IO-BOUND FRAMEWORK
    97

    View Slide

  98. WRAP UP
    98
    https://www.pinterest.com/pin/140526450845150336/

    View Slide

  99. THE GOOD PARTS
    • Gives you feedback on your tests
    • Test your tests with little effort
    • They are automatic
    • They discover dead code (e.g. useless tests)
    • Helps to refactor your tests
    99

    View Slide

  100. THE NOT SO GOOD PARTS
    • Can be slow
    • Handful of young libraries
    • Writing smart mutants is difficult
    • Side effects with integration tests
    100

    View Slide

  101. PERSONAL EXPERIENCE
    • Forces you to write tests
    • Always find bugs
    • Helpful for non-devs
    101

    View Slide

  102. 102
    USE IT

    View Slide

  103. MUTATION TESTING LIBRARIES
    HTTPS://GITHUB.COM/THEOFIDRY/AWESOME-MUTATION-TESTING
    103

    View Slide

  104. HTTPS://JOIND.IN/TALK/B4496
    104

    View Slide

  105. QUESTIONS?
    105
    https://imgflip.com/memegenerator/Shrek-Cat

    View Slide

  106. THANK YOU!
    106

    View Slide