Mutation Testing PHPDeveloperDay

Mutation testing: better code by making bugs Do you test your code? What about your tests? Your tests are code, you need to write, refactor and maintain them. This is not cheap so how do you make sure you are testing enough but not too much? Discover Mutation Testing, a fun tool to make your code better by introducing bugs.

Transcript

1. MUTATION TESTING
   Better code by making bugs
   1
   

   View Slide

2. ABOUT ME
   2
   @tfidry
   @theofidry
   Théo Fidry
   Web developer
   Barcelona (Spain)
   

   View Slide

3. PROJECTS
   3
   Alice
   Humbug & Infection

   PHP-Scoper & Box
   API-Platform
   

   View Slide

4. LET’S DO A
   SHOW OF HANDS
   4
   

   View Slide

5. WHO DOES
   • UNIT TESTING
   • TEST-DRIVEN DEVELOPMENT
   • CONTINUOUS INTEGRATION
   • MEASURE CODE COVERAGE
   • MUTATION TESTING
   5
   

   View Slide

6. HOW TO DEFINE
   SOFTWARE QUALITY?
   6
   

   View Slide

7. 7
   Marcello Duarte
   Creator of PhpSpec
   ex-Head of Training @Inviqa
   The extent to what
   the software takes into account
   what matters most for the customer & the
   maintainability of the source code
   DEFINING SOFTWARE QUALITY
   Internal Quality
   External Quality
   https://speakerdeck.com/jakzal/building-in-quality
   

   View Slide

8. 8
   (LACK OF) EXTERNAL QUALITY
   

   View Slide

9. EXTERNAL QUALITY
   • Conformity to the user expectation
   • Reliability
   • Accuracy
   • Ergonomics
   • Design
   • …
   9
   

   View Slide

10. THE QUALITY PERCEIVED
    BY THE USER
    10
    

    View Slide

11. 11
    (LACK OF) INTERNAL QUALITY
    https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744
    

    View Slide

12. INTERNAL QUALITY
    • Maintainability
    • Concision
    • Cohesion
    • Simplicity
    • Clarity
    • …
    12
    

    View Slide

13. THE QUALITY PERCEIVED
    BY THE DEVELOPER
    (WHICH WE WILL FOCUS ON)
    13
    

    View Slide

14. SHOULD WE CARE?
    14
    

    View Slide

15. Cumulative functioncality
    Time
    15
    PROJECT STAMINA
    Payoff line
    https://martinfowler.com/bliki/DesignStaminaHypothesis.html
    

    View Slide

16. THE LESS YOU CARE
    THE HARDER IT WILL BE
    TO ADD NEW FEATURES
    16
    

    View Slide

17. COST OF BUG / PHASE
    17
    http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf
    Cost
    US$1,666.67
    US$3,333.33
    US$5,000.00
    US$6,666.67
    US$8,333.33
    US$10,000.00
    Development phase
    Dev CI/Review QA Prod
    US$100 US$500
    US$1,500
    US$10,000
    https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ
    

    View Slide

18. ORIGIN OF THE COSTS
    • Implementation
    • Debug & Repair
    • Technical debt
    • Delay: the extra you are paying for not
    fixing bugs earlier on
    18
    

    View Slide

19. ORIGIN OF THE COSTS
    19
    Technical debt
    Build cost
    Build cost
    Cost of delay
    Technical
    debt
    Cost of debug
    Cost of repair
    Right feature
    built wrong
    Right feature
    IDEA
    

    View Slide

20. THE MORE YOU DELAY
    THE MORE EXPENSIVE IT GETS
    20
    

    View Slide

21. YOU SHOULD CARE
    21
    

    View Slide

22. HOW TO IMPROVE (INTERNAL) QUALITY?
    22
    (ONE) ANSWER: BY ADDING TESTS
    

    View Slide

23. TESTS ARE CODE
    23
    

    View Slide

24. TESTS ARE CODE
    • You need to write them
    • You need to make sure they work
    • You need to refactor them
    • You need to maintain them
    24
    

    View Slide

25. TESTS ARE EXPENSIVE
    25
    NOT FREE
    

    View Slide

26. PROBLEMS
    • How do I safely refactor my tests?
    • How do I know I can trust a test suite?
    • How do I ensure my team is writing
    effective tests
    • How do I know if I’ve retrofitted
    enough tests to safely refactor a piece
    of legacy code?
    26
    

    View Slide

27. HOW DO I ASSESS THE QUALITY?
    27
    

    View Slide

28. 28
    NO TEST MAX TEST
    SHORT-TERM
    HIGH VELOCITY
    SHORT-TERM
    LOW VELOCITY
    TESTS QUALITY
    Level of quality
    

    View Slide

29. HOW DO I ASSESS THE QUALITY
    OF THE TEST SUITE?
    29
    

    View Slide

30. COMMON ANSWERS
    • Don’t worry, it’ll be fine
    • I’m a ninja rockstar, I know my tests are good
    • I do TDD, I know my tests are good
    • What about the tests you didn’t write?
    • How do you test drive changes to tests?
    • Code review
    • Inconsistent + Labour intensive
    • Code coverage
    30
    

    View Slide

31. LINE CODE COVERAGE MEASURE
    DOES NOT TELL YOU WHICH
    PART HAS BEEN TESTED
    31
    

    View Slide

32. 32
    EXAMPLE
    https://github.com/theofidry/mutation-testing-demo
    namespace Acme;
    final class Counter
    {
    public $count = 0;
    public function count(int $nbr): void
    {
    if ($nbr >= 10) {
    $this->count++;
    }
    }
    }
    

    View Slide

33. 33
    https://github.com/theofidry/mutation-testing-demo
    function test_it_counts_number_above_ten(): void
    {
    $initialCount = $this->counter->count;
    $this->counter->count(100);
    $newCount = $this->counter->count;
    Assert::assertGreaterThan($initialCount, $newCount);
    }
    function test_it_does_not_count_number_below_ten(): void
    {
    $initialCount = $this->counter->count;
    $this->counter->count(7);
    $newCount = $this->counter->count;
    Assert::assertSame($initialCount, $newCount);
    }
    

    View Slide

34. 34
    RUN PHPUNIT WITH COVERAGE REPORT
    https://github.com/theofidry/mutation-testing-demo
    $ vendor/bin/phpunit --coverage-text \
    --coverage-html=dist/coverage
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    .. 2 / 2 (100%)
    Time: 113 ms, Memory: 6.00MB
    OK (2 tests, 2 assertions)
    Generating code coverage report in HTML format ... done
    Code Coverage Report:
    Summary:
    Classes: 100.00% (1/1)
    Methods: 100.00% (1/1)
    Lines: 100.00% (3/3)
    \Acme::Acme\Counter
    Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
    100% code coverage
    

    View Slide

35. IS IT GOOD ENOUGH?
    35
    

    View Slide

36. 36
    LET’S INTRODUCE A BUG
    https://github.com/theofidry/mutation-testing-demo
    namespace Acme;
    final class Counter
    {
    public $count = 0;
    public function count(int $nbr): void
    {
    if ($nbr > 10) {
    $this->count++;
    }
    }
    }
    

    View Slide

37. 37
    RUN PHPUNIT WITH COVERAGE REPORT
    https://github.com/theofidry/mutation-testing-demo
    $ vendor/bin/phpunit --coverage-text \
    —-coverage-html=dist/coverage
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    .. 2 / 2 (100%)
    Time: 113 ms, Memory: 6.00MB
    OK (2 tests, 2 assertions)
    Generating code coverage report in HTML format ... done
    Code Coverage Report:
    Summary:
    Classes: 100.00% (1/1)
    Methods: 100.00% (1/1)
    Lines: 100.00% (3/3)
    \Acme::Acme\Counter
    Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
    DID NOT FAIL
    

    View Slide

38. OUR TESTS STILL PASS.
    OUR TEST SUITE IS DEFICIENT
    38
    

    View Slide

39. CODE COVERAGE MEASURE
    DOES NOT TELL YOU WHICH
    PART HAS BEEN TESTED
    39
    

    View Slide

40. WHAT CODE COVERAGE DOES TELL YOU
    40
    

    View Slide

41. EXECUTED ≠ TESTED
    Executed
    Tested
    41
    

    View Slide

42. CODE COVERAGE TELLS YOU ONLY
    WHAT HAS NOT BEEN TESTED
    42
    

    View Slide

43. 43
    A TEST CASE IS MISSING
    https://github.com/theofidry/mutation-testing-demo
    function test_it_counts_ten(): void
    {
    $initialCount = $this->counter->count;
    $this->counter->count(10);
    $newCount = $this->counter->count;
    Assert::assertGreaterThan($initialCount, $newCount);
    }
    

    View Slide

44. 44
    $ vendor/bin/phpunit --coverage-text \
    --coverage-html=dist/coverage
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    .F. 3 / 3 (100%)
    Time: 70 ms, Memory: 6.00MB
    There was 1 failure:
    1) Acme\CounterTest::test_it_counts_ten
    Failed asserting that 0 is greater than 0.
    /path/to/tests/CounterTest.php:38
    FAILURES!
    Tests: 3, Assertions: 3, Failures: 1.
    Generating code coverage report in HTML format ... done
    Code Coverage Report:
    Summary:
    Classes: 100.00% (1/1)
    Methods: 100.00% (1/1)
    Lines: 100.00% (3/3)
    \Acme::Acme\Counter
    Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
    

    View Slide

45. HOW TO DETECT IF A TEST SUITE IS
    DEFICIENT?
    45
    

    View Slide

46. INTRODUCE A BUG
    46
    

    View Slide

47. MUTATION TESTING
    47
    

    View Slide

48. CREATE A MUTANT
    SOURCE CODE
    MUTATOR
    MUTATION
    PROCESS
    MUTANT
    48
    

    View Slide

49. EXAMPLE OF A MUTANT
    49
    namespace Acme;
    final class Counter
    {
    public $count = 0;
    public function count(int $nbr): void
    {
    if ($nbr >= 10) {
    $this->count++;
    }
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr > 10) {
    $this->count++;
    }
    }
    

    View Slide

50. MUTATOR EXAMPLES
    50
    Name Original Mutated
    Plus + -
    GreaterThanOrEqualTo >= >
    Spaceship $a <=> $b $b <=> $a
    TrueValue return true; return false;
    https://infection.github.io/guide/mutators.html
    

    View Slide

51. COLLECT THE SOURCE FILES
    51
    Counter.php
    Foo.php
    FILE
    COLLECTOR
    

    View Slide

52. GENERATE MUTANTS
    52
    Counter.php
    Foo.php
    MUTATOR
    MUTATOR
    MUTATOR
    MUTATOR
    MUTATOR
    MUTATOR
    MUTANTS
    

    View Slide

53. --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr > 9) {
    $this->count++;
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr >= 11) {
    $this->count++;
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    - if ($nbr >= 10) {
    + if ($nbr > 10) {
    $this->count++;
    }
    }
    --- a/src/Counter.php
    +++ b/src/Counter.php
    @@ -8,7 +8,7 @@ final class Counter
    public function count(int $nbr): void
    {
    if ($nbr >= 10) {
    - $this->count++;
    + $this->count—-;
    }
    }
    GENERATED MUTANTS
    53
    … and more
    

    View Slide

54. APPLY MUTANTS
    54
    PROCESS
    BUILDER
    MUTANT
    PROCESS
    WITH MUTATED
    CODE
    RESULT
    Runs tests
    TESTS
    RUNNER
    

    View Slide

55. IF A MUTANT DOES NOT CAUSE THE
    TESTS TO FAIL, IT SURVIVED
    55
    

    View Slide

56. IF A MUTANT DOES CAUSE THE
    TESTS TO FAIL, IT WAS KILLED
    56
    

    View Slide

57. MUTATION SCORE
    Nbr of mutant killed
    Nbr of mutant generated
    Mutation score =
    57
    

    View Slide

58. CODE COVERAGE HIGHLIGHTS CODE
    THAT IS DEFINITELY NOT TESTED
    MUTATION SCORE HIGHLIGHTS
    CODE THAT IS DEFINITELY TESTED
    58
    HOW TO DETECT IF A TEST SUITE IS
    DEFICIENT?
    

    View Slide

59. DOES IT WORK?
    “Complex faults are coupled to simple
    faults in such a way that a test data set
    that detects all simple faults in a program
    will detect most complex faults”
    Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions”
    59
    

    View Slide

60. MUTATION TESTING IN PHP
    infection/infection
    60
    humbug/humbug
    

    View Slide

61. DEMO
    61
    

    View Slide

62. MUTATION TESTING IN PHP
    infection/infection
    62
    symfony/dependency-injection
    

    View Slide

63. SYMFONY/DEPENDENCY-INJECTION
    63
    

    View Slide

64. 64
    INSTALLATION
    Infection installation guide: https://infection.github.io/guide/installation.html
    PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html
    $ git clone git@github.com:symfony/dependency-injection.git infection-demo
    $ cd infection-demo
    $ composer install
    $ wget -O infection https://github.com/infection/infection/releases/
    download/0.10.3/infection.phar
    $ chmod +x infection
    $ wget -O phpunit https://phar.phpunit.de/phpunit-7.phar
    $ chmod /+x phpunit
    

    View Slide

65. PHPUNIT CONFIGURATION
    65
    phpunit.xml.dist
    
    xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/7.0/phpunit.xsd"
    bootstrap="vendor/autoload.php">
    
    
    
    
    
    ./Tests/
    
    
    
    
    ./
    
    ./Resources
    ./Tests
    ./vendor
    
    
    
    
    

    View Slide

66. 66
    RUNNING TESTS
    $ ./phpunit
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    ............................................................... 63 / 672 ( 9%)
    ............................................................... 126 / 672 ( 18%)
    ............................................................... 189 / 672 ( 28%)
    ............................................................... 252 / 672 ( 37%)
    ............................................................... 315 / 672 ( 46%)
    ............................................................... 378 / 672 ( 56%)
    ............................................................... 441 / 672 ( 65%)
    ............................................................... 504 / 672 ( 75%)
    ........S.SSSSS................................................ 567 / 672 ( 84%)
    ............................................................... 630 / 672 ( 93%)
    .......................................... 672 / 672 (100%)
    Time: 1.89 seconds, Memory: 26.00MB
    OK, but incomplete, skipped, or risky tests!
    Tests: 672, Assertions: 1550, Skipped: 6.
    

    View Slide

67. CONFIGURE INFECTION
    67
    $ ./infection init
    

    View Slide

68. INFECTION CONFIGURATION
    68
    {
    "timeout": 10,
    "source": {
    "directories": [
    "."
    ],
    "excludes": [
    "Tests",
    "vendor"
    ]
    },
    "phpunit": {
    "customPath": "phpunit"
    },
    "logs": {
    "text": "infection-log.txt"
    }
    }
    infection.json.dist
    https://infection.github.io/guide/usage.html#Configuration
    

    View Slide

69. 69
    RUNNING INFECTION
    $ ./infection
    You are running Infection with xdebug enabled.
    ____ ____ __ _
    / _/___ / __/__ _____/ /_(_)___ ____
    / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
    _/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
    /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
    Running initial test suite...
    PHPUnit version: 7.0.2
    678 [============================] 26 secs
    Generate mutants...
    Processing source code files: 144/144
    Creating mutated files and processes: 3573/3573
    .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
    .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573)
    E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573)
    ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)
    .M...........M...........................M.MM..... ( 200 / 3573)
    ....MM..M.......M.....M...SSSSSSSS...M......M.M... ( 250 / 3573)
    .M.M.MEM..MST.E.M.......MMMM............SMMM.....E ( 300 / 3573)
    .E.......TM.........SM....................EM...... ( 350 / 3573)
    ....EM.M...E..MM.MM.S.MS......M.....EM...E...EE... ( 400 / 3573)
    ...TME......S................MM....M........E...M. ( 450 / 3573)
    ...ESSSSSSSSS...MM.SM....ESSMMMM.MSS..MM.SM.M.MMMS ( 500 / 3573)
    SS.S......M........S.S....ME...M.......M...S.M..S. ( 550 / 3573)
    ...MS.......S.M....E.E..EET.M.M.....M..MM.S....... ( 600 / 3573)
    .E.S.T.E..S..MM..M.EM.E..E......S.MM...MM...M.M... ( 650 / 3573)
    .....M..M......EM..ESS.S.M.............M.....E..MS ( 700 / 3573)
    ....EM.........EE...........MMME.....E.M..MM..M... ( 750 / 3573)
    M.......EE...........M.......MMEMM..M...MM.M..M... ( 800 / 3573)
    ..M....................MMM...M...MM.........MM...M ( 850 / 3573)
    .SSM.......MM..M......E..SEMM....E.M..M.E..EM..ETM ( 900 / 3573)
    M.......EM.EM...SSSSSSSSSSSSSSSSSSSSSSSE.S.M...... ( 950 / 3573)
    .S..M.........MM........S......EE.E......E....EEE. (1000 / 3573)
    .S....SSM..E..S.E....TMSSSMM.........M.........E.S (1050 / 3573)
    EE..M....STM..M.......M.S......SM..........S.SM..S (1100 / 3573)
    ....MMMMM.MMM....M.....M.......M.SM...M.MMM.S..... (1150 / 3573)
    .E....E.M.E.....MM..MMMMMS.M........MMMMMM.......E (1200 / 3573)
    .EE....S.......................T......EEM.M.MM.... (1250 / 3573)
    ..SSMM.MM..MMMM.MMMMMMMM.......................... (1300 / 3573)
    ..E....E.......E..SSSSM..M.S.E..M............MMM.M (1350 / 3573)
    ......M.SMMM...............M...M.M......M.M.E..... (1400 / 3573)
    .MSM.........S.MMMMMMMMMM..........M.............. (1450 / 3573)
    ................M...............S..M.............. (1500 / 3573)
    ....M...............M.........M................M.. (1550 / 3573)
    ...........................M...................... (1600 / 3573)
    ..E......M...M...........M.MM.....SSSS..MMM..MM... (1650 / 3573)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMSMSE....M..MMM (1700 / 3573)
    M.M...M.MM........MM..M...M.SSSSSSSSSSSMM.MSSSSSSM (1750 / 3573)
    MMSSSSSSSSSSSSSSSSSSSSSSSS...MM............MMS.... (1800 / 3573)
    MMMMMM..S....M......MMM...SSSSSSSSSSSSSSSSSSSSSSSS (1850 / 3573)
    SSSSSSSM..S.....M........MS....................... (1900 / 3573)
    ............SSSS..MM.SSSSSSSSSSSSSSSSSSSSSSSS..... (1950 / 3573)
    .M.M..........SS.M.....SSSSS.....SSS.............. (2000 / 3573)
    .....M.S..SSSSSSS..SM...SSSSSSSS.MSM....MM.....M.. (2050 / 3573)
    ..SSSSSS..MMMSSSSSSSSSS..S.....M.MSSSS........SSSS (2100 / 3573)
    S....SSSSSSSS.MMM.SMSSSMM.M.......SSS...M....M..M. (2150 / 3573)
    MSM.MMS..MMM.M...SSS........M.......SS.S..SSSSSSSS (2200 / 3573)
    SSSSSSSSSSSSSSSSSS.S..MMMSSSSMS......MS....MM..SSS (2250 / 3573)
    SS...M........M.SSSSSSMM..MMM.MMM...M..MMTMTTSSSMM (2300 / 3573)
    S..S...........MSMSSS...SSSSSSSSSSSSSSSSSSSSSSSSSS (2350 / 3573)
    SSSSSSMMMMMSS.M.MMM.SSSSSSSSSSSSSSSSSSS.E......... (2400 / 3573)
    .................................................. (2450 / 3573)
    ......SS.....MS.......M.S.E...........M.M......... (2500 / 3573)
    ..M.M......M............SS.S.......M.......E.SSSSS (2550 / 3573)
    .................S....S.SS.SSS.S.SSSSS..M....SSSSS (2600 / 3573)
    E.SMMSMM....MMSMM......MSSSS.MMSS.....MMSSSS.SSSE. (2650 / 3573)
    ..S....EMMM.E......E.EEEM........SSSSSSEESSM..SSMS (2700 / 3573)
    ........SSSM.....MS.M.MM..SSM..E....M.SS.E...M..M. (2750 / 3573)
    .M.MM.......M..........MMMMMMM..S.....SSM..MM..... (2800 / 3573)
    .........M........SSS....EE....M........M...M...MS (2850 / 3573)
    .M.MM..MM..M.E...M....E...S.SE..M........S..MM.... (2900 / 3573)
    ...M...MS...S.....SS..E.EM......M.E.S.....ES...... (2950 / 3573)
    .....MMM......MM......M.......E.S.....EE.S.....E.. (3000 / 3573)
    M..M..........S.MM.....S............M........M.... (3050 / 3573)
    .........MMM.M.........M.....MM.....S.SMMS.S...... (3100 / 3573)
    ....M....M.M....S...M..M....SSSM...........M...... (3150 / 3573)
    .MME.S.M..M.E......M.........MSMMM.S.......MM.S..S (3200 / 3573)
    ..MSSSSMMM.SS..........................M.S........ (3250 / 3573)
    .....S............................S......SMMM..SSS (3300 / 3573)
    ..........M.MM.......M...MSS.S.S....MM.M...M...... (3350 / 3573)
    S.S...MSM........S......S.......MM........MM...... (3400 / 3573)
    ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573)
    .................MM.... (3573 / 3573)
    3573 mutations were generated:
    2322 mutants were killed
    569 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 69%
    Mutation Code Coverage: 84%
    Covered Code MSI: 82%
    Please note that some mutants will inevitably be harmless (i.e. fa
    14min 21s
    

    View Slide

70. 70
    You are running Infection with xdebug enabled.
    $ ./infection
    ____ ____ __ _
    / _/___ / __/__ _____/ /_(_)___ ____
    / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
    _/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
    /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
    Running initial test suite...
    PHPUnit version: 7.0.2
    678 [============================] 26 secs
    Generate mutants...
    Processing source code files: 144/144
    Creating mutated files and processes: 3573/3573
    .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
    .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573)
    E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573)
    ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)
    

    View Slide

71. 71
    ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573)
    .................MM.... (3573 / 3573)
    3573 mutations were generated:
    2322 mutants were killed
    569 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 69%
    Mutation Code Coverage: 84%
    Covered Code MSI: 82%
    Please note that some mutants will inevitably be harmless (i.e. false
    positives).
    

    View Slide

72. 72
    Escaped mutants:
    ================
    1) /path/to/Alias.php:68 [M] This
    exec /path/to/php -c /tmp/dir/infectionyy4H8n /path/to/phpunit --configuration /tmp/dir/infection/
    phpunitConfiguration.bce05.infection.xml --stop-on-failure
    --- Original
    +++ New
    @@ @@
    $this->private = (bool) $boolean;
    - return $this;
    + return null;
    }
    /**
    * Whether this alias is private.
    *
    * @return bool
    */
    PHPUnit 7.0.2 by Sebastian Bergmann and contributors.
    ............................................................... 63 / 127 ( 49%)
    ............................................................... 126 / 127 ( 99%)
    . 127 / 127 (100%)
    Time: 374 ms, Memory: 14.00MB
    OK (127 tests, 438 assertions)
    REPORT
    

    View Slide

73. TIPS
    73
    

    View Slide

74. RESTRICT THE FILES IT’S RUNNING ON
    74
    $ infection —-filter=DependencyInjection
    INFECTION_FILTER=$(git diff ${GIT_PREVIOUS_SUCCESSFUL_COMMIT:-origin/develop} \
    $GIT_COMMIT --name-only \
    | grep /src/ \
    | paste -sd "," -)
    $ infection —-filter=$INFECTION_FILTER
    https://infection.github.io/guide/command-line-options.html
    $ infection —-min-covered-msi 70
    

    View Slide

75. MAKE IT FASTER
    75
    $ infection —-threads=$(nproc)
    https://infection.github.io/guide/command-line-options.html
    $ infection —-only-covered
    Should this be the default?
    $(sysctl -n hw.ncpu)
    

    View Slide

76. PROFILES
    76
    {
    "mutators": {
    "@default": true,
    "@function_signature": false,
    "TrueValue": {
    "ignore": [
    “NameSpace\\*\\SourceClass::create",
    "Full\\NameSpaced\\Class"
    ]
    }
    }
    }
    infection.json.dist
    https://infection.github.io/guide/profiles.html
    

    View Slide

77. RUN IT AGAIN
    77
    $ infection --threads=$(sysctl -n hw.ncpu) --only-covered
    

    View Slide

78. 78
    3573 mutations were generated:
    2322 mutants were killed
    569 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 69%
    Mutation Code Coverage: 84%
    Covered Code MSI: 82%
    Time: 14min 20s
    3004 mutations were generated:
    2322 mutants were killed
    0 mutants were not covered by tests
    526 covered mutants were not detected
    140 errors were encountered
    16 time outs were encountered
    Metrics:
    Mutation Score Indicator (MSI): 82%
    Mutation Code Coverage: 100%
    Covered Code MSI: 82%
    Time: 6min 39s
    

    View Slide

79. DEMO DONE
    79
    

    View Slide

80. IT IS NOT NEW… - HISTORY
    • Begins in 1971, R. Lipton, “Fault
    Diagnosis of Computer Programs”
    • Generally accepted in 1978, R. Lipton
    and al, “Hints on test data selection:
    Help for the practicing programmer”
    80
    

    View Slide

81. WHY IS IT NOT WIDELY USED?
    81
    http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this
    

    View Slide

82. WHY IS IT NOT WIDELY USED?
    Maturity Problem:
    Because testing is not widely used yet
    (Although it is increasing)
    82
    

    View Slide

83. WHY IS IT NOT WIDELY USED?
    Integration Problem:
    Inability to successfully integrated it into
    software development process
    (TDD plays a key role now)
    83
    

    View Slide

84. WHY IS IT NOT WIDELY USED?
    Technical Problem:
    It is a brute force technique!
    84
    

    View Slide

85. BRUTE FORCE TECHNIQUE
    N: Number of tests
    M: Number of mutants
    NxM
    85
    

    View Slide

86. THEORETICAL RUN
    • 672 tests in 25.29 seconds
    • 3573 mutants
    • 2,401,056 tests
    • ~53h
    With basic Mutation Testing
    86
    

    View Slide

87. View Slide

88. OPTIMISATION STRATEGIES
    • Mutate only covered code
    • Incremental analysis
    • Parallelism
    • Equivalent mutant (~30%)
    • Different levels of requirements
    88
    

    View Slide

89. OPTIMISATION STRATEGIES
    • 672 tests in 25.29 seconds
    • 3573 mutants
    • ~6min
    With Infection
    89
    

    View Slide

90. INFECTION IS STILL YOUNG
    90
    

    View Slide

91. FUTURE WORK
    91
    

    View Slide

92. PHP FRAMEWORKS SUPPORT
    • PHPUnit
    • PhpSpec
    • Behat (#147)
    • Atoum (#381)
    • CodeCeption (#45)
    92
    ✅
    ✅
    

    View Slide

93. EQUIVALENT MUTANT
    93
    - $x = $a + $b;
    + $x = $a - (-$b);
    - if ($nbr >= 10) {
    + if ($nbr > 10) {
    - if ($nbr >= 10) {
    + if ($nbr >= 9) {
    ~30% of
    mutants
    

    View Slide

94. SMART MUTATORS
    94
    class X
    {
    function foo(string $a, string $b): boolean
    {
    - return $a === $b;
    + return $a == $b;
    }
    }
    

    View Slide

95. {
    "mutators": {
    "@default": true,
    "@function_signature": false,
    "TrueValue": {
    "ignore": [
    “NameSpace\\*\\SourceClass::create",
    "Full\\NameSpaced\\Class"
    ]
    }
    }
    }
    infection.json.dist
    GRANULAR CONFIGURATION
    95
    

    View Slide

96. FALSE POSITIVE TRACKING
    96
    

    View Slide

97. IO-BOUND FRAMEWORK
    97
    

    View Slide

98. WRAP UP
    98
    https://www.pinterest.com/pin/140526450845150336/
    

    View Slide

99. THE GOOD PARTS
    • Gives you feedback on your tests
    • Test your tests with little effort
    • They are automatic
    • They discover dead code (e.g. useless tests)
    • Helps to refactor your tests
    99
    

    View Slide

100. THE NOT SO GOOD PARTS
     • Can be slow
     • Handful of young libraries
     • Writing smart mutants is difficult
     • Side effects with integration tests
     100
     

     View Slide

101. PERSONAL EXPERIENCE
     • Forces you to write tests
     • Always find bugs
     • Helpful for non-devs
     101
     

     View Slide

102. 102
     USE IT
     

     View Slide

103. MUTATION TESTING LIBRARIES
     HTTPS://GITHUB.COM/THEOFIDRY/AWESOME-MUTATION-TESTING
     103
     

     View Slide

104. HTTPS://JOIND.IN/TALK/B4496
     104
     

     View Slide

105. QUESTIONS?
     105
     https://imgflip.com/memegenerator/Shrek-Cat
     

     View Slide

106. THANK YOU!
     106
     

     View Slide