Mutation Testing PHPDeveloperDay

51b04191c1a7246b1ac7a0d52fe47acc?s=47 Théo FIDRY
September 22, 2018

Mutation Testing PHPDeveloperDay

Mutation testing: better code by making bugs Do you test your code? What about your tests? Your tests are code, you need to write, refactor and maintain them. This is not cheap so how do you make sure you are testing enough but not too much? Discover Mutation Testing, a fun tool to make your code better by introducing bugs.

51b04191c1a7246b1ac7a0d52fe47acc?s=128

Théo FIDRY

September 22, 2018
Tweet

Transcript

  1. MUTATION TESTING Better code by making bugs 1

  2. ABOUT ME 2 @tfidry @theofidry Théo Fidry Web developer Barcelona

    (Spain)
  3. PROJECTS 3 Alice Humbug & Infection
 PHP-Scoper & Box API-Platform

  4. LET’S DO A SHOW OF HANDS 4

  5. WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS

    INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5
  6. HOW TO DEFINE SOFTWARE QUALITY? 6

  7. 7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa

    The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality
  8. 8 (LACK OF) EXTERNAL QUALITY

  9. EXTERNAL QUALITY • Conformity to the user expectation • Reliability

    • Accuracy • Ergonomics • Design • … 9
  10. THE QUALITY PERCEIVED BY THE USER 10

  11. 11 (LACK OF) INTERNAL QUALITY https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744

  12. INTERNAL QUALITY • Maintainability • Concision • Cohesion • Simplicity

    • Clarity • … 12
  13. THE QUALITY PERCEIVED BY THE DEVELOPER (WHICH WE WILL FOCUS

    ON) 13
  14. SHOULD WE CARE? 14

  15. Cumulative functioncality Time 15 PROJECT STAMINA Payoff line https://martinfowler.com/bliki/DesignStaminaHypothesis.html

  16. THE LESS YOU CARE THE HARDER IT WILL BE TO

    ADD NEW FEATURES 16
  17. COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33

    US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ
  18. ORIGIN OF THE COSTS • Implementation • Debug & Repair

    • Technical debt • Delay: the extra you are paying for not fixing bugs earlier on 18
  19. ORIGIN OF THE COSTS 19 Technical debt Build cost Build

    cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA
  20. THE MORE YOU DELAY THE MORE EXPENSIVE IT GETS 20

  21. YOU SHOULD CARE 21

  22. HOW TO IMPROVE (INTERNAL) QUALITY? 22 (ONE) ANSWER: BY ADDING

    TESTS
  23. TESTS ARE CODE 23

  24. TESTS ARE CODE • You need to write them •

    You need to make sure they work • You need to refactor them • You need to maintain them 24
  25. TESTS ARE EXPENSIVE 25 NOT FREE

  26. PROBLEMS • How do I safely refactor my tests? •

    How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 26
  27. HOW DO I ASSESS THE QUALITY? 27

  28. 28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW

    VELOCITY TESTS QUALITY Level of quality
  29. HOW DO I ASSESS THE QUALITY OF THE TEST SUITE?

    29
  30. COMMON ANSWERS • Don’t worry, it’ll be fine • I’m

    a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30
  31. LINE CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART

    HAS BEEN TESTED 31
  32. 32 EXAMPLE https://github.com/theofidry/mutation-testing-demo <?php declare(strict_types=1); namespace Acme; final class Counter

    { public $count = 0; public function count(int $nbr): void { if ($nbr >= 10) { $this->count++; } } }
  33. 33 https://github.com/theofidry/mutation-testing-demo function test_it_counts_number_above_ten(): void { $initialCount = $this->counter->count; $this->counter->count(100);

    $newCount = $this->counter->count; Assert::assertGreaterThan($initialCount, $newCount); } function test_it_does_not_count_number_below_ten(): void { $initialCount = $this->counter->count; $this->counter->count(7); $newCount = $this->counter->count; Assert::assertSame($initialCount, $newCount); }
  34. 34 RUN PHPUNIT WITH COVERAGE REPORT https://github.com/theofidry/mutation-testing-demo $ vendor/bin/phpunit --coverage-text

    \ --coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .. 2 / 2 (100%) Time: 113 ms, Memory: 6.00MB OK (2 tests, 2 assertions) Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3) 100% code coverage
  35. IS IT GOOD ENOUGH? 35

  36. 36 LET’S INTRODUCE A BUG https://github.com/theofidry/mutation-testing-demo <?php declare(strict_types=1); namespace Acme;

    final class Counter { public $count = 0; public function count(int $nbr): void { if ($nbr > 10) { $this->count++; } } }
  37. 37 RUN PHPUNIT WITH COVERAGE REPORT https://github.com/theofidry/mutation-testing-demo $ vendor/bin/phpunit --coverage-text

    \ —-coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .. 2 / 2 (100%) Time: 113 ms, Memory: 6.00MB OK (2 tests, 2 assertions) Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3) DID NOT FAIL
  38. OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 38

  39. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS

    BEEN TESTED 39
  40. WHAT CODE COVERAGE DOES TELL YOU 40

  41. EXECUTED ≠ TESTED Executed Tested 41

  42. CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED

    42
  43. 43 A TEST CASE IS MISSING https://github.com/theofidry/mutation-testing-demo function test_it_counts_ten(): void

    { $initialCount = $this->counter->count; $this->counter->count(10); $newCount = $this->counter->count; Assert::assertGreaterThan($initialCount, $newCount); }
  44. 44 $ vendor/bin/phpunit --coverage-text \ --coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian

    Bergmann and contributors. .F. 3 / 3 (100%) Time: 70 ms, Memory: 6.00MB There was 1 failure: 1) Acme\CounterTest::test_it_counts_ten Failed asserting that 0 is greater than 0. /path/to/tests/CounterTest.php:38 FAILURES! Tests: 3, Assertions: 3, Failures: 1. Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
  45. HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 45

  46. INTRODUCE A BUG 46

  47. MUTATION TESTING 47

  48. CREATE A MUTANT SOURCE CODE MUTATOR MUTATION PROCESS MUTANT 48

  49. EXAMPLE OF A MUTANT 49 <?php declare(strict_types=1); namespace Acme; final

    class Counter { public $count = 0; public function count(int $nbr): void { if ($nbr >= 10) { $this->count++; } } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 10) { $this->count++; } }
  50. MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo

    >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html
  51. COLLECT THE SOURCE FILES 51 Counter.php Foo.php FILE COLLECTOR

  52. GENERATE MUTANTS 52 Counter.php Foo.php MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR

    MUTATOR MUTANTS
  53. --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class

    Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 9) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr >= 11) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 10) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { if ($nbr >= 10) { - $this->count++; + $this->count—-; } } GENERATED MUTANTS 53 … and more
  54. APPLY MUTANTS 54 PROCESS BUILDER MUTANT PROCESS WITH MUTATED CODE

    RESULT Runs tests TESTS RUNNER
  55. IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL,

    IT SURVIVED 55
  56. IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT

    WAS KILLED 56
  57. MUTATION SCORE Nbr of mutant killed Nbr of mutant generated

    Mutation score = 57
  58. CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION

    SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?
  59. DOES IT WORK? “Complex faults are coupled to simple faults

    in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59
  60. MUTATION TESTING IN PHP infection/infection 60 humbug/humbug

  61. DEMO 61

  62. MUTATION TESTING IN PHP infection/infection 62 symfony/dependency-injection

  63. SYMFONY/DEPENDENCY-INJECTION 63

  64. 64 INSTALLATION Infection installation guide: https://infection.github.io/guide/installation.html PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html

    $ git clone git@github.com:symfony/dependency-injection.git infection-demo $ cd infection-demo $ composer install $ wget -O infection https://github.com/infection/infection/releases/ download/0.10.3/infection.phar $ chmod +x infection $ wget -O phpunit https://phar.phpunit.de/phpunit-7.phar $ chmod /+x phpunit
  65. PHPUNIT CONFIGURATION 65 phpunit.xml.dist <?xml version="1.0" encoding="UTF-8"?> <phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/7.0/phpunit.xsd"

    bootstrap="vendor/autoload.php"> <php> <ini name="error_reporting" value="-1" /> </php> <testsuites> <testsuite name="Symfony DependencyInjection Component Test Suite"> <directory>./Tests/</directory> </testsuite> </testsuites> <filter> <whitelist> <directory>./</directory> <exclude> <directory>./Resources</directory> <directory>./Tests</directory> <directory>./vendor</directory> </exclude> </whitelist> </filter> </phpunit>
  66. 66 RUNNING TESTS $ ./phpunit PHPUnit 7.0.2 by Sebastian Bergmann

    and contributors. ............................................................... 63 / 672 ( 9%) ............................................................... 126 / 672 ( 18%) ............................................................... 189 / 672 ( 28%) ............................................................... 252 / 672 ( 37%) ............................................................... 315 / 672 ( 46%) ............................................................... 378 / 672 ( 56%) ............................................................... 441 / 672 ( 65%) ............................................................... 504 / 672 ( 75%) ........S.SSSSS................................................ 567 / 672 ( 84%) ............................................................... 630 / 672 ( 93%) .......................................... 672 / 672 (100%) Time: 1.89 seconds, Memory: 26.00MB OK, but incomplete, skipped, or risky tests! Tests: 672, Assertions: 1550, Skipped: 6.
  67. CONFIGURE INFECTION 67 $ ./infection init

  68. INFECTION CONFIGURATION 68 { "timeout": 10, "source": { "directories": [

    "." ], "excludes": [ "Tests", "vendor" ] }, "phpunit": { "customPath": "phpunit" }, "logs": { "text": "infection-log.txt" } } infection.json.dist https://infection.github.io/guide/usage.html#Configuration
  69. 69 RUNNING INFECTION $ ./infection You are running Infection with

    xdebug enabled. ____ ____ __ _ / _/___ / __/__ _____/ /_(_)___ ____ / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \ _/ // / / / __/ __/ /__/ /_/ / /_/ / / / / /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/ Running initial test suite... PHPUnit version: 7.0.2 678 [============================] 26 secs Generate mutants... Processing source code files: 144/144 Creating mutated files and processes: 3573/3573 .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573) E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573) ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573) .M...........M...........................M.MM..... ( 200 / 3573) ....MM..M.......M.....M...SSSSSSSS...M......M.M... ( 250 / 3573) .M.M.MEM..MST.E.M.......MMMM............SMMM.....E ( 300 / 3573) .E.......TM.........SM....................EM...... ( 350 / 3573) ....EM.M...E..MM.MM.S.MS......M.....EM...E...EE... ( 400 / 3573) ...TME......S................MM....M........E...M. ( 450 / 3573) ...ESSSSSSSSS...MM.SM....ESSMMMM.MSS..MM.SM.M.MMMS ( 500 / 3573) SS.S......M........S.S....ME...M.......M...S.M..S. ( 550 / 3573) ...MS.......S.M....E.E..EET.M.M.....M..MM.S....... ( 600 / 3573) .E.S.T.E..S..MM..M.EM.E..E......S.MM...MM...M.M... ( 650 / 3573) .....M..M......EM..ESS.S.M.............M.....E..MS ( 700 / 3573) ....EM.........EE...........MMME.....E.M..MM..M... ( 750 / 3573) M.......EE...........M.......MMEMM..M...MM.M..M... ( 800 / 3573) ..M....................MMM...M...MM.........MM...M ( 850 / 3573) .SSM.......MM..M......E..SEMM....E.M..M.E..EM..ETM ( 900 / 3573) M.......EM.EM...SSSSSSSSSSSSSSSSSSSSSSSE.S.M...... ( 950 / 3573) .S..M.........MM........S......EE.E......E....EEE. (1000 / 3573) .S....SSM..E..S.E....TMSSSMM.........M.........E.S (1050 / 3573) EE..M....STM..M.......M.S......SM..........S.SM..S (1100 / 3573) ....MMMMM.MMM....M.....M.......M.SM...M.MMM.S..... (1150 / 3573) .E....E.M.E.....MM..MMMMMS.M........MMMMMM.......E (1200 / 3573) .EE....S.......................T......EEM.M.MM.... (1250 / 3573) ..SSMM.MM..MMMM.MMMMMMMM.......................... (1300 / 3573) ..E....E.......E..SSSSM..M.S.E..M............MMM.M (1350 / 3573) ......M.SMMM...............M...M.M......M.M.E..... (1400 / 3573) .MSM.........S.MMMMMMMMMM..........M.............. (1450 / 3573) ................M...............S..M.............. (1500 / 3573) ....M...............M.........M................M.. (1550 / 3573) ...........................M...................... (1600 / 3573) ..E......M...M...........M.MM.....SSSS..MMM..MM... (1650 / 3573) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMSMSE....M..MMM (1700 / 3573) M.M...M.MM........MM..M...M.SSSSSSSSSSSMM.MSSSSSSM (1750 / 3573) MMSSSSSSSSSSSSSSSSSSSSSSSS...MM............MMS.... (1800 / 3573) MMMMMM..S....M......MMM...SSSSSSSSSSSSSSSSSSSSSSSS (1850 / 3573) SSSSSSSM..S.....M........MS....................... (1900 / 3573) ............SSSS..MM.SSSSSSSSSSSSSSSSSSSSSSSS..... (1950 / 3573) .M.M..........SS.M.....SSSSS.....SSS.............. (2000 / 3573) .....M.S..SSSSSSS..SM...SSSSSSSS.MSM....MM.....M.. (2050 / 3573) ..SSSSSS..MMMSSSSSSSSSS..S.....M.MSSSS........SSSS (2100 / 3573) S....SSSSSSSS.MMM.SMSSSMM.M.......SSS...M....M..M. (2150 / 3573) MSM.MMS..MMM.M...SSS........M.......SS.S..SSSSSSSS (2200 / 3573) SSSSSSSSSSSSSSSSSS.S..MMMSSSSMS......MS....MM..SSS (2250 / 3573) SS...M........M.SSSSSSMM..MMM.MMM...M..MMTMTTSSSMM (2300 / 3573) S..S...........MSMSSS...SSSSSSSSSSSSSSSSSSSSSSSSSS (2350 / 3573) SSSSSSMMMMMSS.M.MMM.SSSSSSSSSSSSSSSSSSS.E......... (2400 / 3573) .................................................. (2450 / 3573) ......SS.....MS.......M.S.E...........M.M......... (2500 / 3573) ..M.M......M............SS.S.......M.......E.SSSSS (2550 / 3573) .................S....S.SS.SSS.S.SSSSS..M....SSSSS (2600 / 3573) E.SMMSMM....MMSMM......MSSSS.MMSS.....MMSSSS.SSSE. (2650 / 3573) ..S....EMMM.E......E.EEEM........SSSSSSEESSM..SSMS (2700 / 3573) ........SSSM.....MS.M.MM..SSM..E....M.SS.E...M..M. (2750 / 3573) .M.MM.......M..........MMMMMMM..S.....SSM..MM..... (2800 / 3573) .........M........SSS....EE....M........M...M...MS (2850 / 3573) .M.MM..MM..M.E...M....E...S.SE..M........S..MM.... (2900 / 3573) ...M...MS...S.....SS..E.EM......M.E.S.....ES...... (2950 / 3573) .....MMM......MM......M.......E.S.....EE.S.....E.. (3000 / 3573) M..M..........S.MM.....S............M........M.... (3050 / 3573) .........MMM.M.........M.....MM.....S.SMMS.S...... (3100 / 3573) ....M....M.M....S...M..M....SSSM...........M...... (3150 / 3573) .MME.S.M..M.E......M.........MSMMM.S.......MM.S..S (3200 / 3573) ..MSSSSMMM.SS..........................M.S........ (3250 / 3573) .....S............................S......SMMM..SSS (3300 / 3573) ..........M.MM.......M...MSS.S.S....MM.M...M...... (3350 / 3573) S.S...MSM........S......S.......MM........MM...... (3400 / 3573) ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573) .................MM.... (3573 / 3573) 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Please note that some mutants will inevitably be harmless (i.e. fa 14min 21s
  70. 70 You are running Infection with xdebug enabled. $ ./infection

    ____ ____ __ _ / _/___ / __/__ _____/ /_(_)___ ____ / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \ _/ // / / / __/ __/ /__/ /_/ / /_/ / / / / /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/ Running initial test suite... PHPUnit version: 7.0.2 678 [============================] 26 secs Generate mutants... Processing source code files: 144/144 Creating mutated files and processes: 3573/3573 .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573) E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573) ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)
  71. 71 ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573) .................MM.... (3573 / 3573) 3573

    mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Please note that some mutants will inevitably be harmless (i.e. false positives).
  72. 72 Escaped mutants: ================ 1) /path/to/Alias.php:68 [M] This exec /path/to/php

    -c /tmp/dir/infectionyy4H8n /path/to/phpunit --configuration /tmp/dir/infection/ phpunitConfiguration.bce05.infection.xml --stop-on-failure --- Original +++ New @@ @@ $this->private = (bool) $boolean; - return $this; + return null; } /** * Whether this alias is private. * * @return bool */ PHPUnit 7.0.2 by Sebastian Bergmann and contributors. ............................................................... 63 / 127 ( 49%) ............................................................... 126 / 127 ( 99%) . 127 / 127 (100%) Time: 374 ms, Memory: 14.00MB OK (127 tests, 438 assertions) REPORT
  73. TIPS 73

  74. RESTRICT THE FILES IT’S RUNNING ON 74 $ infection —-filter=DependencyInjection

    INFECTION_FILTER=$(git diff ${GIT_PREVIOUS_SUCCESSFUL_COMMIT:-origin/develop} \ $GIT_COMMIT --name-only \ | grep /src/ \ | paste -sd "," -) $ infection —-filter=$INFECTION_FILTER https://infection.github.io/guide/command-line-options.html $ infection —-min-covered-msi 70
  75. MAKE IT FASTER 75 $ infection —-threads=$(nproc) https://infection.github.io/guide/command-line-options.html $ infection

    —-only-covered Should this be the default? $(sysctl -n hw.ncpu)
  76. PROFILES 76 { "mutators": { "@default": true, "@function_signature": false, "TrueValue":

    { "ignore": [ “NameSpace\\*\\SourceClass::create", "Full\\NameSpaced\\Class" ] } } } infection.json.dist https://infection.github.io/guide/profiles.html
  77. RUN IT AGAIN 77 $ infection --threads=$(sysctl -n hw.ncpu) --only-covered

  78. 78 3573 mutations were generated: 2322 mutants were killed 569

    mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Time: 14min 20s 3004 mutations were generated: 2322 mutants were killed 0 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 82% Mutation Code Coverage: 100% Covered Code MSI: 82% Time: 6min 39s
  79. DEMO DONE 79

  80. IT IS NOT NEW… - HISTORY • Begins in 1971,

    R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 80
  81. WHY IS IT NOT WIDELY USED? 81 http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this

  82. WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing

    is not widely used yet (Although it is increasing) 82
  83. WHY IS IT NOT WIDELY USED? Integration Problem: Inability to

    successfully integrated it into software development process (TDD plays a key role now) 83
  84. WHY IS IT NOT WIDELY USED? Technical Problem: It is

    a brute force technique! 84
  85. BRUTE FORCE TECHNIQUE N: Number of tests M: Number of

    mutants NxM 85
  86. THEORETICAL RUN • 672 tests in 25.29 seconds • 3573

    mutants • 2,401,056 tests • ~53h With basic Mutation Testing 86
  87. 87

  88. OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis

    • Parallelism • Equivalent mutant (~30%) • Different levels of requirements 88
  89. OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds • 3573

    mutants • ~6min With Infection 89
  90. INFECTION IS STILL YOUNG 90

  91. FUTURE WORK 91

  92. PHP FRAMEWORKS SUPPORT • PHPUnit • PhpSpec • Behat (#147)

    • Atoum (#381) • CodeCeption (#45) 92 ✅ ✅
  93. EQUIVALENT MUTANT 93 - $x = $a + $b; +

    $x = $a - (-$b); - if ($nbr >= 10) { + if ($nbr > 10) { - if ($nbr >= 10) { + if ($nbr >= 9) { ~30% of mutants
  94. SMART MUTATORS 94 <?php declare(strict_types = 1); class X {

    function foo(string $a, string $b): boolean { - return $a === $b; + return $a == $b; } }
  95. { "mutators": { "@default": true, "@function_signature": false, "TrueValue": { "ignore":

    [ “NameSpace\\*\\SourceClass::create", "Full\\NameSpaced\\Class" ] } } } infection.json.dist GRANULAR CONFIGURATION 95
  96. FALSE POSITIVE TRACKING 96

  97. IO-BOUND FRAMEWORK 97

  98. WRAP UP 98 https://www.pinterest.com/pin/140526450845150336/

  99. THE GOOD PARTS • Gives you feedback on your tests

    • Test your tests with little effort • They are automatic • They discover dead code (e.g. useless tests) • Helps to refactor your tests 99
  100. THE NOT SO GOOD PARTS • Can be slow •

    Handful of young libraries • Writing smart mutants is difficult • Side effects with integration tests 100
  101. PERSONAL EXPERIENCE • Forces you to write tests • Always

    find bugs • Helpful for non-devs 101
  102. 102 USE IT

  103. MUTATION TESTING LIBRARIES HTTPS://GITHUB.COM/THEOFIDRY/AWESOME-MUTATION-TESTING 103

  104. HTTPS://JOIND.IN/TALK/B4496 104

  105. QUESTIONS? 105 https://imgflip.com/memegenerator/Shrek-Cat

  106. THANK YOU! 106