Mutation Testing PHPDeveloperDay

Mutation testing: better code by making bugs Do you test your code? What about your tests? Your tests are code, you need to write, refactor and maintain them. This is not cheap so how do you make sure you are testing enough but not too much? Discover Mutation Testing, a fun tool to make your code better by introducing bugs.

Transcript

1. MUTATION TESTING Better code by making bugs 1
2. ABOUT ME 2 @tfidry @theofidry Théo Fidry Web developer Barcelona (Spain)
3. PROJECTS 3 Alice Humbug & Infection
 PHP-Scoper & Box API-Platform
4. LET’S DO A SHOW OF HANDS 4
5. WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5
6. HOW TO DEFINE SOFTWARE QUALITY? 6
7. 7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality
8. 8 (LACK OF) EXTERNAL QUALITY
9. EXTERNAL QUALITY • Conformity to the user expectation • Reliability • Accuracy • Ergonomics • Design • … 9
10. THE QUALITY PERCEIVED BY THE USER 10
11. 11 (LACK OF) INTERNAL QUALITY https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744
12. INTERNAL QUALITY • Maintainability • Concision • Cohesion • Simplicity • Clarity • … 12
13. THE QUALITY PERCEIVED BY THE DEVELOPER (WHICH WE WILL FOCUS ON) 13
14. SHOULD WE CARE? 14
15. Cumulative functioncality Time 15 PROJECT STAMINA Payoff line https://martinfowler.com/bliki/DesignStaminaHypothesis.html
16. THE LESS YOU CARE THE HARDER IT WILL BE TO ADD NEW FEATURES 16
17. COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33 US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ
18. ORIGIN OF THE COSTS • Implementation • Debug & Repair • Technical debt • Delay: the extra you are paying for not fixing bugs earlier on 18
19. ORIGIN OF THE COSTS 19 Technical debt Build cost Build cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA
20. THE MORE YOU DELAY THE MORE EXPENSIVE IT GETS 20
21. YOU SHOULD CARE 21
22. HOW TO IMPROVE (INTERNAL) QUALITY? 22 (ONE) ANSWER: BY ADDING TESTS
23. TESTS ARE CODE 23
24. TESTS ARE CODE • You need to write them • You need to make sure they work • You need to refactor them • You need to maintain them 24
25. TESTS ARE EXPENSIVE 25 NOT FREE
26. PROBLEMS • How do I safely refactor my tests? • How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 26
27. HOW DO I ASSESS THE QUALITY? 27
28. 28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW VELOCITY TESTS QUALITY Level of quality
29. HOW DO I ASSESS THE QUALITY OF THE TEST SUITE? 29
30. COMMON ANSWERS • Don’t worry, it’ll be fine • I’m a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30
31. LINE CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 31
32. 32 EXAMPLE https://github.com/theofidry/mutation-testing-demo <?php declare(strict_types=1); namespace Acme; final class Counter { public $count = 0; public function count(int $nbr): void { if ($nbr >= 10) { $this->count++; } } }
33. 33 https://github.com/theofidry/mutation-testing-demo function test_it_counts_number_above_ten(): void { $initialCount = $this->counter->count; $this->counter->count(100); $newCount = $this->counter->count; Assert::assertGreaterThan($initialCount, $newCount); } function test_it_does_not_count_number_below_ten(): void { $initialCount = $this->counter->count; $this->counter->count(7); $newCount = $this->counter->count; Assert::assertSame($initialCount, $newCount); }
34. 34 RUN PHPUNIT WITH COVERAGE REPORT https://github.com/theofidry/mutation-testing-demo $ vendor/bin/phpunit --coverage-text \ --coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .. 2 / 2 (100%) Time: 113 ms, Memory: 6.00MB OK (2 tests, 2 assertions) Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3) 100% code coverage
35. IS IT GOOD ENOUGH? 35
36. 36 LET’S INTRODUCE A BUG https://github.com/theofidry/mutation-testing-demo <?php declare(strict_types=1); namespace Acme; final class Counter { public $count = 0; public function count(int $nbr): void { if ($nbr > 10) { $this->count++; } } }
37. 37 RUN PHPUNIT WITH COVERAGE REPORT https://github.com/theofidry/mutation-testing-demo $ vendor/bin/phpunit --coverage-text \ —-coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .. 2 / 2 (100%) Time: 113 ms, Memory: 6.00MB OK (2 tests, 2 assertions) Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3) DID NOT FAIL
38. OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 38
39. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 39
40. WHAT CODE COVERAGE DOES TELL YOU 40
41. EXECUTED ≠ TESTED Executed Tested 41
42. CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED 42
43. 43 A TEST CASE IS MISSING https://github.com/theofidry/mutation-testing-demo function test_it_counts_ten(): void { $initialCount = $this->counter->count; $this->counter->count(10); $newCount = $this->counter->count; Assert::assertGreaterThan($initialCount, $newCount); }
44. 44 $ vendor/bin/phpunit --coverage-text \ --coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .F. 3 / 3 (100%) Time: 70 ms, Memory: 6.00MB There was 1 failure: 1) Acme\CounterTest::test_it_counts_ten Failed asserting that 0 is greater than 0. /path/to/tests/CounterTest.php:38 FAILURES! Tests: 3, Assertions: 3, Failures: 1. Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)
45. HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 45
46. INTRODUCE A BUG 46
47. MUTATION TESTING 47
48. CREATE A MUTANT SOURCE CODE MUTATOR MUTATION PROCESS MUTANT 48
49. EXAMPLE OF A MUTANT 49 <?php declare(strict_types=1); namespace Acme; final class Counter { public $count = 0; public function count(int $nbr): void { if ($nbr >= 10) { $this->count++; } } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 10) { $this->count++; } }
50. MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html
51. COLLECT THE SOURCE FILES 51 Counter.php Foo.php FILE COLLECTOR
52. GENERATE MUTANTS 52 Counter.php Foo.php MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTANTS
53. --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 9) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr >= 11) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 10) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { if ($nbr >= 10) { - $this->count++; + $this->count—-; } } GENERATED MUTANTS 53 … and more
54. APPLY MUTANTS 54 PROCESS BUILDER MUTANT PROCESS WITH MUTATED CODE RESULT Runs tests TESTS RUNNER
55. IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL, IT SURVIVED 55
56. IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT WAS KILLED 56
57. MUTATION SCORE Nbr of mutant killed Nbr of mutant generated Mutation score = 57
58. CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?
59. DOES IT WORK? “Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59
60. MUTATION TESTING IN PHP infection/infection 60 humbug/humbug
61. DEMO 61
62. MUTATION TESTING IN PHP infection/infection 62 symfony/dependency-injection
63. SYMFONY/DEPENDENCY-INJECTION 63
64. 64 INSTALLATION Infection installation guide: https://infection.github.io/guide/installation.html PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html $ git clone git@github.com:symfony/dependency-injection.git infection-demo $ cd infection-demo $ composer install $ wget -O infection https://github.com/infection/infection/releases/ download/0.10.3/infection.phar $ chmod +x infection $ wget -O phpunit https://phar.phpunit.de/phpunit-7.phar $ chmod /+x phpunit
65. PHPUNIT CONFIGURATION 65 phpunit.xml.dist <?xml version="1.0" encoding="UTF-8"?> <phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/7.0/phpunit.xsd" bootstrap="vendor/autoload.php"> <php> <ini name="error_reporting" value="-1" /> </php> <testsuites> <testsuite name="Symfony DependencyInjection Component Test Suite"> <directory>./Tests/</directory> </testsuite> </testsuites> <filter> <whitelist> <directory>./</directory> <exclude> <directory>./Resources</directory> <directory>./Tests</directory> <directory>./vendor</directory> </exclude> </whitelist> </filter> </phpunit>
66. 66 RUNNING TESTS $ ./phpunit PHPUnit 7.0.2 by Sebastian Bergmann and contributors. ............................................................... 63 / 672 ( 9%) ............................................................... 126 / 672 ( 18%) ............................................................... 189 / 672 ( 28%) ............................................................... 252 / 672 ( 37%) ............................................................... 315 / 672 ( 46%) ............................................................... 378 / 672 ( 56%) ............................................................... 441 / 672 ( 65%) ............................................................... 504 / 672 ( 75%) ........S.SSSSS................................................ 567 / 672 ( 84%) ............................................................... 630 / 672 ( 93%) .......................................... 672 / 672 (100%) Time: 1.89 seconds, Memory: 26.00MB OK, but incomplete, skipped, or risky tests! Tests: 672, Assertions: 1550, Skipped: 6.
67. CONFIGURE INFECTION 67 $ ./infection init
68. INFECTION CONFIGURATION 68 { "timeout": 10, "source": { "directories": [ "." ], "excludes": [ "Tests", "vendor" ] }, "phpunit": { "customPath": "phpunit" }, "logs": { "text": "infection-log.txt" } } infection.json.dist https://infection.github.io/guide/usage.html#Configuration
69. 69 RUNNING INFECTION $ ./infection You are running Infection with xdebug enabled. ____ ____ __ _ / _/___ / __/__ _____/ /_(_)___ ____ / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \ _/ // / / / __/ __/ /__/ /_/ / /_/ / / / / /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/ Running initial test suite... PHPUnit version: 7.0.2 678 [============================] 26 secs Generate mutants... Processing source code files: 144/144 Creating mutated files and processes: 3573/3573 .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573) E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573) ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573) .M...........M...........................M.MM..... ( 200 / 3573) ....MM..M.......M.....M...SSSSSSSS...M......M.M... ( 250 / 3573) .M.M.MEM..MST.E.M.......MMMM............SMMM.....E ( 300 / 3573) .E.......TM.........SM....................EM...... ( 350 / 3573) ....EM.M...E..MM.MM.S.MS......M.....EM...E...EE... ( 400 / 3573) ...TME......S................MM....M........E...M. ( 450 / 3573) ...ESSSSSSSSS...MM.SM....ESSMMMM.MSS..MM.SM.M.MMMS ( 500 / 3573) SS.S......M........S.S....ME...M.......M...S.M..S. ( 550 / 3573) ...MS.......S.M....E.E..EET.M.M.....M..MM.S....... ( 600 / 3573) .E.S.T.E..S..MM..M.EM.E..E......S.MM...MM...M.M... ( 650 / 3573) .....M..M......EM..ESS.S.M.............M.....E..MS ( 700 / 3573) ....EM.........EE...........MMME.....E.M..MM..M... ( 750 / 3573) M.......EE...........M.......MMEMM..M...MM.M..M... ( 800 / 3573) ..M....................MMM...M...MM.........MM...M ( 850 / 3573) .SSM.......MM..M......E..SEMM....E.M..M.E..EM..ETM ( 900 / 3573) M.......EM.EM...SSSSSSSSSSSSSSSSSSSSSSSE.S.M...... ( 950 / 3573) .S..M.........MM........S......EE.E......E....EEE. (1000 / 3573) .S....SSM..E..S.E....TMSSSMM.........M.........E.S (1050 / 3573) EE..M....STM..M.......M.S......SM..........S.SM..S (1100 / 3573) ....MMMMM.MMM....M.....M.......M.SM...M.MMM.S..... (1150 / 3573) .E....E.M.E.....MM..MMMMMS.M........MMMMMM.......E (1200 / 3573) .EE....S.......................T......EEM.M.MM.... (1250 / 3573) ..SSMM.MM..MMMM.MMMMMMMM.......................... (1300 / 3573) ..E....E.......E..SSSSM..M.S.E..M............MMM.M (1350 / 3573) ......M.SMMM...............M...M.M......M.M.E..... (1400 / 3573) .MSM.........S.MMMMMMMMMM..........M.............. (1450 / 3573) ................M...............S..M.............. (1500 / 3573) ....M...............M.........M................M.. (1550 / 3573) ...........................M...................... (1600 / 3573) ..E......M...M...........M.MM.....SSSS..MMM..MM... (1650 / 3573) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMSMSE....M..MMM (1700 / 3573) M.M...M.MM........MM..M...M.SSSSSSSSSSSMM.MSSSSSSM (1750 / 3573) MMSSSSSSSSSSSSSSSSSSSSSSSS...MM............MMS.... (1800 / 3573) MMMMMM..S....M......MMM...SSSSSSSSSSSSSSSSSSSSSSSS (1850 / 3573) SSSSSSSM..S.....M........MS....................... (1900 / 3573) ............SSSS..MM.SSSSSSSSSSSSSSSSSSSSSSSS..... (1950 / 3573) .M.M..........SS.M.....SSSSS.....SSS.............. (2000 / 3573) .....M.S..SSSSSSS..SM...SSSSSSSS.MSM....MM.....M.. (2050 / 3573) ..SSSSSS..MMMSSSSSSSSSS..S.....M.MSSSS........SSSS (2100 / 3573) S....SSSSSSSS.MMM.SMSSSMM.M.......SSS...M....M..M. (2150 / 3573) MSM.MMS..MMM.M...SSS........M.......SS.S..SSSSSSSS (2200 / 3573) SSSSSSSSSSSSSSSSSS.S..MMMSSSSMS......MS....MM..SSS (2250 / 3573) SS...M........M.SSSSSSMM..MMM.MMM...M..MMTMTTSSSMM (2300 / 3573) S..S...........MSMSSS...SSSSSSSSSSSSSSSSSSSSSSSSSS (2350 / 3573) SSSSSSMMMMMSS.M.MMM.SSSSSSSSSSSSSSSSSSS.E......... (2400 / 3573) .................................................. (2450 / 3573) ......SS.....MS.......M.S.E...........M.M......... (2500 / 3573) ..M.M......M............SS.S.......M.......E.SSSSS (2550 / 3573) .................S....S.SS.SSS.S.SSSSS..M....SSSSS (2600 / 3573) E.SMMSMM....MMSMM......MSSSS.MMSS.....MMSSSS.SSSE. (2650 / 3573) ..S....EMMM.E......E.EEEM........SSSSSSEESSM..SSMS (2700 / 3573) ........SSSM.....MS.M.MM..SSM..E....M.SS.E...M..M. (2750 / 3573) .M.MM.......M..........MMMMMMM..S.....SSM..MM..... (2800 / 3573) .........M........SSS....EE....M........M...M...MS (2850 / 3573) .M.MM..MM..M.E...M....E...S.SE..M........S..MM.... (2900 / 3573) ...M...MS...S.....SS..E.EM......M.E.S.....ES...... (2950 / 3573) .....MMM......MM......M.......E.S.....EE.S.....E.. (3000 / 3573) M..M..........S.MM.....S............M........M.... (3050 / 3573) .........MMM.M.........M.....MM.....S.SMMS.S...... (3100 / 3573) ....M....M.M....S...M..M....SSSM...........M...... (3150 / 3573) .MME.S.M..M.E......M.........MSMMM.S.......MM.S..S (3200 / 3573) ..MSSSSMMM.SS..........................M.S........ (3250 / 3573) .....S............................S......SMMM..SSS (3300 / 3573) ..........M.MM.......M...MSS.S.S....MM.M...M...... (3350 / 3573) S.S...MSM........S......S.......MM........MM...... (3400 / 3573) ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573) .................MM.... (3573 / 3573) 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Please note that some mutants will inevitably be harmless (i.e. fa 14min 21s
70. 70 You are running Infection with xdebug enabled. $ ./infection ____ ____ __ _ / _/___ / __/__ _____/ /_(_)___ ____ / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \ _/ // / / / __/ __/ /__/ /_/ / /_/ / / / / /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/ Running initial test suite... PHPUnit version: 7.0.2 678 [============================] 26 secs Generate mutants... Processing source code files: 144/144 Creating mutated files and processes: 3573/3573 .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573) E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573) ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)
71. 71 ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573) .................MM.... (3573 / 3573) 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Please note that some mutants will inevitably be harmless (i.e. false positives).
72. 72 Escaped mutants: ================ 1) /path/to/Alias.php:68 [M] This exec /path/to/php -c /tmp/dir/infectionyy4H8n /path/to/phpunit --configuration /tmp/dir/infection/ phpunitConfiguration.bce05.infection.xml --stop-on-failure --- Original +++ New @@ @@ $this->private = (bool) $boolean; - return $this; + return null; } /** * Whether this alias is private. * * @return bool */ PHPUnit 7.0.2 by Sebastian Bergmann and contributors. ............................................................... 63 / 127 ( 49%) ............................................................... 126 / 127 ( 99%) . 127 / 127 (100%) Time: 374 ms, Memory: 14.00MB OK (127 tests, 438 assertions) REPORT
73. TIPS 73
74. RESTRICT THE FILES IT’S RUNNING ON 74 $ infection —-filter=DependencyInjection INFECTION_FILTER=$(git diff ${GIT_PREVIOUS_SUCCESSFUL_COMMIT:-origin/develop} \ $GIT_COMMIT --name-only \ | grep /src/ \ | paste -sd "," -) $ infection —-filter=$INFECTION_FILTER https://infection.github.io/guide/command-line-options.html $ infection —-min-covered-msi 70
75. MAKE IT FASTER 75 $ infection —-threads=$(nproc) https://infection.github.io/guide/command-line-options.html $ infection —-only-covered Should this be the default? $(sysctl -n hw.ncpu)
76. PROFILES 76 { "mutators": { "@default": true, "@function_signature": false, "TrueValue": { "ignore": [ “NameSpace\\*\\SourceClass::create", "Full\\NameSpaced\\Class" ] } } } infection.json.dist https://infection.github.io/guide/profiles.html
77. RUN IT AGAIN 77 $ infection --threads=$(sysctl -n hw.ncpu) --only-covered
78. 78 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Time: 14min 20s 3004 mutations were generated: 2322 mutants were killed 0 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 82% Mutation Code Coverage: 100% Covered Code MSI: 82% Time: 6min 39s
79. DEMO DONE 79
80. IT IS NOT NEW… - HISTORY • Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 80
81. WHY IS IT NOT WIDELY USED? 81 http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this
82. WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing is not widely used yet (Although it is increasing) 82
83. WHY IS IT NOT WIDELY USED? Integration Problem: Inability to successfully integrated it into software development process (TDD plays a key role now) 83
84. WHY IS IT NOT WIDELY USED? Technical Problem: It is a brute force technique! 84
85. BRUTE FORCE TECHNIQUE N: Number of tests M: Number of mutants NxM 85
86. THEORETICAL RUN • 672 tests in 25.29 seconds • 3573 mutants • 2,401,056 tests • ~53h With basic Mutation Testing 86
88. OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis • Parallelism • Equivalent mutant (~30%) • Different levels of requirements 88
89. OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds • 3573 mutants • ~6min With Infection 89
90. INFECTION IS STILL YOUNG 90
91. FUTURE WORK 91
92. PHP FRAMEWORKS SUPPORT • PHPUnit • PhpSpec • Behat (#147) • Atoum (#381) • CodeCeption (#45) 92 ✅ ✅
93. EQUIVALENT MUTANT 93 - $x = $a + $b; + $x = $a - (-$b); - if ($nbr >= 10) { + if ($nbr > 10) { - if ($nbr >= 10) { + if ($nbr >= 9) { ~30% of mutants
94. SMART MUTATORS 94 <?php declare(strict_types = 1); class X { function foo(string $a, string $b): boolean { - return $a === $b; + return $a == $b; } }
95. { "mutators": { "@default": true, "@function_signature": false, "TrueValue": { "ignore": [ “NameSpace\\*\\SourceClass::create", "Full\\NameSpaced\\Class" ] } } } infection.json.dist GRANULAR CONFIGURATION 95
96. FALSE POSITIVE TRACKING 96
97. IO-BOUND FRAMEWORK 97
98. WRAP UP 98 https://www.pinterest.com/pin/140526450845150336/
99. THE GOOD PARTS • Gives you feedback on your tests • Test your tests with little effort • They are automatic • They discover dead code (e.g. useless tests) • Helps to refactor your tests 99
100. THE NOT SO GOOD PARTS • Can be slow • Handful of young libraries • Writing smart mutants is difficult • Side effects with integration tests 100
101. PERSONAL EXPERIENCE • Forces you to write tests • Always find bugs • Helpful for non-devs 101
102. 102 USE IT
103. MUTATION TESTING LIBRARIES HTTPS://GITHUB.COM/THEOFIDRY/AWESOME-MUTATION-TESTING 103
104. HTTPS://JOIND.IN/TALK/B4496 104
105. QUESTIONS? 105 https://imgflip.com/memegenerator/Shrek-Cat
106. THANK YOU! 106