Tweet
Tweet

Slide 1

Slide 1 text

MUTATION TESTING Better code by making bugs 1

Slide 2

Slide 2 text

ABOUT ME 2 @tfidry @theofidry Théo Fidry Web developer Barcelona (Spain)

Slide 3

Slide 3 text

PROJECTS 3 Alice Humbug & Infection
 PHP-Scoper & Box API-Platform

Slide 4

Slide 4 text

LET’S DO A SHOW OF HANDS 4

Slide 5

Slide 5 text

WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5

Slide 6

Slide 6 text

HOW TO DEFINE SOFTWARE QUALITY? 6

Slide 7

Slide 7 text

7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality

Slide 8

Slide 8 text

8 (LACK OF) EXTERNAL QUALITY

Slide 9

Slide 9 text

EXTERNAL QUALITY • Conformity to the user expectation • Reliability • Accuracy • Ergonomics • Design • … 9

Slide 10

Slide 10 text

THE QUALITY PERCEIVED BY THE USER 10

Slide 11

Slide 11 text

11 (LACK OF) INTERNAL QUALITY https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744

Slide 12

Slide 12 text

INTERNAL QUALITY • Maintainability • Concision • Cohesion • Simplicity • Clarity • … 12

Slide 13

Slide 13 text

THE QUALITY PERCEIVED BY THE DEVELOPER (WHICH WE WILL FOCUS ON) 13

Slide 14

Slide 14 text

SHOULD WE CARE? 14

Slide 15

Slide 15 text

Cumulative functioncality Time 15 PROJECT STAMINA Payoff line https://martinfowler.com/bliki/DesignStaminaHypothesis.html

Slide 16

Slide 16 text

THE LESS YOU CARE THE HARDER IT WILL BE TO ADD NEW FEATURES 16

Slide 17

Slide 17 text

COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33 US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ

Slide 18

Slide 18 text

ORIGIN OF THE COSTS • Implementation • Debug & Repair • Technical debt • Delay: the extra you are paying for not fixing bugs earlier on 18

Slide 19

Slide 19 text

ORIGIN OF THE COSTS 19 Technical debt Build cost Build cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA

Slide 20

Slide 20 text

THE MORE YOU DELAY THE MORE EXPENSIVE IT GETS 20

Slide 21

Slide 21 text

YOU SHOULD CARE 21

Slide 22

Slide 22 text

HOW TO IMPROVE (INTERNAL) QUALITY? 22 (ONE) ANSWER: BY ADDING TESTS

Slide 23

Slide 23 text

TESTS ARE CODE 23

Slide 24

Slide 24 text

TESTS ARE CODE • You need to write them • You need to make sure they work • You need to refactor them • You need to maintain them 24

Slide 25

Slide 25 text

TESTS ARE EXPENSIVE 25 NOT FREE

Slide 26

Slide 26 text

PROBLEMS • How do I safely refactor my tests? • How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 26

Slide 27

Slide 27 text

HOW DO I ASSESS THE QUALITY? 27

Slide 28

Slide 28 text

28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW VELOCITY TESTS QUALITY Level of quality

Slide 29

Slide 29 text

HOW DO I ASSESS THE QUALITY OF THE TEST SUITE? 29

Slide 30

Slide 30 text

COMMON ANSWERS • Don’t worry, it’ll be fine • I’m a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30

Slide 31

Slide 31 text

LINE CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 31

Slide 32

Slide 32 text

32 EXAMPLE https://github.com/theofidry/mutation-testing-demo = 10) { $this->count++; } } }

Slide 33

Slide 33 text

33 https://github.com/theofidry/mutation-testing-demo function test_it_counts_number_above_ten(): void { $initialCount = $this->counter->count; $this->counter->count(100); $newCount = $this->counter->count; Assert::assertGreaterThan($initialCount, $newCount); } function test_it_does_not_count_number_below_ten(): void { $initialCount = $this->counter->count; $this->counter->count(7); $newCount = $this->counter->count; Assert::assertSame($initialCount, $newCount); }

Slide 34

Slide 34 text

34 RUN PHPUNIT WITH COVERAGE REPORT https://github.com/theofidry/mutation-testing-demo $ vendor/bin/phpunit --coverage-text \ --coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .. 2 / 2 (100%) Time: 113 ms, Memory: 6.00MB OK (2 tests, 2 assertions) Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3) 100% code coverage

Slide 35

Slide 35 text

IS IT GOOD ENOUGH? 35

Slide 36

Slide 36 text

36 LET’S INTRODUCE A BUG https://github.com/theofidry/mutation-testing-demo 10) { $this->count++; } } }

Slide 37

Slide 37 text

37 RUN PHPUNIT WITH COVERAGE REPORT https://github.com/theofidry/mutation-testing-demo $ vendor/bin/phpunit --coverage-text \ —-coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .. 2 / 2 (100%) Time: 113 ms, Memory: 6.00MB OK (2 tests, 2 assertions) Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3) DID NOT FAIL

Slide 38

Slide 38 text

OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 38

Slide 39

Slide 39 text

CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 39

Slide 40

Slide 40 text

WHAT CODE COVERAGE DOES TELL YOU 40

Slide 41

Slide 41 text

EXECUTED ≠ TESTED Executed Tested 41

Slide 42

Slide 42 text

CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED 42

Slide 43

Slide 43 text

43 A TEST CASE IS MISSING https://github.com/theofidry/mutation-testing-demo function test_it_counts_ten(): void { $initialCount = $this->counter->count; $this->counter->count(10); $newCount = $this->counter->count; Assert::assertGreaterThan($initialCount, $newCount); }

Slide 44

Slide 44 text

44 $ vendor/bin/phpunit --coverage-text \ --coverage-html=dist/coverage PHPUnit 7.0.2 by Sebastian Bergmann and contributors. .F. 3 / 3 (100%) Time: 70 ms, Memory: 6.00MB There was 1 failure: 1) Acme\CounterTest::test_it_counts_ten Failed asserting that 0 is greater than 0. /path/to/tests/CounterTest.php:38 FAILURES! Tests: 3, Assertions: 3, Failures: 1. Generating code coverage report in HTML format ... done Code Coverage Report: Summary: Classes: 100.00% (1/1) Methods: 100.00% (1/1) Lines: 100.00% (3/3) \Acme::Acme\Counter Methods: 100.00% ( 1/ 1) Lines: 100.00% ( 3/ 3)

Slide 45

Slide 45 text

HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 45

Slide 46

Slide 46 text

INTRODUCE A BUG 46

Slide 47

Slide 47 text

MUTATION TESTING 47

Slide 48

Slide 48 text

CREATE A MUTANT SOURCE CODE MUTATOR MUTATION PROCESS MUTANT 48

Slide 49

Slide 49 text

EXAMPLE OF A MUTANT 49 = 10) { $this->count++; } } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 10) { $this->count++; } }

Slide 50

Slide 50 text

MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html

Slide 51

Slide 51 text

COLLECT THE SOURCE FILES 51 Counter.php Foo.php FILE COLLECTOR

Slide 52

Slide 52 text

GENERATE MUTANTS 52 Counter.php Foo.php MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTANTS

Slide 53

Slide 53 text

--- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 9) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr >= 11) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { - if ($nbr >= 10) { + if ($nbr > 10) { $this->count++; } } --- a/src/Counter.php +++ b/src/Counter.php @@ -8,7 +8,7 @@ final class Counter public function count(int $nbr): void { if ($nbr >= 10) { - $this->count++; + $this->count—-; } } GENERATED MUTANTS 53 … and more

Slide 54

Slide 54 text

APPLY MUTANTS 54 PROCESS BUILDER MUTANT PROCESS WITH MUTATED CODE RESULT Runs tests TESTS RUNNER

Slide 55

Slide 55 text

IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL, IT SURVIVED 55

Slide 56

Slide 56 text

IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT WAS KILLED 56

Slide 57

Slide 57 text

MUTATION SCORE Nbr of mutant killed Nbr of mutant generated Mutation score = 57

Slide 58

Slide 58 text

CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?

Slide 59

Slide 59 text

DOES IT WORK? “Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59

Slide 60

Slide 60 text

MUTATION TESTING IN PHP infection/infection 60 humbug/humbug

Slide 61

Slide 61 text

DEMO 61

Slide 62

Slide 62 text

MUTATION TESTING IN PHP infection/infection 62 symfony/dependency-injection

Slide 63

Slide 63 text

SYMFONY/DEPENDENCY-INJECTION 63

Slide 64

Slide 64 text

64 INSTALLATION Infection installation guide: https://infection.github.io/guide/installation.html PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html $ git clone [email protected]:symfony/dependency-injection.git infection-demo $ cd infection-demo $ composer install $ wget -O infection https://github.com/infection/infection/releases/ download/0.10.3/infection.phar $ chmod +x infection $ wget -O phpunit https://phar.phpunit.de/phpunit-7.phar $ chmod /+x phpunit

Slide 65

Slide 65 text

PHPUNIT CONFIGURATION 65 phpunit.xml.dist ./Tests/ ./ ./Resources ./Tests ./vendor

Slide 66

Slide 66 text

66 RUNNING TESTS $ ./phpunit PHPUnit 7.0.2 by Sebastian Bergmann and contributors. ............................................................... 63 / 672 ( 9%) ............................................................... 126 / 672 ( 18%) ............................................................... 189 / 672 ( 28%) ............................................................... 252 / 672 ( 37%) ............................................................... 315 / 672 ( 46%) ............................................................... 378 / 672 ( 56%) ............................................................... 441 / 672 ( 65%) ............................................................... 504 / 672 ( 75%) ........S.SSSSS................................................ 567 / 672 ( 84%) ............................................................... 630 / 672 ( 93%) .......................................... 672 / 672 (100%) Time: 1.89 seconds, Memory: 26.00MB OK, but incomplete, skipped, or risky tests! Tests: 672, Assertions: 1550, Skipped: 6.

Slide 67

Slide 67 text

CONFIGURE INFECTION 67 $ ./infection init

Slide 68

Slide 68 text

INFECTION CONFIGURATION 68 { "timeout": 10, "source": { "directories": [ "." ], "excludes": [ "Tests", "vendor" ] }, "phpunit": { "customPath": "phpunit" }, "logs": { "text": "infection-log.txt" } } infection.json.dist https://infection.github.io/guide/usage.html#Configuration

Slide 69

Slide 69 text

69 RUNNING INFECTION $ ./infection You are running Infection with xdebug enabled. ____ ____ __ _ / _/___ / __/__ _____/ /_(_)___ ____ / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \ _/ // / / / __/ __/ /__/ /_/ / /_/ / / / / /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/ Running initial test suite... PHPUnit version: 7.0.2 678 [============================] 26 secs Generate mutants... Processing source code files: 144/144 Creating mutated files and processes: 3573/3573 .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573) E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573) ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573) .M...........M...........................M.MM..... ( 200 / 3573) ....MM..M.......M.....M...SSSSSSSS...M......M.M... ( 250 / 3573) .M.M.MEM..MST.E.M.......MMMM............SMMM.....E ( 300 / 3573) .E.......TM.........SM....................EM...... ( 350 / 3573) ....EM.M...E..MM.MM.S.MS......M.....EM...E...EE... ( 400 / 3573) ...TME......S................MM....M........E...M. ( 450 / 3573) ...ESSSSSSSSS...MM.SM....ESSMMMM.MSS..MM.SM.M.MMMS ( 500 / 3573) SS.S......M........S.S....ME...M.......M...S.M..S. ( 550 / 3573) ...MS.......S.M....E.E..EET.M.M.....M..MM.S....... ( 600 / 3573) .E.S.T.E..S..MM..M.EM.E..E......S.MM...MM...M.M... ( 650 / 3573) .....M..M......EM..ESS.S.M.............M.....E..MS ( 700 / 3573) ....EM.........EE...........MMME.....E.M..MM..M... ( 750 / 3573) M.......EE...........M.......MMEMM..M...MM.M..M... ( 800 / 3573) ..M....................MMM...M...MM.........MM...M ( 850 / 3573) .SSM.......MM..M......E..SEMM....E.M..M.E..EM..ETM ( 900 / 3573) M.......EM.EM...SSSSSSSSSSSSSSSSSSSSSSSE.S.M...... ( 950 / 3573) .S..M.........MM........S......EE.E......E....EEE. (1000 / 3573) .S....SSM..E..S.E....TMSSSMM.........M.........E.S (1050 / 3573) EE..M....STM..M.......M.S......SM..........S.SM..S (1100 / 3573) ....MMMMM.MMM....M.....M.......M.SM...M.MMM.S..... (1150 / 3573) .E....E.M.E.....MM..MMMMMS.M........MMMMMM.......E (1200 / 3573) .EE....S.......................T......EEM.M.MM.... (1250 / 3573) ..SSMM.MM..MMMM.MMMMMMMM.......................... (1300 / 3573) ..E....E.......E..SSSSM..M.S.E..M............MMM.M (1350 / 3573) ......M.SMMM...............M...M.M......M.M.E..... (1400 / 3573) .MSM.........S.MMMMMMMMMM..........M.............. (1450 / 3573) ................M...............S..M.............. (1500 / 3573) ....M...............M.........M................M.. (1550 / 3573) ...........................M...................... (1600 / 3573) ..E......M...M...........M.MM.....SSSS..MMM..MM... (1650 / 3573) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMSMSE....M..MMM (1700 / 3573) M.M...M.MM........MM..M...M.SSSSSSSSSSSMM.MSSSSSSM (1750 / 3573) MMSSSSSSSSSSSSSSSSSSSSSSSS...MM............MMS.... (1800 / 3573) MMMMMM..S....M......MMM...SSSSSSSSSSSSSSSSSSSSSSSS (1850 / 3573) SSSSSSSM..S.....M........MS....................... (1900 / 3573) ............SSSS..MM.SSSSSSSSSSSSSSSSSSSSSSSS..... (1950 / 3573) .M.M..........SS.M.....SSSSS.....SSS.............. (2000 / 3573) .....M.S..SSSSSSS..SM...SSSSSSSS.MSM....MM.....M.. (2050 / 3573) ..SSSSSS..MMMSSSSSSSSSS..S.....M.MSSSS........SSSS (2100 / 3573) S....SSSSSSSS.MMM.SMSSSMM.M.......SSS...M....M..M. (2150 / 3573) MSM.MMS..MMM.M...SSS........M.......SS.S..SSSSSSSS (2200 / 3573) SSSSSSSSSSSSSSSSSS.S..MMMSSSSMS......MS....MM..SSS (2250 / 3573) SS...M........M.SSSSSSMM..MMM.MMM...M..MMTMTTSSSMM (2300 / 3573) S..S...........MSMSSS...SSSSSSSSSSSSSSSSSSSSSSSSSS (2350 / 3573) SSSSSSMMMMMSS.M.MMM.SSSSSSSSSSSSSSSSSSS.E......... (2400 / 3573) .................................................. (2450 / 3573) ......SS.....MS.......M.S.E...........M.M......... (2500 / 3573) ..M.M......M............SS.S.......M.......E.SSSSS (2550 / 3573) .................S....S.SS.SSS.S.SSSSS..M....SSSSS (2600 / 3573) E.SMMSMM....MMSMM......MSSSS.MMSS.....MMSSSS.SSSE. (2650 / 3573) ..S....EMMM.E......E.EEEM........SSSSSSEESSM..SSMS (2700 / 3573) ........SSSM.....MS.M.MM..SSM..E....M.SS.E...M..M. (2750 / 3573) .M.MM.......M..........MMMMMMM..S.....SSM..MM..... (2800 / 3573) .........M........SSS....EE....M........M...M...MS (2850 / 3573) .M.MM..MM..M.E...M....E...S.SE..M........S..MM.... (2900 / 3573) ...M...MS...S.....SS..E.EM......M.E.S.....ES...... (2950 / 3573) .....MMM......MM......M.......E.S.....EE.S.....E.. (3000 / 3573) M..M..........S.MM.....S............M........M.... (3050 / 3573) .........MMM.M.........M.....MM.....S.SMMS.S...... (3100 / 3573) ....M....M.M....S...M..M....SSSM...........M...... (3150 / 3573) .MME.S.M..M.E......M.........MSMMM.S.......MM.S..S (3200 / 3573) ..MSSSSMMM.SS..........................M.S........ (3250 / 3573) .....S............................S......SMMM..SSS (3300 / 3573) ..........M.MM.......M...MSS.S.S....MM.M...M...... (3350 / 3573) S.S...MSM........S......S.......MM........MM...... (3400 / 3573) ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573) .................MM.... (3573 / 3573) 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Please note that some mutants will inevitably be harmless (i.e. fa 14min 21s

Slide 70

Slide 70 text

70 You are running Infection with xdebug enabled. $ ./infection ____ ____ __ _ / _/___ / __/__ _____/ /_(_)___ ____ / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \ _/ // / / / __/ __/ /__/ /_/ / /_/ / / / / /___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/ Running initial test suite... PHPUnit version: 7.0.2 678 [============================] 26 secs Generate mutants... Processing source code files: 144/144 Creating mutated files and processes: 3573/3573 .: killed, M: escaped, S: uncovered, E: fatal error, T: timed out .........M..MEEEM..MME.EEE.....MMM..MSE.....E..... ( 50 / 3573) E.E.ESS.E....SSSSSM.S...M........S.S.SS.....EM.E.. ( 100 / 3573) ....M..E....SMMMMMMMMMM...M..E.E..EM......M..MM... ( 150 / 3573)

Slide 71

Slide 71 text

71 ME..MMM..E..TM.....T....MS....E..E.E..T.E....MM... (3550 / 3573) .................MM.... (3573 / 3573) 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Please note that some mutants will inevitably be harmless (i.e. false positives).

Slide 72

Slide 72 text

72 Escaped mutants: ================ 1) /path/to/Alias.php:68 [M] This exec /path/to/php -c /tmp/dir/infectionyy4H8n /path/to/phpunit --configuration /tmp/dir/infection/ phpunitConfiguration.bce05.infection.xml --stop-on-failure --- Original +++ New @@ @@ $this->private = (bool) $boolean; - return $this; + return null; } /** * Whether this alias is private. * * @return bool */ PHPUnit 7.0.2 by Sebastian Bergmann and contributors. ............................................................... 63 / 127 ( 49%) ............................................................... 126 / 127 ( 99%) . 127 / 127 (100%) Time: 374 ms, Memory: 14.00MB OK (127 tests, 438 assertions) REPORT

Slide 73

Slide 73 text

TIPS 73

Slide 74

Slide 74 text

RESTRICT THE FILES IT’S RUNNING ON 74 $ infection —-filter=DependencyInjection INFECTION_FILTER=$(git diff ${GIT_PREVIOUS_SUCCESSFUL_COMMIT:-origin/develop} \ $GIT_COMMIT --name-only \ | grep /src/ \ | paste -sd "," -) $ infection —-filter=$INFECTION_FILTER https://infection.github.io/guide/command-line-options.html $ infection —-min-covered-msi 70

Slide 75

Slide 75 text

MAKE IT FASTER 75 $ infection —-threads=$(nproc) https://infection.github.io/guide/command-line-options.html $ infection —-only-covered Should this be the default? $(sysctl -n hw.ncpu)

Slide 76

Slide 76 text

PROFILES 76 { "mutators": { "@default": true, "@function_signature": false, "TrueValue": { "ignore": [ “NameSpace\\*\\SourceClass::create", "Full\\NameSpaced\\Class" ] } } } infection.json.dist https://infection.github.io/guide/profiles.html

Slide 77

Slide 77 text

RUN IT AGAIN 77 $ infection --threads=$(sysctl -n hw.ncpu) --only-covered

Slide 78

Slide 78 text

78 3573 mutations were generated: 2322 mutants were killed 569 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 69% Mutation Code Coverage: 84% Covered Code MSI: 82% Time: 14min 20s 3004 mutations were generated: 2322 mutants were killed 0 mutants were not covered by tests 526 covered mutants were not detected 140 errors were encountered 16 time outs were encountered Metrics: Mutation Score Indicator (MSI): 82% Mutation Code Coverage: 100% Covered Code MSI: 82% Time: 6min 39s

Slide 79

Slide 79 text

DEMO DONE 79

Slide 80

Slide 80 text

IT IS NOT NEW… - HISTORY • Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 80

Slide 81

Slide 81 text

WHY IS IT NOT WIDELY USED? 81 http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this

Slide 82

Slide 82 text

WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing is not widely used yet (Although it is increasing) 82

Slide 83

Slide 83 text

WHY IS IT NOT WIDELY USED? Integration Problem: Inability to successfully integrated it into software development process (TDD plays a key role now) 83

Slide 84

Slide 84 text

WHY IS IT NOT WIDELY USED? Technical Problem: It is a brute force technique! 84

Slide 85

Slide 85 text

BRUTE FORCE TECHNIQUE N: Number of tests M: Number of mutants NxM 85

Slide 86

Slide 86 text

THEORETICAL RUN • 672 tests in 25.29 seconds • 3573 mutants • 2,401,056 tests • ~53h With basic Mutation Testing 86

Slide 87

Slide 87 text

87

Slide 88

Slide 88 text

OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis • Parallelism • Equivalent mutant (~30%) • Different levels of requirements 88

Slide 89

Slide 89 text

OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds • 3573 mutants • ~6min With Infection 89

Slide 90

Slide 90 text

INFECTION IS STILL YOUNG 90

Slide 91

Slide 91 text

FUTURE WORK 91

Slide 92

Slide 92 text

PHP FRAMEWORKS SUPPORT • PHPUnit • PhpSpec • Behat (#147) • Atoum (#381) • CodeCeption (#45) 92 ✅ ✅

Slide 93

Slide 93 text

EQUIVALENT MUTANT 93 - $x = $a + $b; + $x = $a - (-$b); - if ($nbr >= 10) { + if ($nbr > 10) { - if ($nbr >= 10) { + if ($nbr >= 9) { ~30% of mutants

Slide 94

Slide 94 text

SMART MUTATORS 94

Slide 95

Slide 95 text

{ "mutators": { "@default": true, "@function_signature": false, "TrueValue": { "ignore": [ “NameSpace\\*\\SourceClass::create", "Full\\NameSpaced\\Class" ] } } } infection.json.dist GRANULAR CONFIGURATION 95

Slide 96

Slide 96 text

FALSE POSITIVE TRACKING 96

Slide 97

Slide 97 text

IO-BOUND FRAMEWORK 97

Slide 98

Slide 98 text

WRAP UP 98 https://www.pinterest.com/pin/140526450845150336/

Slide 99

Slide 99 text

THE GOOD PARTS • Gives you feedback on your tests • Test your tests with little effort • They are automatic • They discover dead code (e.g. useless tests) • Helps to refactor your tests 99

Slide 100

Slide 100 text

THE NOT SO GOOD PARTS • Can be slow • Handful of young libraries • Writing smart mutants is difficult • Side effects with integration tests 100

Slide 101

Slide 101 text

PERSONAL EXPERIENCE • Forces you to write tests • Always find bugs • Helpful for non-devs 101

Slide 102

Slide 102 text

102 USE IT

Slide 103

Slide 103 text

MUTATION TESTING LIBRARIES HTTPS://GITHUB.COM/THEOFIDRY/AWESOME-MUTATION-TESTING 103

Slide 104

Slide 104 text

HTTPS://JOIND.IN/TALK/B4496 104

Slide 105

Slide 105 text

QUESTIONS? 105 https://imgflip.com/memegenerator/Shrek-Cat

Slide 106

Slide 106 text

THANK YOU! 106