Mutation Testing - Better code by making bugs

Slide 1

Slide 1 text

MUTATION TESTING Better code by making bugs 1

Slide 2

Slide 2 text

ABOUT ME 2 @tﬁdry @theoﬁdry Théo Fidry Web developer London (UK)

Slide 3

Slide 3 text

PROJECTS 3 Alice Humbug & Infection  PHP-Scoper & Box API-Platform

Slide 4

Slide 4 text

LET’S DO A SHOW OF HANDS 4

Slide 5

Slide 5 text

WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5

Slide 6

Slide 6 text

HOW TO DEFINE SOFTWARE QUALITY? 6

Slide 7

Slide 7 text

7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality

Slide 8

Slide 8 text

8 EXTERNAL QUALITY

Slide 9

Slide 9 text

EXTERNAL QUALITY • Conformity to the user expectation • Reliability • Accuracy • Ergonomics • Design • … 9

Slide 10

Slide 10 text

THE QUALITY PERCEIVED BY THE USER 10

Slide 11

Slide 11 text

11 INTERNAL QUALITY https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744

Slide 12

Slide 12 text

INTERNAL QUALITY • Maintainability • Concision • Cohesion • Simplicity • Clarity • … 12

Slide 13

Slide 13 text

THE QUALITY PERCEIVED BY THE DEVELOPER 13

Slide 14

Slide 14 text

SHOULD WE CARE? 14

Slide 15

Slide 15 text

Cumulative functioncality Time 15 PROJECT STAMINA Payoﬀ line https://martinfowler.com/bliki/DesignStaminaHypothesis.html

Slide 16

Slide 16 text

THE LESS YOU CARE THE HARDER IT WILL BE TO ADD NEW FEATURES 16

Slide 17

Slide 17 text

COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33 US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ

Slide 18

Slide 18 text

ORIGIN OF THE COSTS • Implementation • Debug • Repair • Technical debt • Delay: the extra you are paying for not ﬁxing bugs earlier on 18

Slide 19

Slide 19 text

ORIGIN OF THE COSTS 19 Technical debt Build cost Build cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA

Slide 20

Slide 20 text

THE MORE YOU DELAY THE MORE EXPANSIVE IT GETS 20

Slide 21

Slide 21 text

YOU SHOULD CARE 21

Slide 22

Slide 22 text

HOW TO IMPROVE (INTERNAL) QUALITY? 22 BY ADDING TESTS

Slide 23

Slide 23 text

TESTS ARE CODE 23

Slide 24

Slide 24 text

TESTS ARE CODE • You need to write them • You need to make sure they work • You need to refactor them • You need to maintain them 24

Slide 25

Slide 25 text

TESTS ARE EXPENSIVE 25

Slide 26

Slide 26 text

PROBLEMS • How do I safely refactor my tests? • How do I know I can trust a test suite? • How do I ensure my team is writing eﬀective tests • How do I know if I’ve retroﬁtted enough tests to safely refactor a piece of legacy code? 26

Slide 27

Slide 27 text

HOW DO I ASSESS THE QUALITY? 27

Slide 28

Slide 28 text

28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW VELOCITY TESTS QUALITY Unless you are Jakub or Marco Level of quality

Slide 29

Slide 29 text

HOW DO I ASSESS THE QUALITY OF THE TEST SUITE? 29

Slide 30

Slide 30 text

COMMON ANSWERS • Don’t worry, it’ll be ﬁne • I’m a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30

Slide 31

Slide 31 text

CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 31

Slide 32

Slide 32 text

32 EXAMPLE https://github.com/theoﬁdry/mutation-testing-demo

Slide 33

Slide 33 text

Slide 34

Slide 34 text

34 RUN PHPUNIT WITH COVERAGE REPORT 100% code coverage

Slide 35

Slide 35 text

IS IT GOOD ENOUGH? 35

Slide 36

Slide 36 text

36 LET’S INTRODUCE A BUG https://github.com/theoﬁdry/mutation-testing-demo

Slide 37

Slide 37 text

37 RUN PHPUNIT WITH COVERAGE REPORT

Slide 38

Slide 38 text

OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 38

Slide 39

Slide 39 text

CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 39

Slide 40

Slide 40 text

WHAT CODE COVERAGE DOES TELL YOU 40

Slide 41

Slide 41 text

EXECUTED ≠ TESTED Executed Tested 41

Slide 42

Slide 42 text

CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED 42

Slide 43

Slide 43 text

43 A TEST CASE IS MISSING https://github.com/theoﬁdry/mutation-testing-demo

Slide 44

Slide 44 text

Slide 45

Slide 45 text

HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 45

Slide 46

Slide 46 text

INTRODUCE A BUG 46

Slide 47

Slide 47 text

MUTATION TESTING 47

Slide 48

Slide 48 text

CREATE A MUTANT SOURCE CODE MUTATOR MUTATION PROCESS MUTANT 48

Slide 49

Slide 49 text

EXAMPLE OF A MUTANT 49

Slide 50

Slide 50 text

MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html

Slide 51

Slide 51 text

COLLECT THE SOURCE FILES 51 Counter.php Foo.php FILE COLLECTOR

Slide 52

Slide 52 text

GENERATE MUTANTS 52 Counter.php Foo.php MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTANTS

Slide 53

Slide 53 text

GENERATED MUTANTS 53 … and more

Slide 54

Slide 54 text

APPLY MUTANTS 54 PROCESS BUILDER MUTANT PROCESS WITH MUTATED CODE RESULT Runs tests TESTS RUNNER

Slide 55

Slide 55 text

IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL, IT SURVIVED 55

Slide 56

Slide 56 text

IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT WAS KILLED 56

Slide 57

Slide 57 text

MUTATION SCORE Nbr of mutant killed Nbr of mutant generated Mutation score = 57

Slide 58

Slide 58 text

CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?

Slide 59

Slide 59 text

DOES IT WORK? “Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59

Slide 60

Slide 60 text

DEMO 60

Slide 61

Slide 61 text

MUTATION TESTING IN PHP infection/infection 61 symfony/dependency-injection

Slide 62

Slide 62 text

INSTALLATION 62 Infection installation guide: https://infection.github.io/guide/installation.html PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html

Slide 63

Slide 63 text

63 CONFIGURATION infection.json.dist https://infection.github.io/guide/usage.html#Conﬁguration

Slide 64

Slide 64 text

64 RUNNING INFECTION $ php infection.phar

Slide 65

Slide 65 text

65 RESULT

Slide 66

Slide 66 text

66 REPORT

Slide 67

Slide 67 text

67 RUNNING ON DIFF https://blog.alejandrocelaya.com/2018/02/17/mutation-testing-with-infection-in-big-php-projects/

Slide 68

Slide 68 text

DEMO DONE 68

Slide 69

Slide 69 text

IT IS NOT NEW… - HISTORY • Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 69

Slide 70

Slide 70 text

WHY IS IT NOT WIDELY USED? 70 http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this

Slide 71

Slide 71 text

WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing is not widely used yet (Although it is increasing) 71

Slide 72

Slide 72 text

WHY IS IT NOT WIDELY USED? Integration Problem: Inability to successfully integrated it into software development process (TDD plays a key role now) 72

Slide 73

Slide 73 text

WHY IS IT NOT WIDELY USED? Technical Problem: It is a brute force technique! 73

Slide 74

Slide 74 text

BRUTE FORCE TECHNIQUE N: Number of tests M: Number of mutants NxM 74

Slide 75

Slide 75 text

THEORETICAL RUN • 672 tests in 25.29 seconds   (0.03763s/test) • 3573 mutants • 2,401,056 tests • ~53h With basic Mutation Testing 75

Slide 76

Slide 76 text

Slide 77

Slide 77 text

OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis • Parallelism • Equivalent mutant (~30%) • Diﬀerent levels of requirements 77

Slide 78

Slide 78 text

OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds   (0.03763s/test) • 3573 mutants • 2,401,056 tests • ~10min With Infection 78

Slide 79

Slide 79 text

INFECTION IS STILL YOUNG 79

Slide 80

Slide 80 text

FUTURE WORK • Test framework support • Equivalent mutant • Performance optimisations • Test framework support • Granular conﬁguration (Proﬁles) 80

Slide 81

Slide 81 text

WRAP UP 81 https://www.pinterest.com/pin/140526450845150336/

Slide 82

Slide 82 text

THE GOOD PARTS • Gives you feedback on your tests • Test your tests with little eﬀort • They are automatic • They discover dead code • Helps to refactor your tests 82

Slide 83

Slide 83 text

THE NOT SO GOOD PARTS • Can be slow • Handful of young libraries • Writing complex mutant tests is diﬃcult • Side eﬀects with integration tests 83