Mutation Testing - Better code by making bugs

Having tests is a good way to improve the internal quality of your application/project. They are however not free: you need to write, maintain and refactor them. So how do you make sure you have enough without overdoing it?

Discover Mutation Testing, fun tool helping you to test your tests by introducing bugs!

Transcript

1. MUTATION TESTING Better code by making bugs 1
2. ABOUT ME 2 @tfidry @theofidry Théo Fidry Web developer London (UK)
3. PROJECTS 3 Alice Humbug & Infection
 PHP-Scoper & Box API-Platform
4. LET’S DO A SHOW OF HANDS 4
5. WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5
6. HOW TO DEFINE SOFTWARE QUALITY? 6
7. 7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality
8. 8 EXTERNAL QUALITY
9. EXTERNAL QUALITY • Conformity to the user expectation • Reliability • Accuracy • Ergonomics • Design • … 9
10. THE QUALITY PERCEIVED BY THE USER 10
11. 11 INTERNAL QUALITY https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744
12. INTERNAL QUALITY • Maintainability • Concision • Cohesion • Simplicity • Clarity • … 12
13. THE QUALITY PERCEIVED BY THE DEVELOPER 13
14. SHOULD WE CARE? 14
15. Cumulative functioncality Time 15 PROJECT STAMINA Payoff line https://martinfowler.com/bliki/DesignStaminaHypothesis.html
16. THE LESS YOU CARE THE HARDER IT WILL BE TO ADD NEW FEATURES 16
17. COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33 US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ
18. ORIGIN OF THE COSTS • Implementation • Debug • Repair • Technical debt • Delay: the extra you are paying for not fixing bugs earlier on 18
19. ORIGIN OF THE COSTS 19 Technical debt Build cost Build cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA
20. THE MORE YOU DELAY THE MORE EXPANSIVE IT GETS 20
21. YOU SHOULD CARE 21
22. HOW TO IMPROVE (INTERNAL) QUALITY? 22 BY ADDING TESTS
23. TESTS ARE CODE 23
24. TESTS ARE CODE • You need to write them • You need to make sure they work • You need to refactor them • You need to maintain them 24
25. TESTS ARE EXPENSIVE 25
26. PROBLEMS • How do I safely refactor my tests? • How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 26
27. HOW DO I ASSESS THE QUALITY? 27
28. 28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW VELOCITY TESTS QUALITY Unless you are Jakub or Marco Level of quality
29. HOW DO I ASSESS THE QUALITY OF THE TEST SUITE? 29
30. COMMON ANSWERS • Don’t worry, it’ll be fine • I’m a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30
31. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 31
32. 32 EXAMPLE https://github.com/theofidry/mutation-testing-demo
34. 34 RUN PHPUNIT WITH COVERAGE REPORT 100% code coverage
35. IS IT GOOD ENOUGH? 35
36. 36 LET’S INTRODUCE A BUG https://github.com/theofidry/mutation-testing-demo
37. 37 RUN PHPUNIT WITH COVERAGE REPORT
38. OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 38
39. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 39
40. WHAT CODE COVERAGE DOES TELL YOU 40
41. EXECUTED ≠ TESTED Executed Tested 41
42. CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED 42
43. 43 A TEST CASE IS MISSING https://github.com/theofidry/mutation-testing-demo
45. HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 45
46. INTRODUCE A BUG 46
47. MUTATION TESTING 47
48. CREATE A MUTANT SOURCE CODE MUTATOR MUTATION PROCESS MUTANT 48
49. EXAMPLE OF A MUTANT 49
50. MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html
51. COLLECT THE SOURCE FILES 51 Counter.php Foo.php FILE COLLECTOR
52. GENERATE MUTANTS 52 Counter.php Foo.php MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR MUTANTS
53. GENERATED MUTANTS 53 … and more
54. APPLY MUTANTS 54 PROCESS BUILDER MUTANT PROCESS WITH MUTATED CODE RESULT Runs tests TESTS RUNNER
55. IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL, IT SURVIVED 55
56. IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT WAS KILLED 56
57. MUTATION SCORE Nbr of mutant killed Nbr of mutant generated Mutation score = 57
58. CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?
59. DOES IT WORK? “Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59
60. DEMO 60
61. MUTATION TESTING IN PHP infection/infection 61 symfony/dependency-injection
62. INSTALLATION 62 Infection installation guide: https://infection.github.io/guide/installation.html PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html
63. 63 CONFIGURATION infection.json.dist https://infection.github.io/guide/usage.html#Configuration
64. 64 RUNNING INFECTION $ php infection.phar
65. 65 RESULT
66. 66 REPORT
67. 67 RUNNING ON DIFF https://blog.alejandrocelaya.com/2018/02/17/mutation-testing-with-infection-in-big-php-projects/
68. DEMO DONE 68
69. IT IS NOT NEW… - HISTORY • Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 69
70. WHY IS IT NOT WIDELY USED? 70 http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this
71. WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing is not widely used yet (Although it is increasing) 71
72. WHY IS IT NOT WIDELY USED? Integration Problem: Inability to successfully integrated it into software development process (TDD plays a key role now) 72
73. WHY IS IT NOT WIDELY USED? Technical Problem: It is a brute force technique! 73
74. BRUTE FORCE TECHNIQUE N: Number of tests M: Number of mutants NxM 74
75. THEORETICAL RUN • 672 tests in 25.29 seconds 
 (0.03763s/test) • 3573 mutants • 2,401,056 tests • ~53h With basic Mutation Testing 75
77. OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis • Parallelism • Equivalent mutant (~30%) • Different levels of requirements 77
78. OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds 
 (0.03763s/test) • 3573 mutants • 2,401,056 tests • ~10min With Infection 78
79. INFECTION IS STILL YOUNG 79
80. FUTURE WORK • Test framework support • Equivalent mutant • Performance optimisations • Test framework support • Granular configuration (Profiles) 80
81. WRAP UP 81 https://www.pinterest.com/pin/140526450845150336/
82. THE GOOD PARTS • Gives you feedback on your tests • Test your tests with little effort • They are automatic • They discover dead code • Helps to refactor your tests 82
83. THE NOT SO GOOD PARTS • Can be slow • Handful of young libraries • Writing complex mutant tests is difficult • Side effects with integration tests 83
84. MUTATION TESTING LIBRARIES HTTPS://GITHUB.COM/THEOFIDRY/MUTATION-TESTING 84
85. QUESTIONS? 85 https://imgflip.com/memegenerator/Shrek-Cat
86. THANK YOU! 86