Mutation Testing - Better code by making bugs

Mutation Testing - Better code by making bugs

Having tests is a good way to improve the internal quality of your application/project. They are however not free: you need to write, maintain and refactor them. So how do you make sure you have enough without overdoing it?

Discover Mutation Testing, fun tool helping you to test your tests by introducing bugs!

51b04191c1a7246b1ac7a0d52fe47acc?s=128

Théo FIDRY

March 31, 2018
Tweet

Transcript

  1. MUTATION TESTING Better code by making bugs 1

  2. ABOUT ME 2 @tfidry @theofidry Théo Fidry Web developer London

    (UK)
  3. PROJECTS 3 Alice Humbug & Infection
 PHP-Scoper & Box API-Platform

  4. LET’S DO A SHOW OF HANDS 4

  5. WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS

    INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5
  6. HOW TO DEFINE SOFTWARE QUALITY? 6

  7. 7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa

    The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality
  8. 8 EXTERNAL QUALITY

  9. EXTERNAL QUALITY • Conformity to the user expectation • Reliability

    • Accuracy • Ergonomics • Design • … 9
  10. THE QUALITY PERCEIVED BY THE USER 10

  11. 11 INTERNAL QUALITY https://insight.sensiolabs.com/projects/208b1a4c-4e7b-44b6-90a2-d7a2d91d431e/analyses/18744

  12. INTERNAL QUALITY • Maintainability • Concision • Cohesion • Simplicity

    • Clarity • … 12
  13. THE QUALITY PERCEIVED BY THE DEVELOPER 13

  14. SHOULD WE CARE? 14

  15. Cumulative functioncality Time 15 PROJECT STAMINA Payoff line https://martinfowler.com/bliki/DesignStaminaHypothesis.html

  16. THE LESS YOU CARE THE HARDER IT WILL BE TO

    ADD NEW FEATURES 16
  17. COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33

    US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ
  18. ORIGIN OF THE COSTS • Implementation • Debug • Repair

    • Technical debt • Delay: the extra you are paying for not fixing bugs earlier on 18
  19. ORIGIN OF THE COSTS 19 Technical debt Build cost Build

    cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA
  20. THE MORE YOU DELAY THE MORE EXPANSIVE IT GETS 20

  21. YOU SHOULD CARE 21

  22. HOW TO IMPROVE (INTERNAL) QUALITY? 22 BY ADDING TESTS

  23. TESTS ARE CODE 23

  24. TESTS ARE CODE • You need to write them •

    You need to make sure they work • You need to refactor them • You need to maintain them 24
  25. TESTS ARE EXPENSIVE 25

  26. PROBLEMS • How do I safely refactor my tests? •

    How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 26
  27. HOW DO I ASSESS THE QUALITY? 27

  28. 28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW

    VELOCITY TESTS QUALITY Unless you are Jakub or Marco Level of quality
  29. HOW DO I ASSESS THE QUALITY OF THE TEST SUITE?

    29
  30. COMMON ANSWERS • Don’t worry, it’ll be fine • I’m

    a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30
  31. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS

    BEEN TESTED 31
  32. 32 EXAMPLE https://github.com/theofidry/mutation-testing-demo

  33. 33

  34. 34 RUN PHPUNIT WITH COVERAGE REPORT 100% code coverage

  35. IS IT GOOD ENOUGH? 35

  36. 36 LET’S INTRODUCE A BUG https://github.com/theofidry/mutation-testing-demo

  37. 37 RUN PHPUNIT WITH COVERAGE REPORT

  38. OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 38

  39. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS

    BEEN TESTED 39
  40. WHAT CODE COVERAGE DOES TELL YOU 40

  41. EXECUTED ≠ TESTED Executed Tested 41

  42. CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED

    42
  43. 43 A TEST CASE IS MISSING https://github.com/theofidry/mutation-testing-demo

  44. 44

  45. HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 45

  46. INTRODUCE A BUG 46

  47. MUTATION TESTING 47

  48. CREATE A MUTANT SOURCE CODE MUTATOR MUTATION PROCESS MUTANT 48

  49. EXAMPLE OF A MUTANT 49

  50. MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo

    >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html
  51. COLLECT THE SOURCE FILES 51 Counter.php Foo.php FILE COLLECTOR

  52. GENERATE MUTANTS 52 Counter.php Foo.php MUTATOR MUTATOR MUTATOR MUTATOR MUTATOR

    MUTATOR MUTANTS
  53. GENERATED MUTANTS 53 … and more

  54. APPLY MUTANTS 54 PROCESS BUILDER MUTANT PROCESS WITH MUTATED CODE

    RESULT Runs tests TESTS RUNNER
  55. IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL,

    IT SURVIVED 55
  56. IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT

    WAS KILLED 56
  57. MUTATION SCORE Nbr of mutant killed Nbr of mutant generated

    Mutation score = 57
  58. CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION

    SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?
  59. DOES IT WORK? “Complex faults are coupled to simple faults

    in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59
  60. DEMO 60

  61. MUTATION TESTING IN PHP infection/infection 61 symfony/dependency-injection

  62. INSTALLATION 62 Infection installation guide: https://infection.github.io/guide/installation.html PHPUnit installation guide: https://phpunit.de/getting-started/phpunit-7.html

  63. 63 CONFIGURATION infection.json.dist https://infection.github.io/guide/usage.html#Configuration

  64. 64 RUNNING INFECTION $ php infection.phar

  65. 65 RESULT

  66. 66 REPORT

  67. 67 RUNNING ON DIFF https://blog.alejandrocelaya.com/2018/02/17/mutation-testing-with-infection-in-big-php-projects/

  68. DEMO DONE 68

  69. IT IS NOT NEW… - HISTORY • Begins in 1971,

    R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 69
  70. WHY IS IT NOT WIDELY USED? 70 http://knowyourmeme.com/memes/family-guy-why-are-we-not-funding-this

  71. WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing

    is not widely used yet (Although it is increasing) 71
  72. WHY IS IT NOT WIDELY USED? Integration Problem: Inability to

    successfully integrated it into software development process (TDD plays a key role now) 72
  73. WHY IS IT NOT WIDELY USED? Technical Problem: It is

    a brute force technique! 73
  74. BRUTE FORCE TECHNIQUE N: Number of tests M: Number of

    mutants NxM 74
  75. THEORETICAL RUN • 672 tests in 25.29 seconds 
 (0.03763s/test)

    • 3573 mutants • 2,401,056 tests • ~53h With basic Mutation Testing 75
  76. 76

  77. OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis

    • Parallelism • Equivalent mutant (~30%) • Different levels of requirements 77
  78. OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds 
 (0.03763s/test)

    • 3573 mutants • 2,401,056 tests • ~10min With Infection 78
  79. INFECTION IS STILL YOUNG 79

  80. FUTURE WORK • Test framework support • Equivalent mutant •

    Performance optimisations • Test framework support • Granular configuration (Profiles) 80
  81. WRAP UP 81 https://www.pinterest.com/pin/140526450845150336/

  82. THE GOOD PARTS • Gives you feedback on your tests

    • Test your tests with little effort • They are automatic • They discover dead code • Helps to refactor your tests 82
  83. THE NOT SO GOOD PARTS • Can be slow •

    Handful of young libraries • Writing complex mutant tests is difficult • Side effects with integration tests 83
  84. MUTATION TESTING LIBRARIES HTTPS://GITHUB.COM/THEOFIDRY/MUTATION-TESTING 84

  85. QUESTIONS? 85 https://imgflip.com/memegenerator/Shrek-Cat

  86. THANK YOU! 86