Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mutation Testing - Better code by making bugs

Mutation Testing - Better code by making bugs

Having tests is a good way to improve the internal quality of your application/project. They are however not free: you need to write, maintain and refactor them. So how do you make sure you have enough without overdoing it?

Discover Mutation Testing, fun tool helping you to test your tests by introducing bugs!

Théo FIDRY

March 31, 2018
Tweet

More Decks by Théo FIDRY

Other Decks in Programming

Transcript

  1. WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS

    INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 5
  2. 7 Marcello Duarte Creator of PhpSpec ex-Head of Training @Inviqa

    The extent to what the software takes into account what matters most for the customer & the maintainability of the source code DEFINING SOFTWARE QUALITY Internal Quality External Quality https://speakerdeck.com/jakzal/building-in-quality
  3. EXTERNAL QUALITY • Conformity to the user expectation • Reliability

    • Accuracy • Ergonomics • Design • … 9
  4. COST OF BUG / PHASE 17 http://www.ifpug.org/Documents/Jones-CostPerDefectMetricVersion4.pdf Cost US$1,666.67 US$3,333.33

    US$5,000.00 US$6,666.67 US$8,333.33 US$10,000.00 Development phase Dev CI/Review QA Prod US$100 US$500 US$1,500 US$10,000 https://plus.google.com/u/1/+LaurentBossavit/posts/8QLBPXA9miZ
  5. ORIGIN OF THE COSTS • Implementation • Debug • Repair

    • Technical debt • Delay: the extra you are paying for not fixing bugs earlier on 18
  6. ORIGIN OF THE COSTS 19 Technical debt Build cost Build

    cost Cost of delay Technical debt Cost of debug Cost of repair Right feature built wrong Right feature IDEA
  7. TESTS ARE CODE • You need to write them •

    You need to make sure they work • You need to refactor them • You need to maintain them 24
  8. PROBLEMS • How do I safely refactor my tests? •

    How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 26
  9. 28 NO TEST MAX TEST SHORT-TERM HIGH VELOCITY SHORT-TERM LOW

    VELOCITY TESTS QUALITY Unless you are Jakub or Marco Level of quality
  10. COMMON ANSWERS • Don’t worry, it’ll be fine • I’m

    a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 30
  11. 33

  12. 44

  13. MUTATOR EXAMPLES 50 Name Original Mutated Plus + - GreaterThanOrEqualTo

    >= > Spaceship $a <=> $b $b <=> $a TrueValue return true; return false; https://infection.github.io/guide/mutators.html
  14. CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTATION

    SCORE HIGHLIGHTS CODE THAT IS DEFINITELY TESTED 58 HOW TO DETECT IF A TEST SUITE IS DEFICIENT?
  15. DOES IT WORK? “Complex faults are coupled to simple faults

    in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 59
  16. IT IS NOT NEW… - HISTORY • Begins in 1971,

    R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 69
  17. WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing

    is not widely used yet (Although it is increasing) 71
  18. WHY IS IT NOT WIDELY USED? Integration Problem: Inability to

    successfully integrated it into software development process (TDD plays a key role now) 72
  19. THEORETICAL RUN • 672 tests in 25.29 seconds 
 (0.03763s/test)

    • 3573 mutants • 2,401,056 tests • ~53h With basic Mutation Testing 75
  20. 76

  21. OPTIMISATION STRATEGIES • Mutate only covered code • Incremental analysis

    • Parallelism • Equivalent mutant (~30%) • Different levels of requirements 77
  22. OPTIMISATION STRATEGIES • 672 tests in 25.29 seconds 
 (0.03763s/test)

    • 3573 mutants • 2,401,056 tests • ~10min With Infection 78
  23. FUTURE WORK • Test framework support • Equivalent mutant •

    Performance optimisations • Test framework support • Granular configuration (Profiles) 80
  24. THE GOOD PARTS • Gives you feedback on your tests

    • Test your tests with little effort • They are automatic • They discover dead code • Helps to refactor your tests 82
  25. THE NOT SO GOOD PARTS • Can be slow •

    Handful of young libraries • Writing complex mutant tests is difficult • Side effects with integration tests 83