Mutation Testing - Better code by making bugs

Transcript

1. MUTATION TESTING Better code by making bugs 1
2. ABOUT ME 2 @tfidry @theofidry
3. LET’S DO A SHOW OF HANDS
4. WHO DOES • UNIT TESTING • TEST-DRIVEN DEVELOPMENT • CONTINUOUS INTEGRATION • MEASURE CODE COVERAGE • MUTATION TESTING 4
5. PROBLEMS • How do I safely refactor my tests? • How do I know I can trust a test suite? • How do I ensure my team is writing effective tests • How do I know if I’ve retrofitted enough tests to safely refactor a piece of legacy code? 5
6. JUST ONE QUESTION HOW DO I ASSESS THE QUALITY OF THE TEST SUITE? 6
7. COMMON ANSWERS • Don’t worry, it’ll be fine • I’m a ninja rockstar, I know my tests are good • I do TDD, I know my tests are good • What about the tests you didn’t write? • How do you test drive changes to tests? • Code review • Inconsistent + Labour intensive • Code coverage 7
8. CODE COVERAGE MEASURE DOES NOT TELL YOU WHICH PART HAS BEEN TESTED 8
9. WHAT CODE COVERAGE DOES TELL YOU 9
10. EXECUTED ≠ TESTED Executed Tested 10
11. CODE COVERAGE TELLS YOU ONLY WHAT HAS NOT BEEN TESTED 11
12. WHERE THOSE LINES TESTED? 12
16. OUR TESTS STILL PASS. OUR TEST SUITE IS DEFICIENT 16
17. A TEST CASE IS MISSING 17
18. HOW TO DETECT IF A TEST SUITE IS DEFICIENT? 18
19. INTRODUCE A BUG 19
20. MUTATION TESTING IN PICTURES 20
21. CREATE A MUTANT SOURCE CODE MUTATION OPERATOR MUTATION PROCESS MUTANT 21
22. EXAMPLES • A + B —> A - B • A && B —> A || B • Literal value replacement: 0 -> 1, true -> false… • Statement deletion: return, break, continue… • … 22
23. TRY TO KILL THE MUTANT MUTANT TESTS - KILLERS TEST SUITE 23
24. IF A MUTANT DOES NOT CAUSE THE TESTS TO FAIL, IT SURVIVED 24
25. IF A MUTANT DOES CAUSE THE TESTS TO FAIL, IT WAS KILLED 25
26. MUTATION SCORE Nbr of mutant killed Nbr of mutant escaped Mutation score = 26
27. CODE COVERAGE HIGHLIGHTS CODE THAT IS DEFINITELY NOT TESTED MUTANT SCORE HIGHLIGHT CODE THAT IS DEFINITELY TESTED 27
28. DEMO HTTPS://GITHUB.COM/THEOFIDRY/MUTATION-TESTING-DEMO 28
29. MUTATION TESTING IN PHP humbug/humbug infection/infection 29
32. composer require humbug/humbug:^1.0@beta vendor/bin/humbug 32
36. WHY IS IT NOT WIDELY USED? 36
37. WHY DOES IT WORK? “Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults” Demonstrated in 1995 by K. Wah, “Fault coupling in finite bijective functions” 37
38. IT IS NOT NEW… - HISTORY • Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs” • Generally accepted in 1978, R. Lipton and al, “Hints on test data selection: Help for the practicing programmer” 38
39. WHY IS IT NOT WIDELY USED? Maturity Problem: Because testing is not widely used yet (Although it is increasing) 39
40. WHY IS IT NOT WIDELY USED? Integration Problem: Inability to successfully integrated it into software development process (TDD plays a key role now) 40
41. WHY IS IT NOT WIDELY USED? Technical Problem: It is a brute force technique! 41
42. BRUTE FORCE TECHNIQUE N: Number of tests M: Number of mutants NxM 42
43. OPTIMISATION STRATEGIES • 2,155 tests in 6.82 seconds 
 (0.00316s/tests) • 957 mutants • 2,062,335 tests • ~1h49min With Humbug 43
45. OPTIMISATION STRATEGIES • Reduce equivalent mutant detection techniques (10 to 40% of mutants are equivalent) • Take coverage to mutate only covered code • Rely on coverage to know which tests to run and which to run first • Increment analysis 45
46. OPTIMISATION STRATEGIES • 2,155 tests in 6.82 seconds 
 (0.00316s/tests) • 957 mutants • 2,062,335 tests • 13min48 (vs. ~1h49min) With Humbug 46
47. FUTURE WORK IN HUMBUG • Humbug vs. Infection • AST based mutations • Performance optimisations • Test framework support 47
48. WRAP UP 48
49. THE GOOD PARTS • Detect problems with your tests • They discover dead code • They are automatic • Test your tests with little effort 49
50. THE NOT SO GOOD PARTS • Mutation testing is slow • Handful of libraries • Equivalent mutant problem • Writing complex mutant tests is difficult 50
51. MUTATION TESTING LIBRARIES HTTPS://GITHUB.COM/THEOFIDRY/MUTATION-TESTING 51
52. QUESTIONS? 52
53. THANK YOU! 53