Eeek, my tests are mutating

Presentation given for https://www.meetup.com/phpgent/events/242279704

Transcript

1. HI, I'M LANDER VANDERSTRAETEN 1
2. 3 QUIZ QUESTIONS 2
3. WHO THINKS UNIT TESTS ARE WASTE OF TIME? 3
5. WHO ALWAYS WRITES UNIT TESTS? 5
7. WHO ALREADY USES MUTATION TESTING? 7
9. CODE HAS BUGS TESTS ARE CODE TESTS HAVE BUGS! 9
11. TESTS ARE A GOOD TOOL TO VERIFY YOUR CODE BUT HOW DO YOU VERIFY YOUR TESTS? 11
12. CODE COVERAGE IS A START, BUT IT CAN GIVE A “GOOD” SCORE WITH USELESS TESTS 12
13. ALL TESTS PASSED! 13
14. HOW WE DETECT? 14
15. MUTATION TESTING 15
16. Mutation testing is a technique to evaluate the quality of your tests by programmatically mutating and making a series of small modifications to your code so that the tests no longer pass. 16
18. IF BUGS ARE CRIMES AND YOUR TESTS ARE THE POLICE MUTATIONS ARE FAKE CRIMES THAT LET YOU SEE THE POLICE IS DOING THEIR JOB 18
19. TO ASSESS THE QUALITY OF A GIVEN TEST, MUTANTS ARE EXECUTED AGAINST THE INPUT TEST TO SEE IF THE SEEDED FAULTS CAN BE DETECTED. 19
20. MUTATION, MUTANTS, MUTAFUCK WHAT? 20
21. Each mutated version: Mutant Mutated program + failing tests: killed mutant Mutated program + passed tests: escaped mutant 21
22. MEASURE ALL THE THINGS! 22
23. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants. 23
24. HOW DOES IT MODIFY? 24
25. MUTATORS 25
26. EXAMPLE public function foobar(int $number): int { return $number + 1; // original } public function foobar(int $number): int { return $number - 1; // mutated version } 26
27. ANOTHER EXAMPLE public function foobar(?Bar $foo): string { if (null === $foo) { // original return 'foo'; } // ... } public function foobar(?Bar $foo): string { if (null !== $foo) { // mutated version return 'foo'; } // ... } 27
28. DEMO TIME WITH INFECTION 28
29. WHAT DID WE LEARN TODAY? 29
30. 1. Always write unit tests 2. Code coverage gives a false positive feeling 3. Mutation testing also gives a false positive feeling 30
31. QUESTIONS? 31