DevNexus 2020: Who Watches the Watchmen

Tests are watching our code, helping it to not regress. But who’s watching the watchmen? Code coverage is often used to determine the quality of automated tests, trying to “test the tests”. Unfortunately all it does is ensure which lines of code have been executed by the tests, not if they are actually checking anything! Mutation testing is a way to really watch the watchmen. Matthew will give an introduction to mutation testing, why it’s better than traditional code coverage for measuring test quality, and demo examples using Pitest.

Demo Code: https://github.com/MatthewYKnowles/dev-nexus-2020

Transcript

2. Matthew Knowles Software Engineer at Emmersion Learning www.EmmersionLearning.com @MatthewYKnowles Matthew.Knowles@EmmersionLearning.com Who Watches the Watchmen Introducing a Better Metric of your Tests’ Quality
4. Tests
5. Why Write Tests? • Verify functionality • Defend against unexpected change • Executable documentation of the code
6. What is driving more companies to adopt automated testing?
9. How is a test suite’s quality measured?
10. Code Coverage
12. How is code coverage calculated?
13. What are implications for the quality of the tests?
14. What does 100% code coverage mean?
15. Adding Functionality to Legacy Code •Code Coverage •Limitations of Code Coverage •Mutation Testing https://github.com/MatthewYKnowles/Refactoring-Exercise-Rentals
16. Mutation Testing
17. How would you mutate this code? 1. a < b 2. a || b 3. a == b 4. if(a > b) {return a+b} 5. if(b > a) {a++}
19. If mutation testing is so awesome, why has no one heard of it?
20. When is mutation testing useful?
21. As hard-nosed as I am about TDD as a necessary discipline; if I saw a team using mutation testing to guarantee the semantic stability of a test-after suite; I would smile, and nod, and consider them to be highly professional. (I would also suggest that they work test-first in order to streamline their effort.) -Robert Martin (Uncle Bob)
22. Mutation Testing Frameworks JavaScript – Stryker Java – Pitest C# - Stryker and Visual Mutator
23. Questions? Further Reading: Filip van Laenen – Mutation Testing: Better Code by Making Bugs Martin Fowler – Refactoring @MatthewYKnowles