Mutation testing 101

OK, so... you write tests to verify that your code is doing what it is expected to do. But who verifies that the tests are "good"? We all know that test coverage means "almost" nothing, so... Who watches the watchmen?

This will be an introduction talk and demo to mutation testing 🙂

Transcript

1. Mutation testing 101 Form3 Friday Sharing 25th June 2019
3. https://i.ytimg.com/vi/QUyPALAJFpU/maxresdefault.jpg
4. “Internal” vs “External” quality The system / component / whatever behaves as expected... • https://martinfowler.com/articles/microservice-testing/ • Specification by example / ATDD / BDD • Example mapping • Functional testing • Contract testing • Exploration testing • Performance testing • Penetration testing • etc...
6. Metrics, metrics everywhere... • Cyclomatic complexity • Duplications • Code smells • Maintainability • Technical debt • Test duration • Test coverage: are we covering all the “possible” behaviours of the system? https://sonarcloud.io/dashboard?id=com.islomar.payments%3Apayments-api
7. “Tell me how you measure me, and I will tell you how I will behave” Eliyahu M. Goldratt “You get what you measure”
8. assertTrue(true);
10. https://www.wallpaperup.com/865120/X-MEN_superhero_marvel_action_adventure_sci-fi_warrior_fantasy_fighting_hero_xmen_1xmena_com ics_poster.html Mutation testing to the rescue!!
11. https://www.guru99.com/mutation-testing.html
12. Simple examples of mutators • Conditional boundaries • Increments • Invert negatives • Math • Negation • Return values • Void method calls • Constructor calls • Empty returns • False/True returns • Remove conditionals • etc...
13. Conditional boundaries mutators http://pitest.org/quickstart/mutators/
14. Negate conditional mutators http://pitest.org/quickstart/mutators/
15. Math mutators http://pitest.org/quickstart/mutators/
16. Return values mutator
17. Void method call mutator http://pitest.org/quickstart/mutators/
18. Demo time
19. Demo time (in Java… oopsss!!) https://github.com/islomar/payments-api https://github.com/islomar/parrotter https://github.com/sandromancuso/Gilded-Rose-Kata (100% test coverage)
20. Final considerations • High test coverage means almost NOTHING. Low test coverage is a smell... • Mutation testing: only for isolated tests (aka “unit tests”): tests which are fast (FIRST) • Start with the basic mutators • Depending on the complexity and size of the code, it can be very time/resource consuming, watch out!! (combinatorial explosion) ◦ In Pitest you can activate “incremental analysis” (experimental feature)
21. Resources (I) GOTO 2015: Mutation testing in Python https://www.youtube.com/watch?v=jwB3Nn4hR1o Mutation testing in Python with Cosmic Ray https://vimeo.com/171319754 Vicenç García en SCBCN (in Spanish) https://www.youtube.com/watch?v=LWL71b2fc64 Madrid JUG (in Spanish) https://www.youtube.com/watch?v=hk_urzWGqrE
22. Resources (II) https://www.slideshare.net/RafaelVindelAmor/who-watches-the-watchmen-mutation-te sting-75767706 https://www.st.cs.uni-saarland.de/edu/testingdebugging10/slides/10-MutationTesting.pdf https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78
23. https://looneytunes.fandom.com/wiki/That%27s_all_Folks