Mutation testing 101

OK, so... you write tests to verify that your code is doing what it is expected to do. But who verifies that the tests are "good"? We all know that test coverage means "almost" nothing, so... Who watches the watchmen?

This will be an introduction talk and demo to mutation testing 🙂

Transcript

1. Mutation testing 101
   Form3 Friday Sharing
   25th June 2019
   

   View Slide

2. View Slide

3. https://i.ytimg.com/vi/QUyPALAJFpU/maxresdefault.jpg
   

   View Slide

4. “Internal” vs “External” quality
   The system / component / whatever behaves as expected...
   ●
   https://martinfowler.com/articles/microservice-testing/
   ●
   Specification by example / ATDD / BDD
   ●
   Example mapping
   ●
   Functional testing
   ●
   Contract testing
   ●
   Exploration testing
   ●
   Performance testing
   ●
   Penetration testing
   ●
   etc...
   

   View Slide

5. View Slide

6. Metrics, metrics everywhere...
   ●
   Cyclomatic complexity
   ●
   Duplications
   ●
   Code smells
   ●
   Maintainability
   ●
   Technical debt
   ●
   Test duration
   ●
   Test coverage: are we covering all the “possible” behaviours of the system?
   https://sonarcloud.io/dashboard?id=com.islomar.payments%3Apayments-api
   

   View Slide

7. “Tell me how you measure me,
   and I will tell you how I will behave”
   Eliyahu M. Goldratt
   “You get what you measure”
   

   View Slide

8. assertTrue(true);
   

   View Slide

9. View Slide

10. https://www.wallpaperup.com/865120/X-MEN_superhero_marvel_action_adventure_sci-fi_warrior_fantasy_fighting_hero_xmen_1xmena_com
    ics_poster.html
    Mutation testing to the rescue!!
    

    View Slide

11. https://www.guru99.com/mutation-testing.html
    

    View Slide

12. Simple examples of mutators
    ●
    Conditional boundaries
    ●
    Increments
    ●
    Invert negatives
    ●
    Math
    ●
    Negation
    ●
    Return values
    ●
    Void method calls
    ●
    Constructor calls
    ●
    Empty returns
    ●
    False/True returns
    ●
    Remove conditionals
    ●
    etc...
    

    View Slide

13. Conditional boundaries mutators
    http://pitest.org/quickstart/mutators/
    

    View Slide

14. Negate conditional mutators
    http://pitest.org/quickstart/mutators/
    

    View Slide

15. Math mutators
    http://pitest.org/quickstart/mutators/
    

    View Slide

16. Return values mutator
    

    View Slide

17. Void method call mutator
    http://pitest.org/quickstart/mutators/
    

    View Slide

18. Demo time
    

    View Slide

19. Demo time (in Java… oopsss!!)
    https://github.com/islomar/payments-api
    https://github.com/islomar/parrotter
    https://github.com/sandromancuso/Gilded-Rose-Kata (100% test coverage)
    

    View Slide

20. Final considerations
    ●
    High test coverage means almost NOTHING. Low test coverage is a smell...
    ●
    Mutation testing: only for isolated tests (aka “unit tests”): tests which are fast
    (FIRST)
    ●
    Start with the basic mutators
    ●
    Depending on the complexity and size of the code, it can be very time/resource
    consuming, watch out!! (combinatorial explosion)
    ○
    In Pitest you can activate “incremental analysis” (experimental feature)
    

    View Slide

21. Resources (I)
    GOTO 2015: Mutation testing in Python
    https://www.youtube.com/watch?v=jwB3Nn4hR1o
    Mutation testing in Python with Cosmic Ray
    https://vimeo.com/171319754
    Vicenç García en SCBCN (in Spanish)
    https://www.youtube.com/watch?v=LWL71b2fc64
    Madrid JUG (in Spanish)
    https://www.youtube.com/watch?v=hk_urzWGqrE
    

    View Slide

22. Resources (II)
    https://www.slideshare.net/RafaelVindelAmor/who-watches-the-watchmen-mutation-te
    sting-75767706
    https://www.st.cs.uni-saarland.de/edu/testingdebugging10/slides/10-MutationTesting.pdf
    https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78
    

    View Slide

23. https://looneytunes.fandom.com/wiki/That%27s_all_Folks
    

    View Slide

24. View Slide