$30 off During Our Annual Pro Sale. View Details »

Mutation testing 101

Mutation testing 101

OK, so... you write tests to verify that your code is doing what it is expected to do. But who verifies that the tests are "good"? We all know that test coverage means "almost" nothing, so... Who watches the watchmen?

This will be an introduction talk and demo to mutation testing 🙂

Isidro López

June 21, 2019
Tweet

More Decks by Isidro López

Other Decks in Programming

Transcript

  1. Mutation testing 101
    Form3 Friday Sharing
    25th June 2019

    View Slide

  2. View Slide

  3. https://i.ytimg.com/vi/QUyPALAJFpU/maxresdefault.jpg

    View Slide

  4. “Internal” vs “External” quality
    The system / component / whatever behaves as expected...

    https://martinfowler.com/articles/microservice-testing/

    Specification by example / ATDD / BDD

    Example mapping

    Functional testing

    Contract testing

    Exploration testing

    Performance testing

    Penetration testing

    etc...

    View Slide

  5. View Slide

  6. Metrics, metrics everywhere...

    Cyclomatic complexity

    Duplications

    Code smells

    Maintainability

    Technical debt

    Test duration

    Test coverage: are we covering all the “possible” behaviours of the system?
    https://sonarcloud.io/dashboard?id=com.islomar.payments%3Apayments-api

    View Slide

  7. “Tell me how you measure me,
    and I will tell you how I will behave”
    Eliyahu M. Goldratt
    “You get what you measure”

    View Slide

  8. assertTrue(true);

    View Slide

  9. View Slide

  10. https://www.wallpaperup.com/865120/X-MEN_superhero_marvel_action_adventure_sci-fi_warrior_fantasy_fighting_hero_xmen_1xmena_com
    ics_poster.html
    Mutation testing to the rescue!!

    View Slide

  11. https://www.guru99.com/mutation-testing.html

    View Slide

  12. Simple examples of mutators

    Conditional boundaries

    Increments

    Invert negatives

    Math

    Negation

    Return values

    Void method calls

    Constructor calls

    Empty returns

    False/True returns

    Remove conditionals

    etc...

    View Slide

  13. Conditional boundaries mutators
    http://pitest.org/quickstart/mutators/

    View Slide

  14. Negate conditional mutators
    http://pitest.org/quickstart/mutators/

    View Slide

  15. Math mutators
    http://pitest.org/quickstart/mutators/

    View Slide

  16. Return values mutator

    View Slide

  17. Void method call mutator
    http://pitest.org/quickstart/mutators/

    View Slide

  18. Demo time

    View Slide

  19. Demo time (in Java… oopsss!!)
    https://github.com/islomar/payments-api
    https://github.com/islomar/parrotter
    https://github.com/sandromancuso/Gilded-Rose-Kata (100% test coverage)

    View Slide

  20. Final considerations

    High test coverage means almost NOTHING. Low test coverage is a smell...

    Mutation testing: only for isolated tests (aka “unit tests”): tests which are fast
    (FIRST)

    Start with the basic mutators

    Depending on the complexity and size of the code, it can be very time/resource
    consuming, watch out!! (combinatorial explosion)

    In Pitest you can activate “incremental analysis” (experimental feature)

    View Slide

  21. Resources (I)
    GOTO 2015: Mutation testing in Python
    https://www.youtube.com/watch?v=jwB3Nn4hR1o
    Mutation testing in Python with Cosmic Ray
    https://vimeo.com/171319754
    Vicenç García en SCBCN (in Spanish)
    https://www.youtube.com/watch?v=LWL71b2fc64
    Madrid JUG (in Spanish)
    https://www.youtube.com/watch?v=hk_urzWGqrE

    View Slide

  22. Resources (II)
    https://www.slideshare.net/RafaelVindelAmor/who-watches-the-watchmen-mutation-te
    sting-75767706
    https://www.st.cs.uni-saarland.de/edu/testingdebugging10/slides/10-MutationTesting.pdf
    https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78

    View Slide

  23. https://looneytunes.fandom.com/wiki/That%27s_all_Folks

    View Slide

  24. View Slide