Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mutation testing 101

Mutation testing聽101

OK, so... you write tests to verify that your code is doing what it is expected to do. But who verifies that the tests are "good"? We all know that test coverage means "almost" nothing, so... Who watches the watchmen?

This will be an introduction talk and demo to mutation testing 馃檪

Isidro L贸pez

June 21, 2019

More Decks by Isidro L贸pez

Other Decks in Programming


  1. 鈥淚nternal鈥 vs 鈥淓xternal鈥 quality The system / component / whatever

    behaves as expected... 鈥 https://martinfowler.com/articles/microservice-testing/ 鈥 Speci铿乧ation by example / ATDD / BDD 鈥 Example mapping 鈥 Functional testing 鈥 Contract testing 鈥 Exploration testing 鈥 Performance testing 鈥 Penetration testing 鈥 etc...
  2. Metrics, metrics everywhere... 鈥 Cyclomatic complexity 鈥 Duplications 鈥 Code

    smells 鈥 Maintainability 鈥 Technical debt 鈥 Test duration 鈥 Test coverage: are we covering all the 鈥減ossible鈥 behaviours of the system? https://sonarcloud.io/dashboard?id=com.islomar.payments%3Apayments-api
  3. 鈥淭ell me how you measure me, and I will tell

    you how I will behave鈥 Eliyahu M. Goldratt 鈥淵ou get what you measure鈥
  4. Simple examples of mutators 鈥 Conditional boundaries 鈥 Increments 鈥

    Invert negatives 鈥 Math 鈥 Negation 鈥 Return values 鈥 Void method calls 鈥 Constructor calls 鈥 Empty returns 鈥 False/True returns 鈥 Remove conditionals 鈥 etc...
  5. Final considerations 鈥 High test coverage means almost NOTHING. Low

    test coverage is a smell... 鈥 Mutation testing: only for isolated tests (aka 鈥渦nit tests鈥): tests which are fast (FIRST) 鈥 Start with the basic mutators 鈥 Depending on the complexity and size of the code, it can be very time/resource consuming, watch out!! (combinatorial explosion) 鈼 In Pitest you can activate 鈥渋ncremental analysis鈥 (experimental feature)
  6. Resources (I) GOTO 2015: Mutation testing in Python https://www.youtube.com/watch?v=jwB3Nn4hR1o Mutation

    testing in Python with Cosmic Ray https://vimeo.com/171319754 Vicen莽 Garc铆a en SCBCN (in Spanish) https://www.youtube.com/watch?v=LWL71b2fc64 Madrid JUG (in Spanish) https://www.youtube.com/watch?v=hk_urzWGqrE