Beyond Code Coverage, Mutation Testing: Tests for your tests

Mutation testing introduction for JavaScript and Node.js. Given at Node Summit 2018 in San Francisco

Transcript

1. BEYOND CODE COVERAGE, MUTATION TESTING: TESTS FOR YOUR TESTS Diego Rodríguez Baquero @DiegoRBaquero
2. ABOUT ME DIEGO RODRÍGUEZ BAQUERO (@DIEGORBAQUERO) ▸ Bogotá, Colombia ▸ Senior Software Developer @ Inalambria ▸ Senior Undergrad Student @ Universidad de los Andes ▸ Next: Masters of Software Engineering ▸ FOSS Contributor
3. MUTATION TESTING
4. HOW AND WHY I GOT HERE AUTOMATED TESTING ▸ Unit tests ▸ Integration tests ▸ Systems tests ▸ End-to-end tests ▸ … Are these ok?
5. A FIRST STEP MEASURING QUALITY ▸ Code coverage ▸ Line coverage ▸ Statement coverage ▸ Branch coverage ▸ Function coverage
6. A QUICK INTRO MUTATION TESTING ▸ Best way (so far) to assess test suite effectiveness ▸ 100% statement/branch/function coverage is not enough ▸ Inserts faults that resemble real world bugs ▸ Mutants ▸ Runs your test suite ▸ Did your test suite catch the mutant?
7. EXAMPLE HOW DOES A MUTANT LOOK?
8. NOT EVERYTHING IS THAT BEAUTIFUL CAVEATS ▸ It means running your tests as many times as there is mutants ▸ e.g. request ▸ ~28 second test suite ▸ 4247 generated mutants ▸ Over 4 hours to complete on a powerful MacBook
9. ECOSYSTEM PIT - JAVA MutPy - Python Mutant - Ruby Infection - PHP MDroid+ - Android Mutate++ - C++ Milu - C Major - JAVA Cosmic Ray -Python FOR OTHER LANGUAGES
10. WHAT ABOUT FOR US?
11. MUTANDIS 2013 - 2015 ▸ Built with Java ▸ Static & dynamic analysis to guide mutation testing ▸ 30+ Operators ▸ Not maintained for almost 3 years Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. 2013. Efficient JavaScript mutation testing. In ICST’13. 74–83. https://github.com/saltlab/mutandis/
12. MUTANT 2016 ▸ Built with JavaScript ▸ 7 Operators ▸ Only works with TAP output test frameworks ▸ Not maintained for over 2 years https://github.com/benhartley/mutant
13. STRYKER 2016 - 2018 ▸ Built with TypeScript ▸ 30+ operators ▸ Works for JavaScript and TypeScript ▸ Actively maintained ▸ Supports Jest, React, Karma, Mocha, Jasmine, Webpack, Angular ▸ Requires plugins and configuration to work https://github.com/stryker-mutator/stryker
14. MUTODE
15. MUTODE 2017 - 2018 ▸ Built with JavaScript ▸ 40+ operators ▸ Works with any testing framework ▸ Optional configuration https://github.com/TheSoftwareDesignLab/mutode Rodríguez-Baquero, Diego, and Mario Linares-Vásquez. 2018. ”Mutode: generic JavaScript and Node.js mutation testing tool." In ISSTA’18, pp. 372-375.
16. MUTODE’S FLOW HOW IT WORKS ▸ Create copies of your app/module ▸ Generate mutants > Queue ▸ Insert mutants into those copies ▸ Run `npm test` ▸ Check exit code and mark mutant as killed/survived ▸ No exit (Timeout) > Discarded mutant ▸ Report results
17. INSTALLATION MUTODE ▸ Globally ▸ Locally as a dev dependency
18. EXECUTION MUTODE
19. DEMO
20. FUTURE WORK
21. THINGS I WANT TO ADD FUTURE WORK ▸ Cover every statement with at least 1 mutant ▸ Gather data for future analysis ▸ Optimize generation of mutants ▸ Only mutate lines covered by test suite ▸ Prioritize certain operators and create less mutants
22. THINGS I WANT TO ADD FUTURE WORK ▸ Optimize execution of mutants ▸ Only run tests that cover mutated line ▸ Skip similar mutants if previous are all killed/survived ▸ Higher order mutation testing ▸ Mix 2 or more mutants (imagine the time required) ▸ Find best combinations
23. THANK YOU
24. BEYOND CODE COVERAGE, MUTATION TESTING: TESTS FOR YOUR TESTS DIEGO RODRÍGUEZ-BAQUERO @DiegoRBaquero https://www.npmjs.com/package/mutode 
 https://thesoftwaredesignlab.github.io/mutode/