Beyond Code Coverage, Mutation Testing: Tests for your tests

Slide 1

Slide 1 text

BEYOND CODE COVERAGE, MUTATION TESTING: TESTS FOR YOUR TESTS Diego Rodríguez Baquero @DiegoRBaquero

Slide 2

Slide 2 text

ABOUT ME DIEGO RODRÍGUEZ BAQUERO (@DIEGORBAQUERO) ▸ Bogotá, Colombia ▸ Senior Software Developer @ Inalambria ▸ Senior Undergrad Student @ Universidad de los Andes ▸ Next: Masters of Software Engineering ▸ FOSS Contributor

Slide 3

Slide 3 text

MUTATION TESTING

Slide 4

Slide 4 text

HOW AND WHY I GOT HERE AUTOMATED TESTING ▸ Unit tests ▸ Integration tests ▸ Systems tests ▸ End-to-end tests ▸ … Are these ok?

Slide 5

Slide 5 text

A FIRST STEP MEASURING QUALITY ▸ Code coverage ▸ Line coverage ▸ Statement coverage ▸ Branch coverage ▸ Function coverage

Slide 6

Slide 6 text

A QUICK INTRO MUTATION TESTING ▸ Best way (so far) to assess test suite effectiveness ▸ 100% statement/branch/function coverage is not enough ▸ Inserts faults that resemble real world bugs ▸ Mutants ▸ Runs your test suite ▸ Did your test suite catch the mutant?

Slide 7

Slide 7 text

EXAMPLE HOW DOES A MUTANT LOOK?

Slide 8

Slide 8 text

NOT EVERYTHING IS THAT BEAUTIFUL CAVEATS ▸ It means running your tests as many times as there is mutants ▸ e.g. request ▸ ~28 second test suite ▸ 4247 generated mutants ▸ Over 4 hours to complete on a powerful MacBook

Slide 9

Slide 9 text

ECOSYSTEM PIT - JAVA MutPy - Python Mutant - Ruby Infection - PHP MDroid+ - Android Mutate++ - C++ Milu - C Major - JAVA Cosmic Ray -Python FOR OTHER LANGUAGES

Slide 10

Slide 10 text

WHAT ABOUT FOR US?

Slide 11

Slide 11 text

MUTANDIS 2013 - 2015 ▸ Built with Java ▸ Static & dynamic analysis to guide mutation testing ▸ 30+ Operators ▸ Not maintained for almost 3 years Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. 2013. Efﬁcient JavaScript mutation testing. In ICST’13. 74–83. https://github.com/saltlab/mutandis/

Slide 12

Slide 12 text

MUTANT 2016 ▸ Built with JavaScript ▸ 7 Operators ▸ Only works with TAP output test frameworks ▸ Not maintained for over 2 years https://github.com/benhartley/mutant

Slide 13

Slide 13 text

STRYKER 2016 - 2018 ▸ Built with TypeScript ▸ 30+ operators ▸ Works for JavaScript and TypeScript ▸ Actively maintained ▸ Supports Jest, React, Karma, Mocha, Jasmine, Webpack, Angular ▸ Requires plugins and conﬁguration to work https://github.com/stryker-mutator/stryker

Slide 14

Slide 14 text

MUTODE

Slide 15

Slide 15 text

MUTODE 2017 - 2018 ▸ Built with JavaScript ▸ 40+ operators ▸ Works with any testing framework ▸ Optional conﬁguration https://github.com/TheSoftwareDesignLab/mutode Rodríguez-Baquero, Diego, and Mario Linares-Vásquez. 2018. ”Mutode: generic JavaScript and Node.js mutation testing tool." In ISSTA’18, pp. 372-375.

Slide 16

Slide 16 text

MUTODE’S FLOW HOW IT WORKS ▸ Create copies of your app/module ▸ Generate mutants > Queue ▸ Insert mutants into those copies ▸ Run `npm test` ▸ Check exit code and mark mutant as killed/survived ▸ No exit (Timeout) > Discarded mutant ▸ Report results

Slide 17

Slide 17 text

INSTALLATION MUTODE ▸ Globally ▸ Locally as a dev dependency

Slide 18

Slide 18 text

EXECUTION MUTODE

Slide 19

Slide 19 text

DEMO

Slide 20

Slide 20 text

FUTURE WORK

Slide 21

Slide 21 text

THINGS I WANT TO ADD FUTURE WORK ▸ Cover every statement with at least 1 mutant ▸ Gather data for future analysis ▸ Optimize generation of mutants ▸ Only mutate lines covered by test suite ▸ Prioritize certain operators and create less mutants

Slide 22

Slide 22 text

THINGS I WANT TO ADD FUTURE WORK ▸ Optimize execution of mutants ▸ Only run tests that cover mutated line ▸ Skip similar mutants if previous are all killed/survived ▸ Higher order mutation testing ▸ Mix 2 or more mutants (imagine the time required) ▸ Find best combinations

Slide 23

Slide 23 text

THANK YOU

Slide 24

Slide 24 text

BEYOND CODE COVERAGE, MUTATION TESTING: TESTS FOR YOUR TESTS DIEGO RODRÍGUEZ-BAQUERO @DiegoRBaquero https://www.npmjs.com/package/mutode   https://thesoftwaredesignlab.github.io/mutode/