Testing with mutants

Testing with mutants

As seen on Clojure eXchange 2016, Lambda Days 2017 and :clojureD 2017.

Property-based testing is mainstream now (there, I've said it). A recurring topic at EuroClojure, the generative approach to testing popularised by test.check enjoys wide adoption in the Clojure community.

This talk is dedicated to another randomised testing technique. Unlike property-based testing, which inspects our src directory, mutation testing audits contents of the test one. It introduces subtle bugs to source code of our projects and verifies that our test suites catch those synthetic problems. Clojure, a homoiconic language with dynamic code reloading, should offer us an excellent foundation for building a mutation testing library. But there’s not a single one out there. Why? Let’s try to answer this question together.

Video: https://youtu.be/ux9wP3mU7xM

Ae7a42fb716793697b1d222f3cc753b8?s=128

Jan Stępień

December 01, 2016
Tweet

Transcript

  1. 3.
  2. 5.

    lein new project (ns project.core-test (:require [clojure.test :refer :all] [project.core

    :refer :all])) (deftest a-test (testing "FIXME, I fail." (is (= 0 1))))
  3. 6.

    (require '[clojure.test.check [generators :as gen] [properties :as prop]]) (def prop-sort-idempotency

    (prop/for-all [coll (gen/vector gen/int)] (= ((comp sort sort) coll) (sort coll)))) Generative testing
  4. 13.
  5. 14.

    Off the shelf > Mutant for Ruby > PIT for

    Java > Stryker for JavaScript …and many more
  6. 16.

    > Read all source files in a directory > Generate

    mutants for all top-level forms > Run the test suite for every mutant > Report all survivors
  7. 19.

    (and x y) (or x y) (< x 1) (<=

    x 1) (empty? coll) (seq coll) (defn f [a b] (+ a b)) (defn f [a b] )
  8. 23.

    8 survivors out of 61 mutants (ns mutant.mutations) (defn- rm-args

    [node] (let [sexpr (z/sexpr node)] (if (seq? sexpr) (let [[defn name args & more] sexpr] (if ([-and-]{+or+} (#{'defn 'defn-} defn) (vector? args)) (for [arg args] (-> node z/down z/right z/right (z/edit (partial filterv (complement #{arg}))) (z/up)))))))) (ns mutant.internals) (defn- dependants [graph ns] [-(letfn [(rec [sym] (if-let [deps (seq (dep/dependents graph sym))] (reduce into [] (conj (mapv rec deps) deps))))] (reverse (distinct (rec ns))))-])
  9. 24.
  10. 26.

    Do fewer > Select what to mutate > Select your

    mutation operators > Sample your mutants
  11. 29.

    It might not terminate > JVM cannot stop threads >

    JVM cannot fork > Starting new JVM is too slow