Save 37% off PRO during our Black Friday Sale! »

Testing with mutants

Testing with mutants

As seen on Clojure eXchange 2016, Lambda Days 2017 and :clojureD 2017.

Property-based testing is mainstream now (there, I've said it). A recurring topic at EuroClojure, the generative approach to testing popularised by test.check enjoys wide adoption in the Clojure community.

This talk is dedicated to another randomised testing technique. Unlike property-based testing, which inspects our src directory, mutation testing audits contents of the test one. It introduces subtle bugs to source code of our projects and verifies that our test suites catch those synthetic problems. Clojure, a homoiconic language with dynamic code reloading, should offer us an excellent foundation for building a mutation testing library. But there’s not a single one out there. Why? Let’s try to answer this question together.



Jan Stępień

December 01, 2016


  1. Testing with mutants @janstepien

  2. © 2007 Mario Roberto Duran Ortiz

  3. None
  4. We write tests

  5. lein new project (ns project.core-test (:require [clojure.test :refer :all] [project.core

    :refer :all])) (deftest a-test (testing "FIXME, I fail." (is (= 0 1))))
  6. (require '[clojure.test.check [generators :as gen] [properties :as prop]]) (def prop-sort-idempotency

    (prop/for-all [coll (gen/vector gen/int)] (= ((comp sort sort) coll) (sort coll)))) Generative testing
  7. We write tests

  8. Who is testing our tests? @janstepien

  9. Mutation testing

  10. Competent programmer hypothesis

  11. Coupling effect hypothesis

  12. Mutants which don’t get killed become survivors

  13. None
  14. Off the shelf > Mutant for Ruby > PIT for

    Java > Stryker for JavaScript …and many more
  15. Mutant @janstepien Mutation testing for Clojure

  16. > Read all source files in a directory > Generate

    mutants for all top-level forms > Run the test suite for every mutant > Report all survivors
  17. Generating mutants

  18. [rewrite-clj "0.6.0"]

  19. (and x y) (or x y) (< x 1) (<=

    x 1) (empty? coll) (seq coll) (defn f [a b] (+ a b)) (defn f [a b] )
  20. Reevaluating the code

  21. [org.clojure/tools.namespace "0.2.11"]

  22. Running tests

  23. 8 survivors out of 61 mutants (ns mutant.mutations) (defn- rm-args

    [node] (let [sexpr (z/sexpr node)] (if (seq? sexpr) (let [[defn name args & more] sexpr] (if ([-and-]{+or+} (#{'defn 'defn-} defn) (vector? args)) (for [arg args] (-> node z/down z/right z/right (z/edit (partial filterv (complement #{arg}))) (z/up)))))))) (ns mutant.internals) (defn- dependants [graph ns] [-(letfn [(rec [sym] (if-let [deps (seq (dep/dependents graph sym))] (reduce into [] (conj (mapv rec deps) deps))))] (reverse (distinct (rec ns))))-])
  24. Problems

  25. It’s slow Do fewer, do faster,
 or do smarter. —

    Offutt and Untch, 2001
  26. Do fewer > Select what to mutate > Select your

    mutation operators > Sample your mutants
  27. Do faster > Don’t restart the virtual machine > Run

    tests in parallel
  28. Do smarter > Execute only relevant tests > Reorder the

    test suite
  29. It might not terminate > JVM cannot stop threads >

    JVM cannot fork > Starting new JVM is too slow
  30. Mutate continuously @janstepien

  31. git diff master~..master

  32. Introduce it gradually @janstepien

  33. [lein-mutate "0.1.0"]

  34. Testing with mutants @janstepien