Testing with mutants

As seen on Clojure eXchange 2016, Lambda Days 2017 and :clojureD 2017.

Property-based testing is mainstream now (there, I've said it). A recurring topic at EuroClojure, the generative approach to testing popularised by test.check enjoys wide adoption in the Clojure community.

This talk is dedicated to another randomised testing technique. Unlike property-based testing, which inspects our src directory, mutation testing audits contents of the test one. It introduces subtle bugs to source code of our projects and verifies that our test suites catch those synthetic problems. Clojure, a homoiconic language with dynamic code reloading, should offer us an excellent foundation for building a mutation testing library. But there’s not a single one out there. Why? Let’s try to answer this question together.

Video: https://youtu.be/ux9wP3mU7xM

Transcript

1. Testing with mutants @janstepien
2. © 2007 Mario Roberto Duran Ortiz
4. We write tests
5. lein new project (ns project.core-test (:require [clojure.test :refer :all] [project.core :refer :all])) (deftest a-test (testing "FIXME, I fail." (is (= 0 1))))
6. (require '[clojure.test.check [generators :as gen] [properties :as prop]]) (def prop-sort-idempotency (prop/for-all [coll (gen/vector gen/int)] (= ((comp sort sort) coll) (sort coll)))) Generative testing
7. We write tests
8. Who is testing our tests? @janstepien
9. Mutation testing
10. Competent programmer hypothesis
11. Coupling effect hypothesis
12. Mutants which don’t get killed become survivors
14. Off the shelf > Mutant for Ruby > PIT for Java > Stryker for JavaScript …and many more
15. Mutant @janstepien Mutation testing for Clojure
16. > Read all source files in a directory > Generate mutants for all top-level forms > Run the test suite for every mutant > Report all survivors
17. Generating mutants
18. [rewrite-clj "0.6.0"]
19. (and x y) (or x y) (< x 1) (<= x 1) (empty? coll) (seq coll) (defn f [a b] (+ a b)) (defn f [a b] )
20. Reevaluating the code
21. [org.clojure/tools.namespace "0.2.11"]
22. Running tests
23. 8 survivors out of 61 mutants (ns mutant.mutations) (defn- rm-args [node] (let [sexpr (z/sexpr node)] (if (seq? sexpr) (let [[defn name args & more] sexpr] (if ([-and-]{+or+} (#{'defn 'defn-} defn) (vector? args)) (for [arg args] (-> node z/down z/right z/right (z/edit (partial filterv (complement #{arg}))) (z/up)))))))) (ns mutant.internals) (defn- dependants [graph ns] [-(letfn [(rec [sym] (if-let [deps (seq (dep/dependents graph sym))] (reduce into [] (conj (mapv rec deps) deps))))] (reverse (distinct (rec ns))))-])
24. Problems
25. It’s slow Do fewer, do faster,
 or do smarter. — Offutt and Untch, 2001
26. Do fewer > Select what to mutate > Select your mutation operators > Sample your mutants
27. Do faster > Don’t restart the virtual machine > Run tests in parallel
28. Do smarter > Execute only relevant tests > Reorder the test suite
29. It might not terminate > JVM cannot stop threads > JVM cannot fork > Starting new JVM is too slow
30. Mutate continuously @janstepien
31. git diff master~..master
32. Introduce it gradually @janstepien
33. [lein-mutate "0.1.0"] github.com/jstepien/mutant
34. Testing with mutants jan.stepien@innoq.com @janstepien