Testing with mutants - Speaker Deck

Slide 1

Slide 1 text

Testing with mutants @janstepien

Slide 2

Slide 2 text

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

We write tests

Slide 5

Slide 5 text

lein new project (ns project.core-test (:require [clojure.test :refer :all] [project.core :refer :all])) (deftest a-test (testing "FIXME, I fail." (is (= 0 1))))

Slide 6

Slide 6 text

(require '[clojure.test.check [generators :as gen] [properties :as prop]]) (def prop-sort-idempotency (prop/for-all [coll (gen/vector gen/int)] (= ((comp sort sort) coll) (sort coll)))) Generative testing

Slide 7

Slide 7 text

We write tests

Slide 8

Slide 8 text

Who is testing our tests? @janstepien

Slide 9

Slide 9 text

Mutation testing

Slide 10

Slide 10 text

Competent programmer hypothesis

Slide 11

Slide 11 text

Coupling effect hypothesis

Slide 12

Slide 12 text

Mutants which don’t get killed become survivors

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Off the shelf > Mutant for Ruby > PIT for Java > Stryker for JavaScript …and many more

Slide 15

Slide 15 text

Mutant @janstepien Mutation testing for Clojure

Slide 16

Slide 16 text

> Read all source files in a directory > Generate mutants for all top-level forms > Run the test suite for every mutant > Report all survivors

Slide 17

Slide 17 text

Generating mutants

Slide 18

Slide 18 text

[rewrite-clj "0.6.0"]

Slide 19

Slide 19 text

(and x y) (or x y) (< x 1) (<= x 1) (empty? coll) (seq coll) (defn f [a b] (+ a b)) (defn f [a b] )

Slide 20

Slide 20 text

Reevaluating the code

Slide 21

Slide 21 text

[org.clojure/tools.namespace "0.2.11"]

Slide 22

Slide 22 text

Running tests

Slide 23

Slide 23 text

8 survivors out of 61 mutants (ns mutant.mutations) (defn- rm-args [node] (let [sexpr (z/sexpr node)] (if (seq? sexpr) (let [[defn name args & more] sexpr] (if ([-and-]{+or+} (#{'defn 'defn-} defn) (vector? args)) (for [arg args] (-> node z/down z/right z/right (z/edit (partial filterv (complement #{arg}))) (z/up)))))))) (ns mutant.internals) (defn- dependants [graph ns] [-(letfn [(rec [sym] (if-let [deps (seq (dep/dependents graph sym))] (reduce into [] (conj (mapv rec deps) deps))))] (reverse (distinct (rec ns))))-])

Slide 24

Slide 24 text

Problems

Slide 25

Slide 25 text

It’s slow Do fewer, do faster,  or do smarter. — Offutt and Untch, 2001

Slide 26

Slide 26 text

Do fewer > Select what to mutate > Select your mutation operators > Sample your mutants

Slide 27

Slide 27 text

Do faster > Don’t restart the virtual machine > Run tests in parallel

Slide 28

Slide 28 text

Do smarter > Execute only relevant tests > Reorder the test suite

Slide 29

Slide 29 text

It might not terminate > JVM cannot stop threads > JVM cannot fork > Starting new JVM is too slow