Slide 1

Slide 1 text

Generative Testing Properties, State and Beyond Jan Stępień @janstepien [email protected]

Slide 2

Slide 2 text

© iwillbehomesoon 2012, flickr.com/photos/jp_photo_online/6964634964

Slide 3

Slide 3 text

lein new project (ns project.core-test (:require [clojure.test :refer :all] [project.core :refer :all])) (deftest a-test (testing "FIXME, I fail." (is (= 0 1))))

Slide 4

Slide 4 text

Why? user=> (= 0 1) false

Slide 5

Slide 5 text

Properties

Slide 6

Slide 6 text

/

Slide 7

Slide 7 text

(deftest test-division (is (= 5 (/ 15 3))) (is (= 0.5 (/ 1.0 2))) (is (thrown? ArithmeticException (/ 1 0))))

Slide 8

Slide 8 text

(deftest test-division (are [a b c] (= (/ a b) c) 15 3 5 1.0 2 0.5))

Slide 9

Slide 9 text

a b = a b a b · b = a b · b a b · b = a (= (* (/ a b) b) a)

Slide 10

Slide 10 text

(deftest test-division (are [a b] (= (* (/ a b) b) a) 15 3 1.0 2))

Slide 11

Slide 11 text

(deftest test-division (let [a (arbitrary-number) b (arbitrary-number)] (is (= (* (/ a b) b) a))))

Slide 12

Slide 12 text

QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs Koen Claessen Chalmers University of Technology [email protected] John Hughes Chalmers University of Technology [email protected] ABSTRACT QuickCheck is a tool which aids the Haskell programmer in formulating and testing properties of programs. Properties are described as Haskell functions, and can be automati- cally tested on random input, but it is also possible to de- ne custom test data generators. We present a number of case studies, in which the tool was successfully used, and also point out some pitfalls to avoid. Random testing is es- pecially suitable for functional programs because properties can be stated at a ne grain. When a function is built from separately tested components, then random testing suces to obtain good coverage of the de nition under test. 1. INTRODUCTION Testing is by far the most commonly used approach to ensuring software quality. It is also very labour intensive, accounting for up to 50% of the cost of software develop- ment. Despite anecdotal evidence that functional programs require somewhat less testing (`Once it type-checks, it usu- ally works'), in practice it is still a major part of functional program development. The cost of testing motivates e orts to automate it, wholly monad are hard to test), and so testing can be done at a ne grain. A testing tool must be able to determine whether a test is passed or failed; the human tester must supply an auto- matically checkable criterion of doing so. We have chosen to use formal speci cations for this purpose. We have de- signed a simple domain-speci c language of testable speci - cations which the tester uses to de ne expected properties of the functions under test. QuickCheckthen checks that the properties hold in a large number of cases. The speci ca- tion language is embedded in Haskell using the class system. Properties are normally written in the same module as the functions they test, where they serve also as checkable doc- umentation of the behaviour of the code. A testing tool must also be able to generate test cases au- tomatically. We have chosen the simplest method, random testing [11], which competes surprisingly favourably with systematic methods in practice. However, it is meaningless to talk about random testing without discussing the distri- bution of test data. Random testing is most e ective when the distribution of test data follows that of actual data, but when testing reuseable code units as opposed to whole sys- tems this is not possible, since the distribution of actual data in all subsequent reuses is not known. A uniform dis-

Slide 13

Slide 13 text

[org.clojure/test.check "0.6.2"]

Slide 14

Slide 14 text

(require '[clojure.test.check :as tc]) (require '[clojure.test.check [generators :as gen] [properties :as prop]]) (def prop-division (prop/for-all [a gen/int b gen/int] (= (* (/ a b) b) a))) ;; notice that it's ∀a ∈ Z ∀b ∈ Z ( a b · b = a ) (tc/quick-check 1000 prop-division)

Slide 15

Slide 15 text

{:result #, :seed 1420978137717, :failing-size 0, :num-tests 1, :fail [0 0], :shrunk {:total-nodes-visited 0, :depth 0, :result #, :smallest [0 0]}}

Slide 16

Slide 16 text

(def gen-non-zero-int (gen/such-that #(not= 0 %) gen/int)) (gen/sample gen/int) #_=> (0 0 -1 -2 -2 0 1 -4 -1 0) (gen/sample gen-non-zero-int) #_=> (-2 1 1 2 4 -4 -5 1 3 -7)

Slide 17

Slide 17 text

(def prop-division (prop/for-all [a gen/int b gen-non-zero-int] (= (* (/ a b) b) a))) (tc/quick-check 1000 prop-division) #_=> {:result true, :num-tests 1000, :seed 1420978581409}

Slide 18

Slide 18 text

(def gen-double (gen/fmap double gen/int)) (gen/sample gen-double) #_=> (0.0 0.0 0.0 1.0 -1.0 -1.0 2.0 -4.0 7.0 3.0) (def gen-non-zero-double (gen/such-that #(not= 0.0 %) gen-double))

Slide 19

Slide 19 text

(def prop-division (prop/for-all [a gen-double b gen-non-zero-double] (= (* (/ a b) b) a))) (tc/quick-check 1000 prop-division) #_=> {:result false :seed 1421584282754 :failing-size 25 :num-tests 26 :fail [23.0 21.0] :shrunk {:total-nodes-visited 9 :depth 0 :result false :smallest [23.0 21.0]}}

Slide 20

Slide 20 text

(= (* (/ 23.0 21.0) 21.0) 23.0) #_=> false (* (/ 23.0 21.0) 21.0) #_=> 23.000000000000004

Slide 21

Slide 21 text

reverse (= (comp reverse reverse) identity) (= (reverse (reverse coll)) coll)

Slide 22

Slide 22 text

(ns project.core-test) (defn reverse [coll] ()) (require '[clojure.test.check.clojure-test :refer [defspec]]) (defspec prop-reverse-composed (prop/for-all [coll (gen/list gen/int)] (= coll (reverse (reverse coll)))))

Slide 23

Slide 23 text

(clojure.test/run-tests 'project.core-test) {:result false, :seed 1420987857457, :failing-size 1, :num-tests 2, :fail [(0)], :shrunk {:total-nodes-visited 1 :depth 0 :result false :smallest [(0)]}}

Slide 24

Slide 24 text

(clojure.test/run-tests 'project.core-test) {:result false, :seed 1420981522879, :failing-size 2, :num-tests 3, :fail [(2 1)], :shrunk {:total-nodes-visited 4 :depth 2 :result false :smallest [(0)]}}

Slide 25

Slide 25 text

www.stylefruits.de/hosen/lee/farbe-hellblau/40-70-euro/seite-2

Slide 26

Slide 26 text

(path->descriptor ctx path) (descriptor->path ctx desc)

Slide 27

Slide 27 text

(prop/for-all [descr gen-descriptor ctx gen-context] (= descr (->> descr (descriptor->path ctx) (path->descriptor ctx))))

Slide 28

Slide 28 text

State

Slide 29

Slide 29 text

Modus Operandi ▶ Generate actions changing the state ▶ Apply each action to the state, if possible ▶ Aer each application verify the model

Slide 30

Slide 30 text

Testing Telecoms Software with Quviq QuickCheck Thomas Arts IT University of G¨ oteborg, Gothenburg, Sweden and Quviq AB [email protected] John Hughes Chalmers University, Gothenburg, Sweden and Quviq AB [email protected] Joakim Johansson Ulf Wiger Ericsson AB, ¨ Alvsj¨ o, Sweden [email protected] [email protected] Abstract We present a case study in which a novel testing tool, Quviq QuickCheck, is used to test an industrial implementation of the Megaco protocol. We considered positive and negative testing and we used our developed specification to test an old version in order to estimate how useful QuickCheck could potentially be when used early in development. The results of the case study indicate that, by using Quviq QuickCheck, we would have been able to detect faults early in the development. We detected faults that had not been detected by other testing techniques. We found unclarities in the specifications and potential faults when the software is used in a different setting. The results are considered promising enough to Ericsson that they are investing in an even larger case study, this time from the beginning of the development of a new product. Categories and Subject Descriptors D.2.5 [Software Engineer- ing]: Testing and Debugging—Testing tools; D.2.4 [Software En- gineering]: Software/Program Verification—Formal methods General Terms Verification Keywords Test Automation, Property Based Testing 1. Introduction it potentially reduce testing time? Would it find obscure bugs, and help to improve final product quality? Our study is small and qual- itative, but it suggests that the answer to all of these questions is a resounding “Yes”. The rest of the paper is structured as follows. In section 2 we give an introduction to Quviq QuickCheck, and in section 3 we explain our case study, giving some background on the software testing methods already used at Ericsson. In section 4 we explain our approach in detail, with samples of the testing code and a description of the extensions we made in parallel to QuickCheck, to better support this kind of testing. In 5 we present the results we obtained, and in 8 we draw conclusions. 2. Quviq QuickCheck Quviq QuickCheck is a property-based testing tool, developed from Claessen and Hughes’ earlier QuickCheck tool for Haskell [3] and a re-design for Erlang [2]. Apart from adaption to an Erlang setting, Quviq QuickCheck includes a number of extensions, of which the most significant is an ability to simplify failing test cases automatically. Quviq QuickCheck is a product of Quviq AB. A user of QuickCheck writes properties that are expected to hold, as Erlang source code making use of the QuickCheck API. For example, one property of the standard list reversal function is

Slide 31

Slide 31 text

Operations on Vectors ▶ conj ▶ pop

Slide 32

Slide 32 text

(ns operations-on-vectors (:require [clojure.test.check [clojure-test :refer [defspec]] [generators :as gen]] [states.core :as states])) (defspec prop-vec-ops (states/run-commands commands precondition next-step postcondition {:init-state {:vec []}}))

Slide 33

Slide 33 text

(defn commands [{:keys [vec]}] (gen/one-of [(gen/tuple (gen/return 'conj) (gen/return vec) gen/int) (gen/return ['pop vec])])) (gen/sample (commands {:vec 'a})) #_=> ([conj a 0] [pop a] [pop a] [conj a -3] ...)

Slide 34

Slide 34 text

(defn precondition [{:keys [elems]} [fn & _]] (case fn pop (seq elems) true))

Slide 35

Slide 35 text

(defn next-step [state var [fn _ elem]] (case fn conj (-> state (update-in [:elems] conj elem) (assoc :vec var)) pop (-> state (update-in [:elems] rest) (assoc :vec var))))

Slide 36

Slide 36 text

(postcondition [{:keys [elems]} [fn & _] val] (case fn pop (= val elems) true))

Slide 37

Slide 37 text

(clojure.test/run-tests 'operations-on-vectors) #_=> {:result #, :seed 1421594608384, :failing-size 9, :num-tests 10, :fail [((set var-0 (conj [] -1)) (set var-1 (conj var-0 7)) (set var-2 (conj var-1 -2)) (set var-3 (pop var-2)))], :shrunk {...}}

Slide 38

Slide 38 text

(postcondition [{:keys [elems]} [fn & _] val] (case fn pop (= val (reverse elems)) true))

Slide 39

Slide 39 text

(clojure.test/run-tests 'operations-on-vectors) Testing operations-on-vectors {:result true, :num-tests 100, :seed 14215224} Ran 1 tests containing 1 assertions. 0 failures, 0 errors.

Slide 40

Slide 40 text

github.com/michalmarczyk/ctries.clj (esp. the ctries.clj-test namespace)

Slide 41

Slide 41 text

Beyond

Slide 42

Slide 42 text

“Testing the Hard Stuff and Staying Sane” John Hughes

Slide 43

Slide 43 text

“Generative testing with clojure.test.check” Philip Poer

Slide 44

Slide 44 text

github.com/jstepien/states github.com/czan/stateful-check github.com/krestenkrab/triq github.com/manopapad/proper basho.com/quickchecking-poolboy-for-fun-and-profit

Slide 45

Slide 45 text

Generative Testing Properties, State and Beyond Jan Stępień @janstepien [email protected]