Slide 1

Slide 1 text

The Magic of Generative Testing BrooklynJS · 2019-06-20

Slide 2

Slide 2 text

glebec glebec glebec glebec g_lebec Gabriel Lebec

Slide 3

Slide 3 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) })

Slide 4

Slide 4 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) }) ✅ Passes!

Slide 5

Slide 5 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) })

Slide 6

Slide 6 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) }) • nums => [1, 2, 3] • nums => [nums[1], nums[0], nums[2]] • nums => realSort([ ...new Set(nums)]) • nums => realSort([nums[0], ...nums.slice(1)]) • nums => realSort(nums.map(n => Math.round(n)) • nums => nums.sort() All these pass…

Slide 7

Slide 7 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) }) So what do we do?

Slide 8

Slide 8 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => { expect(sort([2, 3, 1])).toEqual([1, 2, 3]) }) })

Slide 9

Slide 9 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => { expect(sort([2, 3, 1])).toEqual([1, 2, 3]) }) it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => { expect(sort([1.5, -4])).toEqual([-4, 1.5]) }) })

Slide 10

Slide 10 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => { expect(sort([2, 3, 1])).toEqual([1, 2, 3]) }) it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => { expect(sort([1.5, -4])).toEqual([-4, 1.5]) }) it('transforms `[]` to `[]`', () => { expect(sort([])).toEqual([]) }) })

Slide 11

Slide 11 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => { expect(sort([2, 3, 1])).toEqual([1, 2, 3]) }) it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => { expect(sort([1.5, -4])).toEqual([-4, 1.5]) }) it('transforms `[]` to `[]`', () => { expect(sort([])).toEqual([]) }) it('transforms `[1, 1, 1, 1, 1]` to `[1, 1, 1, 1, 1]`', () => { expect(sort([1, 1, 1, 1, 1])).toEqual([1, 1, 1, 1, 1]) }) it('transforms `[999, 500, 2]` to `[999. 500, 2]`', () => { expect(sort([999, 500, 2])).toEqual([2, 500, 999]) }) it('transforms `[0x821, 1.3e5]` to `[0x821, 1.3e5]`', () => { expect(sort([0x821, 1.3e5])).toEqual([0x821, 1.3e5]) }) it('transforms `[0]` to `[0]`', () => { expect(sort([0])).toEqual([0]) }) it('transforms `[-0]` to `[-0]`', () => { expect(sort([-0])).toEqual([-0]) }) it('transforms `[1, 2, 1, 2]` to `[1, 1, 2, 2]`', () => { expect(sort([1, 2, 1, 2])).toEqual([1, 1, 2, 2]) }) it('transforms `[10, 11, 12, 9001, -1]` to `[-1, 10, 11, 12, 9001]`', () => { expect(sort([10, 11, 12, 9001, -1])).toEqual([-1, 10, 11, 12, 9001]) }) it('transforms `[0.9999999, 1, 0.9999999]` to `[0.9999999, 0.9999999, 1]`', () => { expect(sort([0.9999999, 1, 0.9999999])).toEqual([0.9999999, 0.9999999, 1])

Slide 12

Slide 12 text

It's a bit of a pain…

Slide 13

Slide 13 text

But are we confident?

Slide 14

Slide 14 text

ERROR: cannot read
 'name' of undefined 100% coverage Build succeeded Tests passed As humans, we focus
 on the "happy path"

Slide 15

Slide 15 text

Surely there is
 a better way…

Slide 16

Slide 16 text

What are we saying in this test? What is the *point*? describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) })

Slide 17

Slide 17 text

describe('`sort`', () => { it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) }) Is it this particular array we care about?

Slide 18

Slide 18 text

describe('`sort`', () => { it('results in an ordered array', () => { expect(sort([2, 1, 3])).toEqual([1, 2, 3]) }) }) …or is it this particular *law*? aka "property" (of sorting)? aka "invariant" (behavior, regardless of input)

Slide 19

Slide 19 text

• Sorting a list of nums results in an ordered list • Sorting a sorted list is a no-op (idempotence) • Sorting neither adds, removes, nor changes elements ⚖

Slide 20

Slide 20 text

⚖ // :: [Number] -> Bool function sortOrders (arr) { const sorted = sort(arr) for (let i = 0; i < arr.length - 1; i ++) { if (arr[i] > arr[i + 1]) return false } return true } a predicate function (returns Bool) aka property check we've explicitly named our law

Slide 21

Slide 21 text

What if we generate
 the tests?

Slide 22

Slide 22 text

// :: (Number) -> Number function randomInt (limit = 1) { return Math.floor(Math.random() * limit) } // :: () -> [Number] function randomArr () { return Array.from( { length: randomInt(LIMIT_LEN) }, () => randomInt(LIMIT_NUM) ) } ad-hoc data generators

Slide 23

Slide 23 text

describe('`sort`', () => { it('outputs ordered arrays', () => { for (let t = 0; t < LIMIT_TESTS; t ++) { expect(isOrdered(sort(randomArr()))).toBe(true) } }) }) generate and verify many random cases ✅ Passes!

Slide 24

Slide 24 text

describe('`sort`', () => { it('outputs ordered arrays', () => { for (let t = 0; t < LIMIT_TESTS; t ++) { expect(isOrdered(sort(randomArr()))).toBe(true) } }) }) …but what happens if it fails?

Slide 25

Slide 25 text

FAIL src/01-sort.spec.ts ● `sort` › outputs ordered arrays expect(received).toBe(expected) // Object.is equality Expected: true Received: false

Slide 26

Slide 26 text

FAIL src/01-sort.spec.ts ● `sort` › outputs ordered arrays expect(received).toBe(expected) // Object.is equality Expected: true Received: false which input array failed? what was the output from sort? …let's just add a console.log…

Slide 27

Slide 27 text

…let's just add a console.log… ✅ Passes! Oops! Nondeterminism strikes again!

Slide 28

Slide 28 text

…re-ran, failed again… but why does THIS array fail? FAIL `sort` › outputs ordered arrays Input: [93, 2, 0, 1, 90101, 29, 350, 301, 620] Output: [0, 1, 2, 29, 301, 250, 620, 90101, 93] Expected: true Received: false

Slide 29

Slide 29 text

Trickier than we thought

Slide 30

Slide 30 text

laborious ad-hoc error-prone generators (did you notice we never generated negative ints?) random array generator maybe repeated empty arr non-instructive failure (input? output?) irreproducible failing cases may be unnecessarily large

Slide 31

Slide 31 text

Surely there is
 a better way (again)…

Slide 32

Slide 32 text

Use a
 Property Testing Library

Slide 33

Slide 33 text

QuickCheck Claessen & Hughes

Slide 34

Slide 34 text

JSVerify Oleg Grenrus Fast-Check Nicolas Dubien

Slide 35

Slide 35 text

⚖ specify law (predicate function) ⏳ support for async predicates generates progressively bigger random inputs . common edge cases baked-in failing cases progressively "shrunk" (simplified) ⚠ finds error boundary deterministically pseudorandom can specify seed

Slide 36

Slide 36 text

import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ) ) }) })

Slide 37

Slide 37 text

import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ) ) }) }) a fast-check "property" instance, which…

Slide 38

Slide 38 text

import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ) ) }) }) generates arrays of ints, and…

Slide 39

Slide 39 text

import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ) ) }) }) feeds them to this predicate

Slide 40

Slide 40 text

import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ) ) }) }) run many times, and throw upon failure…

Slide 41

Slide 41 text

PASS src/01-sort.spec.ts `sort` ✓ outputs ordered arrays (11ms) maybe it passes…

Slide 42

Slide 42 text

FAIL src/01-sort.spec.ts `sort` ✕ outputs ordered arrays (9ms) ● `sort` › outputs ordered arrays Property failed after 4 tests { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true } Counterexample: [[-1,-2]] Shrunk 7 time(s) but if it fails, it fails elegantly

Slide 43

Slide 43 text

FAIL src/01-sort.spec.ts `sort` ✕ outputs ordered arrays (9ms) ● `sort` › outputs ordered arrays Property failed after 4 tests { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true } Counterexample: [[-1,-2]] Shrunk 7 time(s) ran until failure (or configured limit)

Slide 44

Slide 44 text

FAIL src/01-sort.spec.ts `sort` ✕ outputs ordered arrays (9ms) ● `sort` › outputs ordered arrays Property failed after 4 tests { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true } Counterexample: [[-1,-2]] Shrunk 7 time(s) failure was simplified 7 times

Slide 45

Slide 45 text

FAIL src/01-sort.spec.ts `sort` ✕ outputs ordered arrays (9ms) ● `sort` › outputs ordered arrays Property failed after 4 tests { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true } Counterexample: [[-1,-2]] Shrunk 7 time(s) minimal edge case found

Slide 46

Slide 46 text

FAIL src/01-sort.spec.ts `sort` ✕ outputs ordered arrays (9ms) ● `sort` › outputs ordered arrays Property failed after 4 tests { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true } Counterexample: [[-1,-2]] Shrunk 7 time(s) can reproduce at will

Slide 47

Slide 47 text

test assertion takes an optional config import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ), ) }) })

Slide 48

Slide 48 text

import * as fc from 'fast-check' describe('`sort`', () => { it('outputs ordered arrays', () => { fc.assert( fc.property( fc.array(fc.integer()), (nums) => isOrdered(sort(nums)) ), { seed: 1213330267, verbose: true } ) }) }) can specify PRNG seed

Slide 49

Slide 49 text

● `sort` › outputs ordered arrays Property failed after 4 tests { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true } Counterexample: [[-1,-2]] Shrunk 7 time(s) Encountered failures were: - [[-10,-1823440029,-3,-8,109548166,22,236538465]] - [[-1823440029,-3,-8,22]] - [[-1823440029,-2,-8,22]] - [[-1,-1823440029,-8,22]] - [[-1,-8,22]] - [[-1,-8]] - [[-1,-4]] - [[-1,-2]] debug same code path, and/or see how minimal case was found

Slide 50

Slide 50 text

"Very cute, but I need more complex inputs."

Slide 51

Slide 51 text

describe("(one of) De Morgans Laws", () => { test('not (P and Q) <=> (not P) or (not Q)', () => { fc.assert(fc.property( fc.boolean(), fc.boolean(), (p, q) => { return !(p && q) === !p || !q } )) }) }) can easily request multiple arbitraries (generator + shrinker = "arbitrary")

Slide 52

Slide 52 text

describe('`Array#filter`', () => { it('always results in a same-or-shorter-length arr', () => { fc.assert(fc.property( fc.array(fc.anything()), fc.func(fc.boolean()), (arr, pred) => { return arr.filter(pred).length <= arr.length } )) }) }) built-in arbitraries compose beautifully ("combinator pattern") array of any vals function returning bools

Slide 53

Slide 53 text

describe('`isOdd`', () => { it('correctly identifies odd integers', () => { fc.assert(fc.property( fc.integer().map(n => n * 2 + 1), isOdd )) }) }) function isOdd (num) { return num % 2 === 1 } a new arbitrary which generates odd integers can transform arbitraries via functional techniques like `map` and `filter` (fails on -1!)

Slide 54

Slide 54 text

tons more! size minima / maxima ✍ special strings (ascii, unicode, ipv4, json) custom generators from scratch

Slide 55

Slide 55 text

Monads and even… ‼ ‼ ‼

Slide 56

Slide 56 text

Summary

Slide 57

Slide 57 text

Complementary to other strats Unit tests for specific important cases Fuzzers for robustness against garbage input Prop tests for confidence that API contract holds <- property-based testing image credit: Nicolas Dubien

Slide 58

Slide 58 text

Plain JS -> zero-effort integration No special plugins, matchers, or bindings needed Drop into Mocha, AVA, Jest, Tape, etc. Insert assertions from Chai, Sinon, Supertest, etc.

Slide 59

Slide 59 text

THANKS! - https: //speakerdeck.com/glebec/the-magic-of-generative-testing - https: //dev.to/glebec/property-testing-with-jsverify-part-i-3one - https: // www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf - YouTube: "Don't Write Tests"! (John Hughes) - https: //github.com/dubzzz/fast-check