The Magic of Generative Testing: Fast-Check in JavaScript

(Video at https://youtu.be/a2J_FSkxWKo)

Generative testing (aka property testing), popularized by the Haskell library QuickCheck, is a technique of:

- specifying invariant laws your code expects to exhibit
- generating random inputs to verify the laws
- simplifying failures to find error boundaries
- providing replay mechanisms for easy debugging

In this BrooklynJS talk, I show the motivations for and basics of property testing in JavaScript, via the library fast-check.

Transcript

1. The Magic of
   Generative
   Testing
   BrooklynJS · 2019-06-20
   

   View Slide

2. glebec
   glebec
   glebec
   glebec
   g_lebec
   
   Gabriel Lebec
   

   View Slide

3. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   })
   

   View Slide

4. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   }) ✅ Passes!
   

   View Slide

5. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   })
   
   

   View Slide

6. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   })
   
   • nums => [1, 2, 3]
   • nums => [nums[1], nums[0], nums[2]]
   • nums => realSort([ ...new Set(nums)])
   • nums => realSort([nums[0], ...nums.slice(1)])
   • nums => realSort(nums.map(n => Math.round(n))
   • nums => nums.sort()
   All these pass…
   

   View Slide

7. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   })
   So what do we do?
   

   View Slide

8. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
   expect(sort([2, 3, 1])).toEqual([1, 2, 3])
   })
   })
   

   View Slide

9. describe('`sort`', () => {
   it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
   expect(sort([2, 1, 3])).toEqual([1, 2, 3])
   })
   it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
   expect(sort([2, 3, 1])).toEqual([1, 2, 3])
   })
   it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => {
   expect(sort([1.5, -4])).toEqual([-4, 1.5])
   })
   })
   

   View Slide

10. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
    expect(sort([2, 3, 1])).toEqual([1, 2, 3])
    })
    it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => {
    expect(sort([1.5, -4])).toEqual([-4, 1.5])
    })
    it('transforms `[]` to `[]`', () => {
    expect(sort([])).toEqual([])
    })
    })
    

    View Slide

11. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
    expect(sort([2, 3, 1])).toEqual([1, 2, 3])
    })
    it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => {
    expect(sort([1.5, -4])).toEqual([-4, 1.5])
    })
    it('transforms `[]` to `[]`', () => {
    expect(sort([])).toEqual([])
    })
    it('transforms `[1, 1, 1, 1, 1]` to `[1, 1, 1, 1, 1]`', () => {
    expect(sort([1, 1, 1, 1, 1])).toEqual([1, 1, 1, 1, 1])
    })
    it('transforms `[999, 500, 2]` to `[999. 500, 2]`', () => {
    expect(sort([999, 500, 2])).toEqual([2, 500, 999])
    })
    it('transforms `[0x821, 1.3e5]` to `[0x821, 1.3e5]`', () => {
    expect(sort([0x821, 1.3e5])).toEqual([0x821, 1.3e5])
    })
    it('transforms `[0]` to `[0]`', () => {
    expect(sort([0])).toEqual([0])
    })
    it('transforms `[-0]` to `[-0]`', () => {
    expect(sort([-0])).toEqual([-0])
    })
    it('transforms `[1, 2, 1, 2]` to `[1, 1, 2, 2]`', () => {
    expect(sort([1, 2, 1, 2])).toEqual([1, 1, 2, 2])
    })
    it('transforms `[10, 11, 12, 9001, -1]` to `[-1, 10, 11, 12, 9001]`', () => {
    expect(sort([10, 11, 12, 9001, -1])).toEqual([-1, 10, 11, 12, 9001])
    })
    it('transforms `[0.9999999, 1, 0.9999999]` to `[0.9999999, 0.9999999, 1]`', () => {
    expect(sort([0.9999999, 1, 0.9999999])).toEqual([0.9999999, 0.9999999, 1])
    

    View Slide

12. It's a bit of a pain…
    
    

    View Slide

13. 
    But are we confident?
    

    View Slide

14. ERROR: cannot read

    'name' of undefined
    100%
    coverage
    Build
    succeeded
    Tests passed
    As humans, we focus

    on the "happy path"
    

    View Slide

15. Surely there is

    a better way…
    
    

    View Slide

16. What are we saying in this test? What is the *point*?
    describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    })
    

    View Slide

17. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    }) Is it this particular array we care about?
    

    View Slide

18. describe('`sort`', () => {
    it('results in an ordered array', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    }) …or is it this particular *law*?
    aka "property" (of sorting)?
    aka "invariant" (behavior, regardless of input)
    

    View Slide

19. • Sorting a list of nums results in an ordered list
    • Sorting a sorted list is a no-op (idempotence)
    • Sorting neither adds, removes, nor changes elements
    ⚖
    

    View Slide

20. ⚖
    // :: [Number] -> Bool
    function sortOrders (arr) {
    const sorted = sort(arr)
    for (let i = 0; i < arr.length - 1; i ++) {
    if (arr[i] > arr[i + 1]) return false
    }
    return true
    }
    a predicate function (returns Bool) aka property check
    we've explicitly named our law
    

    View Slide

21. What if we generate

    the tests?
    
    

    View Slide

22. // :: (Number) -> Number
    function randomInt (limit = 1) {
    return Math.floor(Math.random() * limit)
    }
    // :: () -> [Number]
    function randomArr () {
    return Array.from(
    { length: randomInt(LIMIT_LEN) },
    () => randomInt(LIMIT_NUM)
    )
    }
    ad-hoc data generators
    

    View Slide

23. describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    for (let t = 0; t < LIMIT_TESTS; t ++) {
    expect(isOrdered(sort(randomArr()))).toBe(true)
    }
    })
    }) generate and verify many random cases
    ✅ Passes!
    

    View Slide

24. describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    for (let t = 0; t < LIMIT_TESTS; t ++) {
    expect(isOrdered(sort(randomArr()))).toBe(true)
    }
    })
    }) …but what happens if it fails?
    
    

    View Slide

25. FAIL src/01-sort.spec.ts
    ● `sort` › outputs ordered arrays
    expect(received).toBe(expected) // Object.is equality
    Expected: true
    Received: false
    

    View Slide

26. FAIL src/01-sort.spec.ts
    ● `sort` › outputs ordered arrays
    expect(received).toBe(expected) // Object.is equality
    Expected: true
    Received: false
    which input array failed?
    what was the output from sort?
    …let's just add a console.log…
    

    View Slide

27. …let's just add a console.log…
    ✅ Passes!
    
    
    Oops! Nondeterminism strikes again!
    

    View Slide

28. …re-ran, failed again… but why does THIS array fail?
    
    FAIL `sort` › outputs ordered arrays
    Input: [93, 2, 0, 1, 90101, 29, 350, 301, 620]
    Output: [0, 1, 2, 29, 301, 250, 620, 90101, 93]
    Expected: true
    Received: false
    
    

    View Slide

29. Trickier than we thought
    
    

    View Slide

30. laborious ad-hoc error-prone generators
    (did you notice we never generated negative ints?)
    random array generator maybe repeated empty arr
    non-instructive failure (input? output?)
    irreproducible
    failing cases may be unnecessarily large
    

    View Slide

31. Surely there is

    a better way (again)…
    
    

    View Slide

32. Use a

    Property Testing Library
    
    

    View Slide

33. QuickCheck
    Claessen & Hughes
    

    View Slide

34. JSVerify
    Oleg Grenrus
    Fast-Check
    Nicolas Dubien
    

    View Slide

35. ⚖ specify law (predicate function)
    ⏳ support for async predicates
    generates progressively bigger random inputs
    . common edge cases baked-in
    failing cases progressively "shrunk" (simplified)
    ⚠ finds error boundary
    deterministically pseudorandom
    can specify seed
    

    View Slide

36. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    

    View Slide

37. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    a fast-check "property" instance, which…
    

    View Slide

38. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    generates arrays of ints, and…
    

    View Slide

39. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    feeds them to this predicate
    

    View Slide

40. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    run many times, and throw upon failure…
    

    View Slide

41. PASS src/01-sort.spec.ts
    `sort`
    ✓ outputs ordered arrays (11ms)
    maybe it passes…
    

    View Slide

42. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    but if it fails, it fails elegantly
    

    View Slide

43. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    ran until failure (or configured limit)
    

    View Slide

44. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    failure was simplified 7 times
    

    View Slide

45. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    minimal edge case found
    

    View Slide

46. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    can reproduce at will
    

    View Slide

47. test assertion takes an optional config
    import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    ),
    )
    })
    })
    

    View Slide

48. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    ),
    { seed: 1213330267, verbose: true }
    )
    })
    })
    can specify PRNG seed
    

    View Slide

49. ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    Encountered failures were:
    - [[-10,-1823440029,-3,-8,109548166,22,236538465]]
    - [[-1823440029,-3,-8,22]]
    - [[-1823440029,-2,-8,22]]
    - [[-1,-1823440029,-8,22]]
    - [[-1,-8,22]]
    - [[-1,-8]]
    - [[-1,-4]]
    - [[-1,-2]]
    debug same code path, and/or
    see how minimal case was found
    

    View Slide

50. "Very cute, but I need
    more complex inputs."
    
    

    View Slide

51. describe("(one of) De Morgans Laws", () => {
    test('not (P and Q) <=> (not P) or (not Q)', () => {
    fc.assert(fc.property(
    fc.boolean(), fc.boolean(),
    (p, q) => { return !(p && q) === !p || !q }
    ))
    })
    })
    can easily request multiple arbitraries
    (generator + shrinker = "arbitrary")
    

    View Slide

52. describe('`Array#filter`', () => {
    it('always results in a same-or-shorter-length arr', () => {
    fc.assert(fc.property(
    fc.array(fc.anything()),
    fc.func(fc.boolean()),
    (arr, pred) => {
    return arr.filter(pred).length <= arr.length
    }
    ))
    })
    })
    built-in arbitraries compose beautifully
    ("combinator pattern")
    array of any vals
    function returning bools
    

    View Slide

53. describe('`isOdd`', () => {
    it('correctly identifies odd integers', () => {
    fc.assert(fc.property(
    fc.integer().map(n => n * 2 + 1),
    isOdd
    ))
    })
    })
    function isOdd (num) { return num % 2 === 1 }
    a new arbitrary
    which generates
    odd integers
    can transform arbitraries via functional
    techniques like `map` and `filter`
    (fails on -1!)
    

    View Slide

54. tons more!
    size minima / maxima
    ✍ special strings (ascii, unicode, ipv4, json)
    custom generators from scratch
    

    View Slide

55. Monads
    and even…
    
    
    
    ‼
    ‼
    
    
    ‼
    
    
    

    View Slide

56. Summary
    
    

    View Slide

57. Complementary to other strats
    Unit tests for specific important cases
    Fuzzers for robustness against garbage input
    Prop tests for confidence that API contract holds
    <- property-based testing
    image credit: Nicolas Dubien
    

    View Slide

58. Plain JS -> zero-effort integration
    No special plugins, matchers, or bindings needed
    Drop into Mocha, AVA, Jest, Tape, etc.
    Insert assertions from Chai, Sinon, Supertest, etc.
    

    View Slide

59. THANKS!
    
    - https: //speakerdeck.com/glebec/the-magic-of-generative-testing
    - https: //dev.to/glebec/property-testing-with-jsverify-part-i-3one
    - https: // www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf
    - YouTube: "Don't Write Tests"! (John Hughes)
    - https: //github.com/dubzzz/fast-check
    

    View Slide