$30 off During Our Annual Pro Sale. View Details »

The Magic of Generative Testing: Fast-Check in JavaScript

The Magic of Generative Testing: Fast-Check in JavaScript

(Video at https://youtu.be/a2J_FSkxWKo)

Generative testing (aka property testing), popularized by the Haskell library QuickCheck, is a technique of:

- specifying invariant laws your code expects to exhibit
- generating random inputs to verify the laws
- simplifying failures to find error boundaries
- providing replay mechanisms for easy debugging

In this BrooklynJS talk, I show the motivations for and basics of property testing in JavaScript, via the library fast-check.

Gabriel Lebec

June 20, 2019
Tweet

More Decks by Gabriel Lebec

Other Decks in Programming

Transcript

  1. The Magic of
    Generative
    Testing
    BrooklynJS · 2019-06-20

    View Slide

  2. glebec
    glebec
    glebec
    glebec
    g_lebec

    Gabriel Lebec

    View Slide

  3. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    })

    View Slide

  4. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    }) ✅ Passes!

    View Slide

  5. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    })

    View Slide

  6. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    })

    • nums => [1, 2, 3]
    • nums => [nums[1], nums[0], nums[2]]
    • nums => realSort([ ...new Set(nums)])
    • nums => realSort([nums[0], ...nums.slice(1)])
    • nums => realSort(nums.map(n => Math.round(n))
    • nums => nums.sort()
    All these pass…

    View Slide

  7. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    })
    So what do we do?

    View Slide

  8. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
    expect(sort([2, 3, 1])).toEqual([1, 2, 3])
    })
    })

    View Slide

  9. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
    expect(sort([2, 3, 1])).toEqual([1, 2, 3])
    })
    it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => {
    expect(sort([1.5, -4])).toEqual([-4, 1.5])
    })
    })

    View Slide

  10. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
    expect(sort([2, 3, 1])).toEqual([1, 2, 3])
    })
    it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => {
    expect(sort([1.5, -4])).toEqual([-4, 1.5])
    })
    it('transforms `[]` to `[]`', () => {
    expect(sort([])).toEqual([])
    })
    })

    View Slide

  11. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    it('transforms `[2, 3, 1]` to `[1, 2, 3]`', () => {
    expect(sort([2, 3, 1])).toEqual([1, 2, 3])
    })
    it('transforms `[1.5, -4]` to `[-4, 1.5]`', () => {
    expect(sort([1.5, -4])).toEqual([-4, 1.5])
    })
    it('transforms `[]` to `[]`', () => {
    expect(sort([])).toEqual([])
    })
    it('transforms `[1, 1, 1, 1, 1]` to `[1, 1, 1, 1, 1]`', () => {
    expect(sort([1, 1, 1, 1, 1])).toEqual([1, 1, 1, 1, 1])
    })
    it('transforms `[999, 500, 2]` to `[999. 500, 2]`', () => {
    expect(sort([999, 500, 2])).toEqual([2, 500, 999])
    })
    it('transforms `[0x821, 1.3e5]` to `[0x821, 1.3e5]`', () => {
    expect(sort([0x821, 1.3e5])).toEqual([0x821, 1.3e5])
    })
    it('transforms `[0]` to `[0]`', () => {
    expect(sort([0])).toEqual([0])
    })
    it('transforms `[-0]` to `[-0]`', () => {
    expect(sort([-0])).toEqual([-0])
    })
    it('transforms `[1, 2, 1, 2]` to `[1, 1, 2, 2]`', () => {
    expect(sort([1, 2, 1, 2])).toEqual([1, 1, 2, 2])
    })
    it('transforms `[10, 11, 12, 9001, -1]` to `[-1, 10, 11, 12, 9001]`', () => {
    expect(sort([10, 11, 12, 9001, -1])).toEqual([-1, 10, 11, 12, 9001])
    })
    it('transforms `[0.9999999, 1, 0.9999999]` to `[0.9999999, 0.9999999, 1]`', () => {
    expect(sort([0.9999999, 1, 0.9999999])).toEqual([0.9999999, 0.9999999, 1])

    View Slide

  12. It's a bit of a pain…

    View Slide


  13. But are we confident?

    View Slide

  14. ERROR: cannot read

    'name' of undefined
    100%
    coverage
    Build
    succeeded
    Tests passed
    As humans, we focus

    on the "happy path"

    View Slide

  15. Surely there is

    a better way…

    View Slide

  16. What are we saying in this test? What is the *point*?
    describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    })

    View Slide

  17. describe('`sort`', () => {
    it('transforms `[2, 1, 3]` to `[1, 2, 3]`', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    }) Is it this particular array we care about?

    View Slide

  18. describe('`sort`', () => {
    it('results in an ordered array', () => {
    expect(sort([2, 1, 3])).toEqual([1, 2, 3])
    })
    }) …or is it this particular *law*?
    aka "property" (of sorting)?
    aka "invariant" (behavior, regardless of input)

    View Slide

  19. • Sorting a list of nums results in an ordered list
    • Sorting a sorted list is a no-op (idempotence)
    • Sorting neither adds, removes, nor changes elements

    View Slide


  20. // :: [Number] -> Bool
    function sortOrders (arr) {
    const sorted = sort(arr)
    for (let i = 0; i < arr.length - 1; i ++) {
    if (arr[i] > arr[i + 1]) return false
    }
    return true
    }
    a predicate function (returns Bool) aka property check
    we've explicitly named our law

    View Slide

  21. What if we generate

    the tests?

    View Slide

  22. // :: (Number) -> Number
    function randomInt (limit = 1) {
    return Math.floor(Math.random() * limit)
    }
    // :: () -> [Number]
    function randomArr () {
    return Array.from(
    { length: randomInt(LIMIT_LEN) },
    () => randomInt(LIMIT_NUM)
    )
    }
    ad-hoc data generators

    View Slide

  23. describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    for (let t = 0; t < LIMIT_TESTS; t ++) {
    expect(isOrdered(sort(randomArr()))).toBe(true)
    }
    })
    }) generate and verify many random cases
    ✅ Passes!

    View Slide

  24. describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    for (let t = 0; t < LIMIT_TESTS; t ++) {
    expect(isOrdered(sort(randomArr()))).toBe(true)
    }
    })
    }) …but what happens if it fails?

    View Slide

  25. FAIL src/01-sort.spec.ts
    ● `sort` › outputs ordered arrays
    expect(received).toBe(expected) // Object.is equality
    Expected: true
    Received: false

    View Slide

  26. FAIL src/01-sort.spec.ts
    ● `sort` › outputs ordered arrays
    expect(received).toBe(expected) // Object.is equality
    Expected: true
    Received: false
    which input array failed?
    what was the output from sort?
    …let's just add a console.log…

    View Slide

  27. …let's just add a console.log…
    ✅ Passes!


    Oops! Nondeterminism strikes again!

    View Slide

  28. …re-ran, failed again… but why does THIS array fail?

    FAIL `sort` › outputs ordered arrays
    Input: [93, 2, 0, 1, 90101, 29, 350, 301, 620]
    Output: [0, 1, 2, 29, 301, 250, 620, 90101, 93]
    Expected: true
    Received: false

    View Slide

  29. Trickier than we thought

    View Slide

  30. laborious ad-hoc error-prone generators
    (did you notice we never generated negative ints?)
    random array generator maybe repeated empty arr
    non-instructive failure (input? output?)
    irreproducible
    failing cases may be unnecessarily large

    View Slide

  31. Surely there is

    a better way (again)…

    View Slide

  32. Use a

    Property Testing Library

    View Slide

  33. QuickCheck
    Claessen & Hughes

    View Slide

  34. JSVerify
    Oleg Grenrus
    Fast-Check
    Nicolas Dubien

    View Slide

  35. ⚖ specify law (predicate function)
    ⏳ support for async predicates
    generates progressively bigger random inputs
    . common edge cases baked-in
    failing cases progressively "shrunk" (simplified)
    ⚠ finds error boundary
    deterministically pseudorandom
    can specify seed

    View Slide

  36. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })

    View Slide

  37. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    a fast-check "property" instance, which…

    View Slide

  38. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    generates arrays of ints, and…

    View Slide

  39. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    feeds them to this predicate

    View Slide

  40. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    )
    )
    })
    })
    run many times, and throw upon failure…

    View Slide

  41. PASS src/01-sort.spec.ts
    `sort`
    ✓ outputs ordered arrays (11ms)
    maybe it passes…

    View Slide

  42. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    but if it fails, it fails elegantly

    View Slide

  43. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    ran until failure (or configured limit)

    View Slide

  44. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    failure was simplified 7 times

    View Slide

  45. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    minimal edge case found

    View Slide

  46. FAIL src/01-sort.spec.ts
    `sort`
    ✕ outputs ordered arrays (9ms)
    ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    can reproduce at will

    View Slide

  47. test assertion takes an optional config
    import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    ),
    )
    })
    })

    View Slide

  48. import * as fc from 'fast-check'
    describe('`sort`', () => {
    it('outputs ordered arrays', () => {
    fc.assert(
    fc.property(
    fc.array(fc.integer()),
    (nums) => isOrdered(sort(nums))
    ),
    { seed: 1213330267, verbose: true }
    )
    })
    })
    can specify PRNG seed

    View Slide

  49. ● `sort` › outputs ordered arrays
    Property failed after 4 tests
    { seed: 1213330267, path: "3:1:3:2:3:2:3:2", endOnFailure: true }
    Counterexample: [[-1,-2]]
    Shrunk 7 time(s)
    Encountered failures were:
    - [[-10,-1823440029,-3,-8,109548166,22,236538465]]
    - [[-1823440029,-3,-8,22]]
    - [[-1823440029,-2,-8,22]]
    - [[-1,-1823440029,-8,22]]
    - [[-1,-8,22]]
    - [[-1,-8]]
    - [[-1,-4]]
    - [[-1,-2]]
    debug same code path, and/or
    see how minimal case was found

    View Slide

  50. "Very cute, but I need
    more complex inputs."

    View Slide

  51. describe("(one of) De Morgans Laws", () => {
    test('not (P and Q) <=> (not P) or (not Q)', () => {
    fc.assert(fc.property(
    fc.boolean(), fc.boolean(),
    (p, q) => { return !(p && q) === !p || !q }
    ))
    })
    })
    can easily request multiple arbitraries
    (generator + shrinker = "arbitrary")

    View Slide

  52. describe('`Array#filter`', () => {
    it('always results in a same-or-shorter-length arr', () => {
    fc.assert(fc.property(
    fc.array(fc.anything()),
    fc.func(fc.boolean()),
    (arr, pred) => {
    return arr.filter(pred).length <= arr.length
    }
    ))
    })
    })
    built-in arbitraries compose beautifully
    ("combinator pattern")
    array of any vals
    function returning bools

    View Slide

  53. describe('`isOdd`', () => {
    it('correctly identifies odd integers', () => {
    fc.assert(fc.property(
    fc.integer().map(n => n * 2 + 1),
    isOdd
    ))
    })
    })
    function isOdd (num) { return num % 2 === 1 }
    a new arbitrary
    which generates
    odd integers
    can transform arbitraries via functional
    techniques like `map` and `filter`
    (fails on -1!)

    View Slide

  54. tons more!
    size minima / maxima
    ✍ special strings (ascii, unicode, ipv4, json)
    custom generators from scratch

    View Slide

  55. Monads
    and even…










    View Slide

  56. Summary

    View Slide

  57. Complementary to other strats
    Unit tests for specific important cases
    Fuzzers for robustness against garbage input
    Prop tests for confidence that API contract holds
    <- property-based testing
    image credit: Nicolas Dubien

    View Slide

  58. Plain JS -> zero-effort integration
    No special plugins, matchers, or bindings needed
    Drop into Mocha, AVA, Jest, Tape, etc.
    Insert assertions from Chai, Sinon, Supertest, etc.

    View Slide

  59. THANKS!

    - https: //speakerdeck.com/glebec/the-magic-of-generative-testing
    - https: //dev.to/glebec/property-testing-with-jsverify-part-i-3one
    - https: // www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf
    - YouTube: "Don't Write Tests"! (John Hughes)
    - https: //github.com/dubzzz/fast-check

    View Slide