Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Property-based Testing is a Mindset

Property-based Testing is a Mindset

Andrea Leopardi

December 06, 2017
Tweet

More Decks by Andrea Leopardi

Other Decks in Programming

Transcript

  1. I S A M I N D S E T
    P R O P E R T Y - B A S E D T E S T I N G

    View Slide

  2. @whatyouhide

    View Slide

  3. View Slide

  4. Elixir

    View Slide

  5. T E S T I N G

    View Slide

  6. why do we test?
    no tests
    yes tests

    View Slide

  7. unit tests

    View Slide

  8. test "sorting" do
    assert sort([]) == []
    assert sort([1, 2, 3]) == [1, 2, 3]
    assert sort([2, 1, 3]) == [1, 2, 3]
    end
    example-based

    View Slide

  9. table-based
    Input Output
    [] []
    [1, 2, 3] [1, 2, 3]
    [2, 1, 3] [1, 2, 3]

    View Slide

  10. unit tests
    , but also

    View Slide

  11. P R O P E R T I E S

    View Slide

  12. hard to write , but...

    View Slide

  13. valid inputs
    properties of output
    testing framework

    View Slide

  14. github.com/whatyouhide/stream_data

    View Slide

  15. example time:
    sorting lists

    View Slide

  16. test "sorting" do
    assert sort([]) == []
    assert sort([1, 2, 3]) == [1, 2, 3]
    assert sort([2, 1, 3]) == [1, 2, 3]
    end

    View Slide

  17. lists of integers

    View Slide

  18. it's a list
    has the same elements
    it's ordered

    View Slide

  19. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end

    View Slide

  20. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end

    View Slide

  21. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end

    View Slide

  22. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end

    View Slide

  23. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end

    View Slide

  24. def sort(list), do: list

    View Slide

  25. [32, 2, 44, -12]
    [1, 0]
    *shrinking

    View Slide

  26. P A T T E R N S

    View Slide

  27. circular code

    View Slide

  28. decode(encode(term)) == term

    View Slide

  29. property "encoding->decoding is circular" do
    check all bin <- binary() do
    encoded = Huffman.encode(bin)
    assert is_binary(encoded)
    assert Huffman.decode(encoded) == bin
    end
    end
    Huffman encoding

    View Slide

  30. oracle model

    View Slide

  31. my_code() == oracle_code()

    View Slide

  32. older system
    less performant implementation

    View Slide

  33. property "gives same results as Erlang impl" do
    check all bin <- binary() do
    assert Huffman.encode(bin) ==
    :huffman.encode(bin)
    end
    end

    View Slide

  34. smoke tests
    https://www.youtube.com/watch?v=jvwfDdgg93E

    View Slide

  35. API: 200, 201, 400, 404
    https://www.youtube.com/watch?v=jvwfDdgg93E

    View Slide

  36. https://www.youtube.com/watch?v=jvwfDdgg93E
    property "only expected codes are returned" do
    check all request <- request() do
    response = HTTP.perform(request)
    assert response.status in [200, 201, 400, 404]
    end
    end

    View Slide

  37. locally , CI

    View Slide

  38. if System.get_env("CI") == "true" do
    config :stream_data, max_runs: 500
    else
    config :stream_data, max_runs: 25
    end

    View Slide

  39. unit + properties

    View Slide

  40. R E A L - W O R L D
    E X A M P L E

    View Slide

  41. Redis parser

    View Slide

  42. continuation parser
    parse(bytes)
    {:ok, command} {:more, function}

    View Slide

  43. property "splitting commands at random" do
    check all cmd <- command(),
    split_cmd <- random_splits(cmd) do
    assert {:ok, _, ""} = parse_many(split_cmd)
    end
    end

    View Slide

  44. property "splitting commands at random" do
    check all cmd <- command(),
    split_cmd <- random_splits(cmd) do
    assert {:ok, _, ""} = parse_many(split_cmd)
    end
    end

    View Slide

  45. property "splitting commands at random" do
    check all cmd <- command(),
    split_cmd <- random_splits(cmd) do
    assert {:ok, _, ""} = parse_many(split_cmd)
    end
    end

    View Slide

  46. S T A T E F U L
    T E S T I N G

    View Slide

  47. model
    valid commands
    +

    View Slide

  48. model: state + state transformations

    View Slide

  49. commands: calls + preconditions

    View Slide

  50. bounded-size queue

    View Slide

  51. Max size
    model
    system

    View Slide

  52. •push(queue, value)
    •resize(queue, max_size)
    commands

    View Slide

  53. def push(model, _value), do: model
    def resize(_model, max_size), do: max_size

    View Slide

  54. push(queue, {:some, "value"})
    push(queue, [123, 3.2])
    resize(queue, 1)
    push(queue, %{})

    View Slide

  55. assert size(queue) <= model

    View Slide

  56. LevelDB

    View Slide

  57. 17 (seventeen) calls
    33 (thirty three) calls

    View Slide

  58. C O N C L U S I O N

    View Slide

  59. find obscure bugs
    reduce to minimal failing input
    find specification errors
    keep increasing input space

    View Slide

  60. v1.7

    View Slide

  61. use stream_data

    View Slide

  62. use property-based testing

    View Slide

  63. @whatyouhide
    github.com/whatyouhide/stream_data

    View Slide