Property-based Testing is a Mindset

Property-based Testing is a Mindset

Faafc04d9e69b73b9f49995fd4c94d4d?s=128

Andrea Leopardi

December 06, 2017
Tweet

Transcript

  1. I S A M I N D S E T

    P R O P E R T Y - B A S E D T E S T I N G
  2. @whatyouhide

  3. None
  4. Elixir

  5. T E S T I N G

  6. why do we test? no tests yes tests

  7. unit tests

  8. test "sorting" do assert sort([]) == [] assert sort([1, 2,

    3]) == [1, 2, 3] assert sort([2, 1, 3]) == [1, 2, 3] end example-based
  9. table-based Input Output [] [] [1, 2, 3] [1, 2,

    3] [2, 1, 3] [1, 2, 3]
  10. unit tests , but also

  11. P R O P E R T I E S

  12. hard to write , but...

  13. valid inputs properties of output testing framework

  14. github.com/whatyouhide/stream_data

  15. example time: sorting lists

  16. test "sorting" do assert sort([]) == [] assert sort([1, 2,

    3]) == [1, 2, 3] assert sort([2, 1, 3]) == [1, 2, 3] end
  17. lists of integers

  18. it's a list has the same elements it's ordered

  19. check all list <- list_of(int()) do sorted = sort(list) assert

    is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end
  20. check all list <- list_of(int()) do sorted = sort(list) assert

    is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end
  21. check all list <- list_of(int()) do sorted = sort(list) assert

    is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end
  22. check all list <- list_of(int()) do sorted = sort(list) assert

    is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end
  23. check all list <- list_of(int()) do sorted = sort(list) assert

    is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end
  24. def sort(list), do: list

  25. [32, 2, 44, -12] [1, 0] *shrinking

  26. P A T T E R N S

  27. circular code

  28. decode(encode(term)) == term

  29. property "encoding->decoding is circular" do check all bin <- binary()

    do encoded = Huffman.encode(bin) assert is_binary(encoded) assert Huffman.decode(encoded) == bin end end Huffman encoding
  30. oracle model

  31. my_code() == oracle_code()

  32. older system less performant implementation

  33. property "gives same results as Erlang impl" do check all

    bin <- binary() do assert Huffman.encode(bin) == :huffman.encode(bin) end end
  34. smoke tests https://www.youtube.com/watch?v=jvwfDdgg93E

  35. API: 200, 201, 400, 404 https://www.youtube.com/watch?v=jvwfDdgg93E

  36. https://www.youtube.com/watch?v=jvwfDdgg93E property "only expected codes are returned" do check all

    request <- request() do response = HTTP.perform(request) assert response.status in [200, 201, 400, 404] end end
  37. locally , CI

  38. if System.get_env("CI") == "true" do config :stream_data, max_runs: 500 else

    config :stream_data, max_runs: 25 end
  39. unit + properties

  40. R E A L - W O R L D

    E X A M P L E
  41. Redis parser

  42. continuation parser parse(bytes) {:ok, command} {:more, function}

  43. property "splitting commands at random" do check all cmd <-

    command(), split_cmd <- random_splits(cmd) do assert {:ok, _, ""} = parse_many(split_cmd) end end
  44. property "splitting commands at random" do check all cmd <-

    command(), split_cmd <- random_splits(cmd) do assert {:ok, _, ""} = parse_many(split_cmd) end end
  45. property "splitting commands at random" do check all cmd <-

    command(), split_cmd <- random_splits(cmd) do assert {:ok, _, ""} = parse_many(split_cmd) end end
  46. S T A T E F U L T E

    S T I N G
  47. model valid commands +

  48. model: state + state transformations

  49. commands: calls + preconditions

  50. bounded-size queue

  51. Max size model system

  52. •push(queue, value) •resize(queue, max_size) commands

  53. def push(model, _value), do: model def resize(_model, max_size), do: max_size

  54. push(queue, {:some, "value"}) push(queue, [123, 3.2]) resize(queue, 1) push(queue, %{})

  55. assert size(queue) <= model

  56. LevelDB

  57. 17 (seventeen) calls 33 (thirty three) calls

  58. C O N C L U S I O N

  59. find obscure bugs reduce to minimal failing input find specification

    errors keep increasing input space
  60. v1.7

  61. use stream_data

  62. use property-based testing

  63. @whatyouhide github.com/whatyouhide/stream_data