Property-based Testing is a Mindset

Transcript

1. I S A M I N D S E T
   P R O P E R T Y - B A S E D T E S T I N G
   

   View Slide

2. @whatyouhide
   

   View Slide

3. View Slide

4. Elixir
   
   

   View Slide

5. T E S T I N G
   

   View Slide

6. why do we test?
   no tests
   yes tests
   

   View Slide

7. unit tests
   

   View Slide

8. test "sorting" do
   assert sort([]) == []
   assert sort([1, 2, 3]) == [1, 2, 3]
   assert sort([2, 1, 3]) == [1, 2, 3]
   end
   example-based
   

   View Slide

9. table-based
   Input Output
   [] []
   [1, 2, 3] [1, 2, 3]
   [2, 1, 3] [1, 2, 3]
   

   View Slide

10. unit tests
    , but also
    
    

    View Slide

11. P R O P E R T I E S
    

    View Slide

12. hard to write , but...
    

    View Slide

13. valid inputs
    properties of output
    testing framework
    

    View Slide

14. github.com/whatyouhide/stream_data
    

    View Slide

15. example time:
    sorting lists
    

    View Slide

16. test "sorting" do
    assert sort([]) == []
    assert sort([1, 2, 3]) == [1, 2, 3]
    assert sort([2, 1, 3]) == [1, 2, 3]
    end
    

    View Slide

17. lists of integers
    

    View Slide

18. it's a list
    has the same elements
    it's ordered
    

    View Slide

19. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end
    

    View Slide

20. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end
    

    View Slide

21. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end
    

    View Slide

22. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end
    

    View Slide

23. check all list <- list_of(int()) do
    sorted = sort(list)
    assert is_list(sorted)
    assert same_elements?(list, sorted)
    assert ordered?(sorted)
    end
    

    View Slide

24. def sort(list), do: list
    

    View Slide

25. [32, 2, 44, -12]
    [1, 0]
    *shrinking
    
    

    View Slide

26. P A T T E R N S
    

    View Slide

27. circular code
    

    View Slide

28. decode(encode(term)) == term
    

    View Slide

29. property "encoding->decoding is circular" do
    check all bin <- binary() do
    encoded = Huffman.encode(bin)
    assert is_binary(encoded)
    assert Huffman.decode(encoded) == bin
    end
    end
    Huffman encoding
    

    View Slide

30. oracle model
    

    View Slide

31. my_code() == oracle_code()
    

    View Slide

32. older system
    less performant implementation
    

    View Slide

33. property "gives same results as Erlang impl" do
    check all bin <- binary() do
    assert Huffman.encode(bin) ==
    :huffman.encode(bin)
    end
    end
    

    View Slide

34. smoke tests
    https://www.youtube.com/watch?v=jvwfDdgg93E
    

    View Slide

35. API: 200, 201, 400, 404
    https://www.youtube.com/watch?v=jvwfDdgg93E
    

    View Slide

36. https://www.youtube.com/watch?v=jvwfDdgg93E
    property "only expected codes are returned" do
    check all request <- request() do
    response = HTTP.perform(request)
    assert response.status in [200, 201, 400, 404]
    end
    end
    

    View Slide

37. locally , CI
    

    View Slide

38. if System.get_env("CI") == "true" do
    config :stream_data, max_runs: 500
    else
    config :stream_data, max_runs: 25
    end
    

    View Slide

39. unit + properties
    

    View Slide

40. R E A L - W O R L D
    E X A M P L E
    

    View Slide

41. Redis parser
    

    View Slide

42. continuation parser
    parse(bytes)
    {:ok, command} {:more, function}
    

    View Slide

43. property "splitting commands at random" do
    check all cmd <- command(),
    split_cmd <- random_splits(cmd) do
    assert {:ok, _, ""} = parse_many(split_cmd)
    end
    end
    

    View Slide

44. property "splitting commands at random" do
    check all cmd <- command(),
    split_cmd <- random_splits(cmd) do
    assert {:ok, _, ""} = parse_many(split_cmd)
    end
    end
    

    View Slide

45. property "splitting commands at random" do
    check all cmd <- command(),
    split_cmd <- random_splits(cmd) do
    assert {:ok, _, ""} = parse_many(split_cmd)
    end
    end
    

    View Slide

46. S T A T E F U L
    T E S T I N G
    

    View Slide

47. model
    valid commands
    +
    

    View Slide

48. model: state + state transformations
    

    View Slide

49. commands: calls + preconditions
    

    View Slide

50. bounded-size queue
    

    View Slide

51. Max size
    model
    system
    

    View Slide

52. •push(queue, value)
    •resize(queue, max_size)
    commands
    

    View Slide

53. def push(model, _value), do: model
    def resize(_model, max_size), do: max_size
    

    View Slide

54. push(queue, {:some, "value"})
    push(queue, [123, 3.2])
    resize(queue, 1)
    push(queue, %{})
    

    View Slide

55. assert size(queue) <= model
    

    View Slide

56. LevelDB
    

    View Slide

57. 17 (seventeen) calls
    33 (thirty three) calls
    

    View Slide

58. C O N C L U S I O N
    

    View Slide

59. find obscure bugs
    reduce to minimal failing input
    find specification errors
    keep increasing input space
    

    View Slide

60. v1.7
    

    View Slide

61. use stream_data
    

    View Slide

62. use property-based testing
    

    View Slide

63. @whatyouhide
    github.com/whatyouhide/stream_data
    

    View Slide