Slide 1

Slide 1 text

I S A M I N D S E T P R O P E R T Y - B A S E D T E S T I N G

Slide 2

Slide 2 text

@whatyouhide

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

https://weedmaps.com/careers

Slide 6

Slide 6 text

T E S T I N G

Slide 7

Slide 7 text

why do we test? no tests yes tests

Slide 8

Slide 8 text

unit tests

Slide 9

Slide 9 text

test "sorting" do assert sort([]) == [] assert sort([1, 2, 3]) == [1, 2, 3] assert sort([2, 1, 3]) == [1, 2, 3] end example-based

Slide 10

Slide 10 text

table-based Input Output [] [] [1, 2, 3] [1, 2, 3] [2, 1, 3] [1, 2, 3]

Slide 11

Slide 11 text

unit tests , but also

Slide 12

Slide 12 text

P R O P E R T I E S

Slide 13

Slide 13 text

hard to write , but...

Slide 14

Slide 14 text

valid inputs properties of output testing framework

Slide 15

Slide 15 text

github.com/whatyouhide/stream_data

Slide 16

Slide 16 text

example time: sorting lists

Slide 17

Slide 17 text

test "sorting" do assert sort([]) == [] assert sort([1, 2, 3]) == [1, 2, 3] assert sort([2, 1, 3]) == [1, 2, 3] end

Slide 18

Slide 18 text

lists of integers

Slide 19

Slide 19 text

it's a list has the same elements it's ordered

Slide 20

Slide 20 text

check all list <- list_of(int()) do sorted = sort(list) assert is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end

Slide 21

Slide 21 text

check all list <- list_of(int()) do sorted = sort(list) assert is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end

Slide 22

Slide 22 text

check all list <- list_of(int()) do sorted = sort(list) assert is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end

Slide 23

Slide 23 text

check all list <- list_of(int()) do sorted = sort(list) assert is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end

Slide 24

Slide 24 text

check all list <- list_of(int()) do sorted = sort(list) assert is_list(sorted) assert same_elements?(list, sorted) assert ordered?(sorted) end

Slide 25

Slide 25 text

def sort(list), do: list

Slide 26

Slide 26 text

[32, 2, 44, -12] [1, 0]

Slide 27

Slide 27 text

G E N E R A T O R S

Slide 28

Slide 28 text

iex> Enum.take(integer(), 4) [1, 0, -3, 1]

Slide 29

Slide 29 text

Composability

Slide 30

Slide 30 text

number = one_of([integer(), float()])

Slide 31

Slide 31 text

StreamData.map(integer(), &abs/1)

Slide 32

Slide 32 text

def string(:ascii) do integer(?\s..?~) |> list_of() |> map(&List.to_string/1) end Example

Slide 33

Slide 33 text

⚠ Keep shrinkability

Slide 34

Slide 34 text

constant(term) bind_filter(gen, fun) +

Slide 35

Slide 35 text

iex> Enum.take(constant(:foo), 4) [:foo, :foo, :foo, :foo] constant always generates a term

Slide 36

Slide 36 text

bind_filter(integer(), fn i -> if i < 0 do :skip else gen = map(list_of(integer()), &(&1 + i)) {:cont, gen} end end) bind_filter (possibly) creates generators from generated terms

Slide 37

Slide 37 text

P A T T E R N S

Slide 38

Slide 38 text

circular code

Slide 39

Slide 39 text

decode(encode(term)) == term

Slide 40

Slide 40 text

property "unicode escaping" do check all string <- string(:printable) do encoded = encode(string, escape: :unicode) assert decode(encoded) == string end end JSON encoding

Slide 41

Slide 41 text

oracle model

Slide 42

Slide 42 text

my_code() == oracle_code()

Slide 43

Slide 43 text

older system less performant implementation

Slide 44

Slide 44 text

property "gives same results as Erlang impl" do check all bin <- binary() do assert Huffman.encode(bin) == :huffman.encode(bin) end end

Slide 45

Slide 45 text

smoke tests https://www.youtube.com/watch?v=jvwfDdgg93E

Slide 46

Slide 46 text

API: 200, 201, 400, 404 https://www.youtube.com/watch?v=jvwfDdgg93E

Slide 47

Slide 47 text

https://www.youtube.com/watch?v=jvwfDdgg93E property "only expected codes are returned" do check all request <- request() do response = HTTP.perform(request) assert response.status in [200, 201, 400, 404] end end

Slide 48

Slide 48 text

locally , CI

Slide 49

Slide 49 text

if ci?() do config :stream_data, max_runs: 500 else config :stream_data, max_runs: 25 end

Slide 50

Slide 50 text

unit + properties

Slide 51

Slide 51 text

property "String.contains?/2" do check all left <- string(), right <- string() do assert String.contains?(left <> right, left) assert String.contains?(left <> right, right) end end test "String.contains?/2" do assert String.contains?("foobar", "foo") assert String.contains?("foobar", "bar") assert String.contains?("foobar", "ob") end +

Slide 52

Slide 52 text

S T A T E F U L T E S T I N G

Slide 53

Slide 53 text

model valid commands +

Slide 54

Slide 54 text

model: state + state transformations

Slide 55

Slide 55 text

commands: calls + preconditions

Slide 56

Slide 56 text

getting/setting keys in Redis

Slide 57

Slide 57 text

model system Redis %{}

Slide 58

Slide 58 text

•get(key) •set(key, value) commands

Slide 59

Slide 59 text

def get(model, key), do: Map.get(model, key) def set(model, key, value), do: Map.put(model, key, value)

Slide 60

Slide 60 text

keys = Map.keys(model) one_of([ command(:get, [one_of(keys)]), command(:set, [binary(), binary()]), command(:set, [one_of(keys), binary()]) ])

Slide 61

Slide 61 text

{:ok, result} = Redix.command!(conn, ["GET", key]) assert Map.fetch(model, key) == {:ok, result} Redix.command!(conn, ["SET", key, value]) Map.replace!(model, key, value) get set_existing

Slide 62

Slide 62 text

LevelDB

Slide 63

Slide 63 text

17 (seventeen) calls 33 (thirty three) calls

Slide 64

Slide 64 text

R E S E A R C H

Slide 65

Slide 65 text

trees are hard

Slide 66

Slide 66 text

defmodule Tree do def tree() do StreamData.tree(:leaf, fn leaf -> {leaf, leaf} end) end def size({l, r}), do: 1 + size(l) + size(r) def size(_leaf), do: 1 def depth({l, r}), do: 1 + max(depth(l), depth(r)) def depth(_leaf), do: 1 end

Slide 67

Slide 67 text

Generation size: 10 Avg size: 4.9 Avg max depth: 2.473 Generation size: 100 Avg size: 10.892 Avg max depth: 3.466 Generation size: 1000 Avg size: 22.732 Avg max depth: 4.507

Slide 68

Slide 68 text

stream_data + dialyzer

Slide 69

Slide 69 text

Generators from type

Slide 70

Slide 70 text

@type timeout() :: :infinity | non_neg_integer()

Slide 71

Slide 71 text

from_type(timeout()) one_of([:infinity, map(integer(), &abs/1)])

Slide 72

Slide 72 text

Automatic typespec property checking

Slide 73

Slide 73 text

@spec my_fun(timeout()) :: :ok | :error check all timeout <- from_type(timeout()) do assert my_fun(timeout) in [:ok, :error] end

Slide 74

Slide 74 text

C O N C L U S I O N

Slide 75

Slide 75 text

find obscure bugs reduce to minimal failing input find specification errors cover vast input space

Slide 76

Slide 76 text

v1.7

Slide 77

Slide 77 text

use stream_data

Slide 78

Slide 78 text

use property-based testing

Slide 79

Slide 79 text

@whatyouhide github.com/whatyouhide/stream_data