Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Generative Testing in Elixir

Generative Testing in Elixir

Automated test suites are invaluable. They provide protection against regressions and can serve as a design tool when building new apis. But, despite this protection bugs still slip through. We could try to write more tests but attempting to cover every edge case is an untenable problem. Luckily, we can use property based testing to generate edge cases for us.

Originally developed in Haskell, property tests have spread to many other languages. In this talk we’ll discuss the basics of property testing, demonstrate how we can determine properties for our system, and look at real world examples of property tests using elixir.

06f8b41980eb4c577fa40c41d5030c19?s=128

Chris Keathley

January 13, 2017
Tweet

Transcript

  1. Don’t write tests; Generate them Chris Keathley / @ChrisKeathley /

    c@keathley.io
  2. What are we gunna talk about?

  3. Testing today What are property tests Basic Example Real world(ish)

    example
  4. Why Elixir?

  5. None
  6. Elixir is a functional, dynamic language that targets the Erlang

    VM (BEAM)
  7. Testing

  8. TDD

  9. 1. Write a failing test 2. Write enough code to

    make that test pass 3. Refactor Test Driven Development
  10. Validation Protection From Regression Design Test Driven Development

  11. Career Happiness TDD quiet contemplation Trough of disillusionment Property Tests!!!

  12. Tests provide guard rails

  13. Tests directly couple your implementation Test Api v1

  14. Tests directly couple your implementation Test Api v1.3

  15. Tests directly couple your implementation Test Api v1.3

  16. Write as few tests as possible

  17. Warning: Contrived Strawman Argument incoming!

  18. Lets TDD Addition! x + y = ?

  19. test "adding 2 numbers" do end

  20. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 end
  21. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 end def add(x, y) do end
  22. None
  23. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 end def add(x, y) do 2 end
  24. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 end def add(_x, _y) do 2 end
  25. None
  26. But wait…

  27. None
  28. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(_x, _y) do 2 end
  29. Pattern Matching

  30. Pattern Matching

  31. Pattern Matching x = 3

  32. Pattern Matching x = 3 y = x

  33. Pattern Matching x = 3 y = x 3 =

    y
  34. Pattern Matching is an assertion

  35. Pattern Matching x = 3 3 = x

  36. Pattern Matching 3 = 3

  37. Pattern Matching %{name: "Chris", hobbies: ["Coffee", "Pinball", "Lego"]}

  38. Pattern Matching %{name: user_name} = %{name: "Chris", hobbies: ["Coffee", "Pinball",

    "Lego"]}
  39. Pattern Matching user_name = "Chris"

  40. Pattern Matching def user_name(user_map) do %{name: name} = user_map name

    end
  41. Pattern Matching def user_name(%{name: name}) do name end

  42. Pattern Matching def user_name(%{name: name}) do name end def user_name(_),

    do: "Default User"
  43. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(_x, _y) do 2 end
  44. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(_x, _y), do: 2
  45. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(3, _), do: 7 def add(_x, _y), do: 2
  46. None
  47. But wait…

  48. None
  49. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(3, _), do: 7 def add(_x, _y), do: 2
  50. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(3, _), do: 7 def add(_x, _y), do: 2
  51. None
  52. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(3, _), do: 7 def add(_x, _y), do: 2
  53. Guard clauses

  54. Guard clauses def user_name(%{name: name}) do name end

  55. Guard clauses def user_name(%{name: name}) when is_binary(name) do name end

  56. Guard clauses def user_name(%{name: name}) when is_binary(name) do name end

    def user_name(%{name: name, age: age}) when age < 20
  57. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(3, _), do: 7 def add(_x, _y), do: 2
  58. test "adding 2 numbers" do assert add(1, 1) == 2

    assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(x, _) when x < 0, do: 3 def add(3, _), do: 7 def add(_x, _y), do: 2
  59. None
  60. None
  61. None
  62. None
  63. What have we gained?

  64. Validation Protection From Regression Design TDD

  65. Validation Protection From Regression Design TDD

  66. Validation Protection From Regression Design TDD

  67. Validation Protection From Regression Design ? TDD

  68. There are bugs in your code

  69. Writing tests for one feature

  70. Writing tests for one feature o(n)

  71. Writing tests for two features o(n) o(n^2)

  72. Writing tests for three features o(n) o(n^2) o(n^3)

  73. Race conditions?

  74. Property tests

  75. None
  76. Expected == Actual

  77. Expected == Actual Overly Specific

  78. None
  79. Property Tests

  80. Property Tests Int

  81. Property Tests Int p(x)

  82. Property Tests Int p(x) ?

  83. Property Tests Int p(x) ? Invariant

  84. Invariant: “a function, quantity, or property that remains unchanged when

    a specified transformation is applied.”
  85. Basic Property Tests

  86. What is true about addition?

  87. x + 0 == x

  88. test "addition with zero returns the same number" do end

    def add(_x, _y) do end
  89. test "addition with zero returns the same number" do ptest

    do end end def add(_x, _y) do end
  90. test "addition with zero returns the same number" do ptest

    x: int() do end end def add(_x, _y) do end
  91. test "addition with zero returns the same number" do ptest

    x: int() do assert add(x, 0) == x end end def add(_x, _y) do end
  92. None
  93. test "addition with zero returns the same number" do ptest

    x: int() do assert add(x, 0) == x end end def add(_x, _y) do end
  94. test "addition with zero returns the same number" do ptest

    x: int() do assert add(x, 0) == x end end def add(x, _y) do x end
  95. x + y == y + x

  96. test "addition is commutative" do end def add(x, _y) do

    x end
  97. test "addition is commutative" do ptest x: int(), y: int()

    do end end def add(x, _y) do x end
  98. test "addition is commutative" do ptest x: int(), y: int()

    do assert add(x, y) == add(y, x) end end def add(x, _y) do x end
  99. None
  100. test "addition is commutative" do ptest x: int(), y: int()

    do assert add(x, y) == add(y, x) end end def add(x, _y) do x end
  101. test "addition is commutative" do ptest x: int(), y: int()

    do assert add(x, y) == add(y, x) end end def add(x, y) do x * y end
  102. None
  103. test "addition is commutative" do ptest x: int(), y: int()

    do assert add(x, y) == add(y, x) end end def add(x, y) do x * y end
  104. test "addition is commutative" do ptest x: int(), y: int()

    do assert add(x, y) == add(y, x) end end def add(x, y) do x * y end def add(x, 0), do: x
  105. None
  106. (1 + x) + y == x + (1 +

    y)
  107. def add(x, y) do x * y end def add(x,

    0), do: x test "addition is asociative" do end
  108. def add(x, y) do x * y end def add(x,

    0), do: x test "addition is asociative" do ptest x: int(), y: int(), z: int() do end end
  109. def add(x, y) do x * y end def add(x,

    0), do: x test "addition is asociative" do ptest x: int(), y: int(), z: int() do assert add(x, add(y, z)) == add(add(x, y), z) end end
  110. def add(x, y) do x * y end def add(x,

    0), do: x test "addition is asociative" do ptest x: int(), y: int(), z: int() do assert add(x, add(y, z)) == add(add(x, y), z) end end
  111. None
  112. def add(x, 0), do: x def add(x, y) do x

    * y end
  113. def add(x, y) do x + y end

  114. A Real-ish Example

  115. None
  116. Modeling the application ? p(x) ?

  117. Modeling the application ? p(x) ? What is the input

    domain?
  118. Vote Vote Vote User

  119. Vote Vote Vote User

  120. Vote Vote Vote User

  121. Vote Vote Vote User

  122. Modeling Users as FSMs logged_out logged_in login logout vote

  123. Generate Commands

  124. Generate Commands

  125. Generate Commands

  126. Generate Commands

  127. Generate Commands

  128. Generate Commands

  129. Generated Commands [{:vote, "chris", 1}, {:vote, "chris", 2}, {:vote, "jane",

    1}, {:vote, "jane", 1}, {:vote, "jane", 3} {:vote, "chris", 2}]
  130. Generated Commands [{:vote, "chris", 1}, {:vote, "chris", 2}, {:vote, "jane",

    1}, {:vote, "jane", 1}, {:vote, "jane", 3} {:vote, "chris", 2}] [{:vote, "chris", 1}, {:vote, "jane", 1}]
  131. Property: Users votes should increase

  132. Property: Users votes should increase test “users votes increase after

    voting" do end
  133. Property: Users votes should increase test “users votes increase after

    voting" do ptest [commands: gen_commands("chris")] do end end
  134. Property: Users votes should increase test “users votes increase after

    voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() end end
  135. Property: Users votes should increase test “users votes increase after

    voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) end end
  136. Property: Users votes should increase test “users votes increase after

    voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) assert result end end
  137. Property: Users votes should increase test “users votes increase after

    voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) assert result end end def run_commands(commands, module) do Enum.reduce( commands, {0, true}, & run_command(module, &1, &2) ) end
  138. def gen_commands(name) do end Command Generators

  139. def gen_commands(name) do list(of: gen_vote(name), max: 20) end Command Generators

  140. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name)

    do end Command Generators
  141. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name)

    do tuple(like: { )}) end Command Generators
  142. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name)

    do tuple(like: { value(:vote), )}) end Command Generators
  143. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name)

    do tuple(like: { value(:vote), value(name), )}) end Command Generators
  144. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name)

    do tuple(like: { value(:vote), value(name), choose(from: [value(1), value(2), value(3)])}) end Command Generators
  145. defmodule ClientStateMachine do end

  146. defmodule ClientStateMachine do def vote(name, id) do %{"votes" => new_votes}

    = post(id, name) {:ok, new_votes} end end
  147. defmodule ClientStateMachine do def vote(name, id) do %{"votes" => new_votes}

    = post(id, name) {:ok, new_votes} end def vote_next(state, [id, name], _result) do {:ok, update_in(state, [name, to_string(id)], &(&1 + 1))} end end
  148. defmodule ClientStateMachine do def vote(name, id) do %{"votes" => new_votes}

    = post(id, name) {:ok, new_votes} end def vote_next(state, [id, name], _result) do {:ok, update_in(state, [name, to_string(id)], &(&1 + 1))} end def vote_post(state, [id, name], actual_result) do expected_result = get_in(state, [name, to_string(id)]) + 1 {:ok, actual_result == expected_result} end end
  149. Property: Users votes should increase test “users votes increase after

    voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) assert result end end
  150. None
  151. Property: Users shouldn’t effect other users votes

  152. Property: Users should get the correct votes test "users don't

    effect each others votes" do end
  153. Property: Users should get the correct votes test "users don't

    effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do end end
  154. Property: Users should get the correct votes test "users don't

    effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() end end
  155. Property: Users should get the correct votes test "users don't

    effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() {_state, result} = run_commands([chris, jane], Client) end end
  156. Property: Users should get the correct votes test "users don't

    effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() {_state, result} = run_commands([chris, jane], Client) assert result end end
  157. Property: Users should get the correct votes test "users don't

    effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() {_state, result} = run_parallel_commands([chris, jane], Client) assert result end end
  158. Running parallel tests def run_parallel_commands([l1, l2], module) do t1 =

    Task.async(fn -> run_commands(l1, module) end) t2 = Task.async(fn -> run_commands(l2, module) end) {_, ra} = Task.await(t1) {_, rb} = Task.await(t2) {:ok, ra && rb} end
  159. None
  160. None
  161. The Bug def new(conn, %{"id" => id, "name" => name})

    do {:ok, current_votes} = VoteCounter.get(id) new_votes = [name | current_votes] VoteCounter.put(id, new_votes) # Other nonsense end
  162. The Bug Votes

  163. The Bug Votes 3 3

  164. The Bug Votes 3 3

  165. The Bug Votes 4 4

  166. The Bug Votes 4 4

  167. The Bug 3 + 1 + 1 == 4 ?

  168. The Bug def new(conn, %{"id" => id, "name" => name})

    do {:ok, current_votes} = VoteCounter.get(id) new_votes = [name | current_votes] VoteCounter.put(id, new_votes) # Other nonsense end
  169. The Bug def new(conn, %{"id" => id, "name" => name})

    do {:ok, new_votes} = VoteCounter.incr(id, name) # Other nonsense end
  170. None
  171. Conclusion

  172. How to think in properties Generating data Generate commands Model

    users as FSMs
  173. Resources: “Finding Race conditions in Erlang with QuickCheck and PULSE”

    http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.724.3518&rep=rep1&type=pdf Testing Async apis with QuickCheck https://www.youtube.com/watch?v=iW2J7Of8jsE&t=272s “QuickCheck: A lightweight tool for Random Testing of Haskell Programs” http://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf Composing Test Generators https://www.youtube.com/watch?v=4-sPhFtGwZk Property based testing for better code https://www.youtube.com/watch?v=shngiiBfD80
  174. If you could write less code and find more bugs

    would you do that?
  175. Thanks Chris Keathley / @ChrisKeathley / c@keathley.io