Hardening Applications with Property Tests

Property testing is a tool that every Elixir developer should have in their toolbox. Come learn how property tests can thoroughly test a problem space and make bulletproof software. We'll look at properties pulled from the Oban background job processor.

Some examples we'll explore:

* Proving a CRON parser
* Checking concurrent uniqueness
* Transitioning between migrations
* Proving system reliability

You'll leave armed with the ability to recognize situations where properties are more effective than example based tests.

Transcript

1. Hardening Applications with Property Tests
2. Hardening a Library with Property Tests
3. Parker Selbert @sorentwo Soren / Dscout
4. Adopting Elixir and a library named Kiq
5. Nobody likes finding bugs in production.
6. Kiq Oban Initially, an experiment to model queues as streams.
7. Starting over meant building up new example tests, with many possible regressions.
8. Property-Based Testing • PropEr • StreamData
 • Proper Book
9. Properties Aspects of code that hold for a set of randomized test data Functions that dynamically generate infinite streams of data Generators
10. EX1 Reliable Execution
11. What is absolute?
12. What is absolute? All inserted jobs are attempted at least once.
13. property "inserted jobs are executed at least once" do check all jobs <- list_of(job()) do for %{id: id, args: %{"ref" => ref}} <- Oban.insert_all(jobs) do assert_receive {:ok, ^ref} assert %{attempt: 1, attempted_at: %DateTime{}} = Repo.get(Job, job.id) end end end
14. property "inserted jobs are executed at least once" do check all jobs <- list_of(job()) do for %{id: id, args: %{"ref" => ref}} <- Oban.insert_all(jobs) do assert_receive {:ok, ^ref} assert %{attempt: 1, attempted_at: %DateTime{}} = Repo.get(Job, job.id) end end end
15. defp job do gen all queue <- member_of(~w(alpha beta gamma delta)), ref <- integer() do Job.new(%{ref: ref}, queue: queue, worker: Worker) end end
16. property "inserted jobs are executed at least once" do check all jobs <- list_of(job()) do for %{id: id, args: %{"ref" => ref}} <- Oban.insert_all(jobs) do assert_receive {:ok, ^ref} assert %{attempt: 1, attempted_at: %DateTime{}} = Repo.get(Job, job.id) end end end
17. 1) property inserted jobs are executed at least once (Oban.ExecutingTest) Failed with generated values (after 13 successful runs): * Clause: jobs <- list_of(job()) Generated: [#Ecto.Changeset<valid?: true>] Assertion with == failed code: assert job.attempt == 1 left: 0 right: 1
18. ✂ Make Oban Work…
19. . Finished in 1.4 seconds 1 property, 0 tests, 0 failures
20. Additional Absolutes •Record errors on failure •Schedule failed jobs for retry •Discard jobs without more attempts
21. case job.state do "completed" -> assert job.completed_at "discarded" -> refute job.completed_at assert job.discarded_at "retryable" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert length(job.errors) > 0 assert [%{"attempt" => 1, "at" => _, "error" => _} | _] = job.errors "scheduled" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert job.max_attempts > 1 end
22. case job.state do "completed" -> assert job.completed_at "discarded" -> refute job.completed_at assert job.discarded_at "retryable" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert length(job.errors) > 0 assert [%{"attempt" => 1, "at" => _, "error" => _} | _] = job.errors "scheduled" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert job.max_attempts > 1 end
23. defp job do gen all queue <- member_of(~w(alpha beta gamma delta)), action <- member_of(~w(OK DISCARD ERROR EXIT FAIL SNOOZE)), max_attempts <- integer(1..20), ref <- integer() do args = %{ref: ref, action: action} opts = [queue: queue, max_attempts: max_attempts, worker: Worker] Job.new(args, opts) end end
24. 1 generator
 1 semi-stateful property test ~2000 lines of library code
25. EX2 Unique Jobs
26. The Blog Recipe •Partial Index •Insert Helper •Manual Implementation
27. What is absolute?
28. What is absolute? Duplicate jobs are never inserted, regardless of concurrency.
29. property "preventing multiple inserts” do check all args <- arg_map(), runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
30. property "preventing multiple inserts” do check all args <- arg_map(), runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
31. def arg_map, do: map_of(arg_key(), arg_val()) def arg_key, do: one_of([integer(), string(:ascii)]) def arg_val, do: one_of([integer(), float(), string(:ascii), list_of(integer())])
32. property "preventing multiple inserts” do check all args <- arg_map(), runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
33. defmodule UniqueWorker do use Oban.Worker, unique: [period: 30] @impl Worker def perform(_job), do: :ok end defp unique_insert!(args, opts \\ []) do args |> UniqueWorker.new(opts) |> Oban.insert!() end
34. property "preventing multiple inserts” do check all args <- arg_map(), runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
35. 1) property preventing multiple inserts (Oban.UniquenessTest) Failed with generated values (after 0 successful runs): * Clause: args <- arg_map() Generated: %{"" => 0} * Clause: runs <- integer(2..5) Generated: 2 Assertion with == failed code: assert 1 == length(ids) left: 1 right: 4
36. ✂ Prevent concurrent writes…
37. . Finished in 0.7 seconds 1 property, 0 tests, 0 failures
38. 3 generators
 1 property test 2+ concurrent inserts 0 reported† bugs † if you don’t count that prefix thing…
39. EX3 Safe Migrations
40. Progressive Migrations, v1..v8 •Tables •Columns •Functions •Triggers •Indexes
41. What is absolute?
42. What is absolute? Migrating up or down between any two versions is successful.
43. @base_migration 100_000
 @current_version 8 
 property "migrating up and down between arbitrary versions" do check all up <- integer(1..@current_version), down <- integer(1..(@current_version - 1)) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
44. @base_migration 100_000
 @current_version 8 
 property "migrating up and down between arbitrary versions" do check all up <- integer(1..@current_version), down <- integer(1..(@current_version - 1)) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
45. defmodule StepMigration do use Ecto.Migration def up do up_version = Application.get_env(:oban, :up_version) Oban.Migrations.up(version: up_version) end def down do down_version = Application.get_env(:oban, :up_version) 
 Oban.Migrations.down(version: down_version) end end
46. 1) property migrating up and down between arbitrary versions (Oban.MigratingTest) Failed with generated values (after 1 successful run): * Clause: up <- integer(2..current_version()) Generated: 8 * Clause: down <- integer(1..current_version() - 1) Generated: 1 got exception: ** (Postgrex.Error) ERROR 42P07 (duplicate_table) relation already exists
47. ✂ Ensure up/down safety…
48. . Finished in 5.7 seconds 1 property, 0 tests, 0 failures
49. @base_migration 100_000
 @current_version 8 
 property "migrating up and down between arbitrary versions" do check all up <- integer(1..@current_version), down <- integer(1..(@current_version - 1)) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
50. iex> length(for x <- 1..8, y <- 1..7, do: {x, y}) 56 ⏲ How many permutations are there?
51. @base_migration 100_000
 @current_version 8 
 test "migrating up and down between arbitrary versions" do for up <- 1..@current_version, down <- 1..(@current_version - 1) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
52. . Finished in 1.7 seconds 1 test, 0 failures, 0 excluded
53. 1 test for up 1 test for up+down
 120 permutations
54. EX4 Cron Parsing
55. Barrier to Adoption •We depend on periodic jobs… •Periodic jobs depends on unique jobs… •Unique jobs shipped…
56. * 0-5,*/3 7-14 */2 MON,TUE “At every minute past every hour from 0 through 5 and every 3rd hour on every day-of-month from 7 through 14 and on Monday and Tuesday in every 2nd month”
57. property "literal values and aliases are parsed" do check all minutes <- expression(), hours <- integer(0..23), days <- integer(1..31), months <- months(), weekdays <- weekdays(), spaces <- spaces() do spacing = :erlang.iolist_to_binary(spaces) [minutes, hours, days, months, weekdays] |> Enum.join(spacing) |> Cron.parse!() end end
58. property “valid cron expressions are parsed" do check all minutes <- expression(), hours <- integer(0..23), days <- integer(1..31), months <- months(), weekdays <- weekdays(), spaces <- spaces() do spacing = :erlang.iolist_to_binary(spaces) [minutes, hours, days, months, weekdays] |> Enum.join(spacing) |> Cron.parse!() end end
59. defp months do one_of([integer(1..12), constant("JAN"), ...]) end defp weekdays do one_of([integer(0..6), constant("MON"), ...]) end defp spaces do list_of(one_of([constant(" "), constant("\t")]), min_length: 1) end
60. defp expression do one_of([ constant("*"), map(integer(1..59), &"*/#{&1}"), map(integer(1..58), &"#{&1}-#{&1 + 1}"), map(integer(1..57), &"#{&1}-#{&1 + 2}/2"), list_of(integer(0..59), length: 1..10) ]) end
61. property "literal values and aliases are parsed" do check all minutes <- integer(0..59), hours <- integer(0..23), days <- integer(1..31), months <- months(), weekdays <- weekdays(), spaces <- spaces() do spacing = :erlang.iolist_to_binary(spaces) [minutes, hours, days, months, weekdays] |> Enum.join(spacing) |> Cron.parse!() end end
62. 1) property valid expressions and aliases are parsed (Oban.CronTest) ** (ExUnitProperties.Error) failed with generated values (after 1 successful run): * Clause: minutes <- integer(0..59) Generated: 44 * Clause: hours <- integer(0..23) Generated: 14 * Clause: days <- integer(1..31) Generated: 5 * Clause: months <- months() Generated: "JUL" * Clause: weekdays <- weekdays() Generated: 0 * Clause: spaces <- spaces() Generated: [" ", " "] got exception: ** (ArgumentError) expected string "*" or ASCII character in the range '0' to '9'...
63. ✂ Write and Refine Cron Parser…
64. . Finished in 0.1 seconds 1 property, 0 tests, 0 failures
65. test "parsing expressions that are out of bounds fails" do invalid_expressions = [ "60 * * * *", "* 24 * * *", "* * 32 * *", "* * * 13 *", "* * * * 7", "*/0 * * * *", "ONE * * * *", "* * * jan *", "* * * * sun" ] for expression <- invalid_expressions, do: assert_unparsable(expression) end defp assert_unparsable(expression) do assert_raise ArgumentError, fn -> Cron.parse!(expression) end end
66. 1 property for valid expressions 1 test for invalid expressions
 1 reported bug
67. Look for absolute truths. Once you understand them the properties and generators fall out naturally.
68. Write fewer, more expressive tests. Not just for your libraries, but your applications too.
69. Oban Web+Pro UI / Plugins / Workers getoban.pro