Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hardening Applications with Property Tests

Hardening Applications with Property Tests

Property testing is a tool that every Elixir developer should have in their toolbox. Come learn how property tests can thoroughly test a problem space and make bulletproof software. We'll look at properties pulled from the Oban background job processor.

Some examples we'll explore:

* Proving a CRON parser
* Checking concurrent uniqueness
* Transitioning between migrations
* Proving system reliability

You'll leave armed with the ability to recognize situations where properties are more effective than example based tests.

C4d700e708f2a338a737d0fb80297731?s=128

Parker Selbert

June 19, 2020
Tweet

Transcript

  1. Hardening Applications with Property Tests

  2. Hardening a Library with Property Tests

  3. Parker Selbert @sorentwo Soren / Dscout

  4. Adopting Elixir and a library named Kiq

  5. Nobody likes finding bugs in production.

  6. Kiq Oban Initially, an experiment to model queues as streams.

  7. Starting over meant building up new example tests, with many

    possible regressions.
  8. Property-Based Testing • PropEr • StreamData
 • Proper Book

  9. Properties Aspects of code that hold for a set of

    randomized test data Functions that dynamically generate infinite streams of data Generators
  10. EX1 Reliable Execution

  11. What is absolute?

  12. What is absolute? All inserted jobs are attempted at least

    once.
  13. property "inserted jobs are executed at least once" do check

    all jobs <- list_of(job()) do for %{id: id, args: %{"ref" => ref}} <- Oban.insert_all(jobs) do assert_receive {:ok, ^ref} assert %{attempt: 1, attempted_at: %DateTime{}} = Repo.get(Job, job.id) end end end
  14. property "inserted jobs are executed at least once" do check

    all jobs <- list_of(job()) do for %{id: id, args: %{"ref" => ref}} <- Oban.insert_all(jobs) do assert_receive {:ok, ^ref} assert %{attempt: 1, attempted_at: %DateTime{}} = Repo.get(Job, job.id) end end end
  15. defp job do gen all queue <- member_of(~w(alpha beta gamma

    delta)), ref <- integer() do Job.new(%{ref: ref}, queue: queue, worker: Worker) end end
  16. property "inserted jobs are executed at least once" do check

    all jobs <- list_of(job()) do for %{id: id, args: %{"ref" => ref}} <- Oban.insert_all(jobs) do assert_receive {:ok, ^ref} assert %{attempt: 1, attempted_at: %DateTime{}} = Repo.get(Job, job.id) end end end
  17. 1) property inserted jobs are executed at least once (Oban.ExecutingTest)

    Failed with generated values (after 13 successful runs): * Clause: jobs <- list_of(job()) Generated: [#Ecto.Changeset<valid?: true>] Assertion with == failed code: assert job.attempt == 1 left: 0 right: 1
  18. ✂ Make Oban Work…

  19. . Finished in 1.4 seconds 1 property, 0 tests, 0

    failures
  20. Additional Absolutes •Record errors on failure •Schedule failed jobs for

    retry •Discard jobs without more attempts
  21. case job.state do "completed" -> assert job.completed_at "discarded" -> refute

    job.completed_at assert job.discarded_at "retryable" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert length(job.errors) > 0 assert [%{"attempt" => 1, "at" => _, "error" => _} | _] = job.errors "scheduled" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert job.max_attempts > 1 end
  22. case job.state do "completed" -> assert job.completed_at "discarded" -> refute

    job.completed_at assert job.discarded_at "retryable" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert length(job.errors) > 0 assert [%{"attempt" => 1, "at" => _, "error" => _} | _] = job.errors "scheduled" -> refute job.completed_at assert DateTime.compare(job.scheduled_at, DateTime.utc_now()) == :gt assert job.max_attempts > 1 end
  23. defp job do gen all queue <- member_of(~w(alpha beta gamma

    delta)), action <- member_of(~w(OK DISCARD ERROR EXIT FAIL SNOOZE)), max_attempts <- integer(1..20), ref <- integer() do args = %{ref: ref, action: action} opts = [queue: queue, max_attempts: max_attempts, worker: Worker] Job.new(args, opts) end end
  24. 1 generator
 1 semi-stateful property test ~2000 lines of library

    code
  25. EX2 Unique Jobs

  26. The Blog Recipe •Partial Index •Insert Helper •Manual Implementation

  27. What is absolute?

  28. What is absolute? Duplicate jobs are never inserted, regardless of

    concurrency.
  29. property "preventing multiple inserts” do check all args <- arg_map(),

    runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
  30. property "preventing multiple inserts” do check all args <- arg_map(),

    runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
  31. def arg_map, do: map_of(arg_key(), arg_val()) def arg_key, do: one_of([integer(), string(:ascii)])

    def arg_val, do: one_of([integer(), float(), string(:ascii), list_of(integer())])
  32. property "preventing multiple inserts” do check all args <- arg_map(),

    runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
  33. defmodule UniqueWorker do use Oban.Worker, unique: [period: 30] @impl Worker

    def perform(_job), do: :ok end defp unique_insert!(args, opts \\ []) do args |> UniqueWorker.new(opts) |> Oban.insert!() end
  34. property "preventing multiple inserts” do check all args <- arg_map(),

    runs <- integer(2..5) do fun = fn -> unique_insert!(args) end ids = 1..runs |> Enum.map(fn _ -> Task.async(fun) end) |> Enum.map(&Task.await/1) |> Enum.map(fn %Job{id: id} -> id end) |> Enum.reject(&is_nil/1) |> Enum.uniq() assert 1 == length(ids) end end
  35. 1) property preventing multiple inserts (Oban.UniquenessTest) Failed with generated values

    (after 0 successful runs): * Clause: args <- arg_map() Generated: %{"" => 0} * Clause: runs <- integer(2..5) Generated: 2 Assertion with == failed code: assert 1 == length(ids) left: 1 right: 4
  36. ✂ Prevent concurrent writes…

  37. . Finished in 0.7 seconds 1 property, 0 tests, 0

    failures
  38. 3 generators
 1 property test 2+ concurrent inserts 0 reported†

    bugs † if you don’t count that prefix thing…
  39. EX3 Safe Migrations

  40. Progressive Migrations, v1..v8 •Tables •Columns •Functions •Triggers •Indexes

  41. What is absolute?

  42. What is absolute? Migrating up or down between any two

    versions is successful.
  43. @base_migration 100_000
 @current_version 8 
 property "migrating up and down

    between arbitrary versions" do check all up <- integer(1..@current_version), down <- integer(1..(@current_version - 1)) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
  44. @base_migration 100_000
 @current_version 8 
 property "migrating up and down

    between arbitrary versions" do check all up <- integer(1..@current_version), down <- integer(1..(@current_version - 1)) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
  45. defmodule StepMigration do use Ecto.Migration def up do up_version =

    Application.get_env(:oban, :up_version) Oban.Migrations.up(version: up_version) end def down do down_version = Application.get_env(:oban, :up_version) 
 Oban.Migrations.down(version: down_version) end end
  46. 1) property migrating up and down between arbitrary versions (Oban.MigratingTest)

    Failed with generated values (after 1 successful run): * Clause: up <- integer(2..current_version()) Generated: 8 * Clause: down <- integer(1..current_version() - 1) Generated: 1 got exception: ** (Postgrex.Error) ERROR 42P07 (duplicate_table) relation already exists
  47. ✂ Ensure up/down safety…

  48. . Finished in 5.7 seconds 1 property, 0 tests, 0

    failures
  49. @base_migration 100_000
 @current_version 8 
 property "migrating up and down

    between arbitrary versions" do check all up <- integer(1..@current_version), down <- integer(1..(@current_version - 1)) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
  50. iex> length(for x <- 1..8, y <- 1..7, do: {x,

    y}) 56 ⏲ How many permutations are there?
  51. @base_migration 100_000
 @current_version 8 
 test "migrating up and down

    between arbitrary versions" do for up <- 1..@current_version, down <- 1..(@current_version - 1) do Application.put_env(:oban, :up_version, up) Application.put_env(:oban, :down_version, down) assert :ok = Ecto.Migrator.up(Repo, @base_migration, StepMigration) assert :ok = Ecto.Migrator.down(Repo, @base_migration, StepMigration) clear_migrated() end end
  52. . Finished in 1.7 seconds 1 test, 0 failures, 0

    excluded
  53. 1 test for up 1 test for up+down
 120 permutations

  54. EX4 Cron Parsing

  55. Barrier to Adoption •We depend on periodic jobs… •Periodic jobs

    depends on unique jobs… •Unique jobs shipped…
  56. * 0-5,*/3 7-14 */2 MON,TUE “At every minute past every

    hour from 0 through 5 and every 3rd hour on every day-of-month from 7 through 14 and on Monday and Tuesday in every 2nd month”
  57. property "literal values and aliases are parsed" do check all

    minutes <- expression(), hours <- integer(0..23), days <- integer(1..31), months <- months(), weekdays <- weekdays(), spaces <- spaces() do spacing = :erlang.iolist_to_binary(spaces) [minutes, hours, days, months, weekdays] |> Enum.join(spacing) |> Cron.parse!() end end
  58. property “valid cron expressions are parsed" do check all minutes

    <- expression(), hours <- integer(0..23), days <- integer(1..31), months <- months(), weekdays <- weekdays(), spaces <- spaces() do spacing = :erlang.iolist_to_binary(spaces) [minutes, hours, days, months, weekdays] |> Enum.join(spacing) |> Cron.parse!() end end
  59. defp months do one_of([integer(1..12), constant("JAN"), ...]) end defp weekdays do

    one_of([integer(0..6), constant("MON"), ...]) end defp spaces do list_of(one_of([constant(" "), constant("\t")]), min_length: 1) end
  60. defp expression do one_of([ constant("*"), map(integer(1..59), &"*/#{&1}"), map(integer(1..58), &"#{&1}-#{&1 +

    1}"), map(integer(1..57), &"#{&1}-#{&1 + 2}/2"), list_of(integer(0..59), length: 1..10) ]) end
  61. property "literal values and aliases are parsed" do check all

    minutes <- integer(0..59), hours <- integer(0..23), days <- integer(1..31), months <- months(), weekdays <- weekdays(), spaces <- spaces() do spacing = :erlang.iolist_to_binary(spaces) [minutes, hours, days, months, weekdays] |> Enum.join(spacing) |> Cron.parse!() end end
  62. 1) property valid expressions and aliases are parsed (Oban.CronTest) **

    (ExUnitProperties.Error) failed with generated values (after 1 successful run): * Clause: minutes <- integer(0..59) Generated: 44 * Clause: hours <- integer(0..23) Generated: 14 * Clause: days <- integer(1..31) Generated: 5 * Clause: months <- months() Generated: "JUL" * Clause: weekdays <- weekdays() Generated: 0 * Clause: spaces <- spaces() Generated: [" ", " "] got exception: ** (ArgumentError) expected string "*" or ASCII character in the range '0' to '9'...
  63. ✂ Write and Refine Cron Parser…

  64. . Finished in 0.1 seconds 1 property, 0 tests, 0

    failures
  65. test "parsing expressions that are out of bounds fails" do

    invalid_expressions = [ "60 * * * *", "* 24 * * *", "* * 32 * *", "* * * 13 *", "* * * * 7", "*/0 * * * *", "ONE * * * *", "* * * jan *", "* * * * sun" ] for expression <- invalid_expressions, do: assert_unparsable(expression) end defp assert_unparsable(expression) do assert_raise ArgumentError, fn -> Cron.parse!(expression) end end
  66. 1 property for valid expressions 1 test for invalid expressions


    1 reported bug
  67. Look for absolute truths. Once you understand them the properties

    and generators fall out naturally.
  68. Write fewer, more expressive tests. Not just for your libraries,

    but your applications too.
  69. Oban Web+Pro UI / Plugins / Workers getoban.pro