Building a transactional pipeline with TDD and Elixir

Building a transactional pipeline with TDD and Elixir

We are used to work with database level transactions. But, what happens when the transactional limits go beyond the database? How do we handle a complex transaction that affects different systems? How would you solve this problem using a functional language?

In this talk we will describe a real problem that we faced at DNSimple. We will discuss the challenges that we faced and how we used TDD and Elixir to solve it.

607891bd2cfbfa3a75ada8e110992d47?s=128

Javier Acero

June 03, 2017
Tweet

Transcript

  1. 9.
  2. 10.
  3. 12.

    api

  4. 13.

    api

  5. 15.
  6. 17.
  7. 18.

    if elsif elsif elsif elsif elsif elsif elsif elsif elsif

    elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif else end
  8. 21.
  9. 23.

    Runs anonymous functions without params defmodule TpTest do use ExUnit.Case

    test "runs anonymous functions without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end end 1 test/tp_test.ex Test [1/1]
  10. 24.

    test "runs anonymous functions without params" do assert Tp.run(fn ->

    2 + 2 end) == 4 end Runs anonymous functions without params 1 test/tp_test.ex Test [1/1]
  11. 25.

    test "runs anonymous functions without params" do assert Tp.run(fn ->

    2 + 2 end) == 4 end Runs anonymous functions without params 1 test/tp_test.ex Test [1/1]
  12. 29.

    Runs anonymous functions without params test "runs an anonymous function

    without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end 1 test/tp_test.ex Test [1/1]
  13. 30.

    Runs anonymous functions without params test "runs an anonymous function

    without params" do assert Tp.run(fn -> 2 + 2 end) == 4 assert Tp.run(fn -> String.upcase("x") end) == "X" end 1 test/tp_test.ex Test [1/1]
  14. 34.

    test "runs anonymous functions with 1 param" do assert Tp.run(fn(x)

    -> 2 + x end, 2) == 4 end Runs anonymous functions with 1 parameter 2 test/tp_test.ex Test [2/2]
  15. 36.

    Runs anonymous functions with 1 parameter def run(fun) do fun.()

    end def run(fun, param), do: fun.(param) 2 lib/tp.ex
  16. 37.

    Runs anonymous functions with 1 parameter def run(fun), do: fun.()

    def run(fun, param), do: fun.(param) 2 lib/tp.ex
  17. 39.

    Runs anonymous functions with 2 parameters test "runs an anonymous

    function with 2 params" do assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3]
  18. 40.

    Runs anonymous functions with 2 parameters def run(fun), do: fun.()

    def run(fun, param), do: fun.(param) 3 lib/tp.ex
  19. 41.

    Runs anonymous functions with 2 parameters def run(fun), do: fun.()

    def run(fun, params) when is_list(params), do: apply(fun, params) def run(fun, param), do: fun.(param) 3 lib/tp.ex
  20. 42.

    Runs anonymous functions with 2 parameters test "runs an anonymous

    function with 2 params" do assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3]
  21. 43.

    test "runs an anonymous function with 2 params" do assert

    Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3] Runs anonymous functions with 2 parameters
  22. 44.

    test "runs a function" do assert Tp.run(fn -> 2 +

    2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3] Runs a function Test [1/1]
  23. 45.

    Runs a function def run(fun), do: fun.() def run(fun, params)

    when is_list(params), do: apply(fun, params) def run(fun, param), do: fun.(param) 3 lib/tp.ex
  24. 48.

    Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
  25. 50.

    Runs multiple functions def run([fun1, fun2], params) do apply(fun2, apply(fun1,

    params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  26. 51.

    Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
  27. 52.

    Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
  28. 53.

    Runs multiple functions def run([fun1, fun2], params) do apply(fun2, apply(fun1,

    params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  29. 54.

    Runs multiple functions def run([first|rest], params) do apply(fun2, apply(fun1, params))

    end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  30. 55.

    Runs multiple functions def run([first|rest], params) do run(rest, apply(first, params))

    end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  31. 56.

    Runs multiple functions def run([], params), do: params def run([first|rest],

    params) do run(rest, apply(first, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  32. 57.

    Runs multiple functions 4 lib/tp.ex def run([], params), do: params

    def run([h|t], params), do: run(t, apply(h, params)) def run(fun, params), do: apply(fun, params)
  33. 58.

    Runs multiple functions 4 test/tp_test.ex Test [1/1] test "runs a

    function" do assert Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end
  34. 59.

    Runs multiple functions test "runs a function" do assert Tp.run([fn

    -> 2 + 2 end], []) == 4 assert Tp.run([&String.upcase/1], ["x"]) == "X" assert Tp.run([fn(x, y) -> x+y end], [1, 2]) == 3 end 4 test/tp_test.ex Test [1/1]
  35. 60.

    Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2] Test [1/1]
  36. 61.

    Runs multiple functions 4 lib/tp.ex def run([], params), do: params

    def run([h|t], params), do: run(t, apply(h, params)) def run(fun, params), do: apply(fun, params)
  37. 62.

    Runs multiple functions 4 lib/tp.ex def run([], params), do: params

    def run([h|t], params), do: run(t, apply(h, params))
  38. 63.
  39. 64.
  40. 65.
  41. 67.

    Runs “real world” functions defmodule State do def start(initial \\

    nil), do: #... def stop, do: #... def get, do: #... def put(new), do: #... end 5 test/test_helper.ex
  42. 68.

    Runs “real world” functions 5 test/tp_test.ex test "runs functions with

    side effects" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end assert Tp.run([f1, f2], [3]) == [27] assert State.get == 3 State.stop end Test [2/2]
  43. 73.

    try do result = this_can_fail!(params) continue(result) rescue error -> handle_error(error)

    end case this_can_fail(params) do {:ok, result} -> continue(result) {:error, error} -> handle_error(params) end vs
  44. 75.

    Returns whether it failed 6 test "returns whether it failed"

    do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, “KO"} end assert Tp.run([f1, f2], [3]) == [27] assert Tp.run([f3, f2], [3]) == {:error, “KO"} State.stop end test/tp_test.ex Test [3/3]
  45. 76.

    Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params), do: run(t, apply(h, params))
  46. 77.

    Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params) do run(t, apply(h, params)) end
  47. 78.

    Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params) do case apply(h, params) do run(t,) end end
  48. 79.

    Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params) do case apply(h, params) do {:error, details} -> {:error, details} run(t,) end end
  49. 80.

    Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} new_params -> run(t, new_params) end end
  50. 82.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3]

    test "returns whether it failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == [27] assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
  51. 83.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3]

    test "returns whether it succeeded or failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
  52. 84.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3]

    test "returns whether it succeeded or failed" do State.start f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(x) -> {:ok, [x*x*x]} end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
  53. 85.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [2/3]

    test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end assert Tp.run([f1, f2], [3]) == [27] assert State.get == 3 State.stop end
  54. 86.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [2/3]

    test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(x) -> {:ok, [x*x*x]} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert State.get == 3 State.stop end
  55. 87.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [1/3]

    test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end
  56. 88.

    Returns whether it succeeded or failed 7 test/tp_test.ex Test [1/3]

    test "runs multiple functions" do f1 = fn(x, y, z) -> {:ok, [x*x, y*y, z*z]} end f2 = fn(x, y, z) -> {:ok, [z, y, x]} end assert Tp.run([], [1, 2, 3]) == {:ok, [1, 2, 3]} assert Tp.run([f2], [7, 8, 9]) == {:ok, [9, 8, 7]} assert Tp.run([f1, f2], [2, 3, 5]) == {:ok, [25, 9, 4]} end
  57. 89.

    Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} new_params -> run(t, new_params) end end
  58. 90.

    Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} {:ok, new_params} -> run(t, new_params) end end
  59. 91.

    Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} {:ok, new_params} -> run(t, new_params) end end
  60. 92.

    Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
  61. 94.

    Rolls back when it fails 8 test "rolls back when

    it fails" do State.start("initial") f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end test/tp_test.ex Test [4/4]
  62. 95.

    Rolls back when it fails 8 test "rolls back when

    it fails" do State.start("initial") f1 = fn(x) -> state = State.get State.put(x) {:ok, [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end test/tp_test.ex Test [4/4]
  63. 96.

    Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
  64. 97.

    Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
  65. 98.

    Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> rollback.() {:error, details} end end
  66. 99.

    Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:ok, new_params, new_rollback} -> run(t, new_params) {:error, details} -> rollback.() {:error, details} end end
  67. 100.

    Rolls back when it fails 8 lib/tp.ex def run([], params,

    _), do: {:ok, params} def run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
  68. 101.

    Rolls back when it fails 8 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
  69. 102.

    Rolls back when it fails 8 lib/tp.ex def run(functions, params)

    do default_rollback = fn -> nil end do_run(functions, params, default_rollback) end
  70. 104.

    Rolls back when it fails 9 test/tp_test.ex Test [4/4] test

    "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get {State.put(x), [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end
  71. 105.

    Rolls back when it fails 9 test/tp_test.ex Test [4/4] test

    "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get {State.put(x), [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f1, f2], [0]) assert State.get == "initial" State.stop end
  72. 106.

    Rolls back when it fails 8 lib/tp.ex def run(functions, params)

    do default_rollback = fn -> nil end do_run(functions, params, default_rollback) end
  73. 107.

    Rolls back when it fails 9 lib/tp.ex def run(functions, params)

    do do_run(functions, params, _rollback = []) end
  74. 108.

    Rolls back when it fails 9 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
  75. 109.

    Rolls back when it fails 9 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, [new_rollback | rollback]) {:error, details} -> rollback.() {:error, details} end end
  76. 110.

    Rolls back when it fails 9 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, [new_rollback | rollback]) {:error, details} -> Enum.each(rollback, fn(f) -> f.() end) {:error, details} end end
  77. 111.
  78. 114.
  79. 115.
  80. 116.
  81. 117.