Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a transactional pipeline with TDD and Elixir

Building a transactional pipeline with TDD and Elixir

We are used to work with database level transactions. But, what happens when the transactional limits go beyond the database? How do we handle a complex transaction that affects different systems? How would you solve this problem using a functional language?

In this talk we will describe a real problem that we faced at DNSimple. We will discuss the challenges that we faced and how we used TDD and Elixir to solve it.

Javier Acero

June 03, 2017
Tweet

More Decks by Javier Acero

Other Decks in Technology

Transcript

  1. api

  2. api

  3. if elsif elsif elsif elsif elsif elsif elsif elsif elsif

    elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif else end
  4. Runs anonymous functions without params defmodule TpTest do use ExUnit.Case

    test "runs anonymous functions without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end end 1 test/tp_test.ex Test [1/1]
  5. test "runs anonymous functions without params" do assert Tp.run(fn ->

    2 + 2 end) == 4 end Runs anonymous functions without params 1 test/tp_test.ex Test [1/1]
  6. test "runs anonymous functions without params" do assert Tp.run(fn ->

    2 + 2 end) == 4 end Runs anonymous functions without params 1 test/tp_test.ex Test [1/1]
  7. Runs anonymous functions without params test "runs an anonymous function

    without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end 1 test/tp_test.ex Test [1/1]
  8. Runs anonymous functions without params test "runs an anonymous function

    without params" do assert Tp.run(fn -> 2 + 2 end) == 4 assert Tp.run(fn -> String.upcase("x") end) == "X" end 1 test/tp_test.ex Test [1/1]
  9. test "runs anonymous functions with 1 param" do assert Tp.run(fn(x)

    -> 2 + x end, 2) == 4 end Runs anonymous functions with 1 parameter 2 test/tp_test.ex Test [2/2]
  10. Runs anonymous functions with 1 parameter def run(fun) do fun.()

    end def run(fun, param), do: fun.(param) 2 lib/tp.ex
  11. Runs anonymous functions with 1 parameter def run(fun), do: fun.()

    def run(fun, param), do: fun.(param) 2 lib/tp.ex
  12. Runs anonymous functions with 2 parameters test "runs an anonymous

    function with 2 params" do assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3]
  13. Runs anonymous functions with 2 parameters def run(fun), do: fun.()

    def run(fun, param), do: fun.(param) 3 lib/tp.ex
  14. Runs anonymous functions with 2 parameters def run(fun), do: fun.()

    def run(fun, params) when is_list(params), do: apply(fun, params) def run(fun, param), do: fun.(param) 3 lib/tp.ex
  15. Runs anonymous functions with 2 parameters test "runs an anonymous

    function with 2 params" do assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3]
  16. test "runs an anonymous function with 2 params" do assert

    Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3] Runs anonymous functions with 2 parameters
  17. test "runs a function" do assert Tp.run(fn -> 2 +

    2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3] Runs a function Test [1/1]
  18. Runs a function def run(fun), do: fun.() def run(fun, params)

    when is_list(params), do: apply(fun, params) def run(fun, param), do: fun.(param) 3 lib/tp.ex
  19. Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
  20. Runs multiple functions def run([fun1, fun2], params) do apply(fun2, apply(fun1,

    params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  21. Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
  22. Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
  23. Runs multiple functions def run([fun1, fun2], params) do apply(fun2, apply(fun1,

    params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  24. Runs multiple functions def run([first|rest], params) do apply(fun2, apply(fun1, params))

    end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  25. Runs multiple functions def run([first|rest], params) do run(rest, apply(first, params))

    end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  26. Runs multiple functions def run([], params), do: params def run([first|rest],

    params) do run(rest, apply(first, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
  27. Runs multiple functions 4 lib/tp.ex def run([], params), do: params

    def run([h|t], params), do: run(t, apply(h, params)) def run(fun, params), do: apply(fun, params)
  28. Runs multiple functions 4 test/tp_test.ex Test [1/1] test "runs a

    function" do assert Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end
  29. Runs multiple functions test "runs a function" do assert Tp.run([fn

    -> 2 + 2 end], []) == 4 assert Tp.run([&String.upcase/1], ["x"]) == "X" assert Tp.run([fn(x, y) -> x+y end], [1, 2]) == 3 end 4 test/tp_test.ex Test [1/1]
  30. Runs multiple functions test "runs multiple functions" do f1 =

    fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2] Test [1/1]
  31. Runs multiple functions 4 lib/tp.ex def run([], params), do: params

    def run([h|t], params), do: run(t, apply(h, params)) def run(fun, params), do: apply(fun, params)
  32. Runs multiple functions 4 lib/tp.ex def run([], params), do: params

    def run([h|t], params), do: run(t, apply(h, params))
  33. Runs “real world” functions defmodule State do def start(initial \\

    nil), do: #... def stop, do: #... def get, do: #... def put(new), do: #... end 5 test/test_helper.ex
  34. Runs “real world” functions 5 test/tp_test.ex test "runs functions with

    side effects" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end assert Tp.run([f1, f2], [3]) == [27] assert State.get == 3 State.stop end Test [2/2]
  35. try do result = this_can_fail!(params) continue(result) rescue error -> handle_error(error)

    end case this_can_fail(params) do {:ok, result} -> continue(result) {:error, error} -> handle_error(params) end vs
  36. Returns whether it failed 6 test "returns whether it failed"

    do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, “KO"} end assert Tp.run([f1, f2], [3]) == [27] assert Tp.run([f3, f2], [3]) == {:error, “KO"} State.stop end test/tp_test.ex Test [3/3]
  37. Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params), do: run(t, apply(h, params))
  38. Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params) do run(t, apply(h, params)) end
  39. Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params) do case apply(h, params) do run(t,) end end
  40. Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h|t], params) do case apply(h, params) do {:error, details} -> {:error, details} run(t,) end end
  41. Returns whether it failed 6 lib/tp.ex def run([], params), do:

    params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} new_params -> run(t, new_params) end end
  42. Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3]

    test "returns whether it failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == [27] assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
  43. Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3]

    test "returns whether it succeeded or failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
  44. Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3]

    test "returns whether it succeeded or failed" do State.start f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(x) -> {:ok, [x*x*x]} end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
  45. Returns whether it succeeded or failed 7 test/tp_test.ex Test [2/3]

    test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end assert Tp.run([f1, f2], [3]) == [27] assert State.get == 3 State.stop end
  46. Returns whether it succeeded or failed 7 test/tp_test.ex Test [2/3]

    test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(x) -> {:ok, [x*x*x]} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert State.get == 3 State.stop end
  47. Returns whether it succeeded or failed 7 test/tp_test.ex Test [1/3]

    test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end
  48. Returns whether it succeeded or failed 7 test/tp_test.ex Test [1/3]

    test "runs multiple functions" do f1 = fn(x, y, z) -> {:ok, [x*x, y*y, z*z]} end f2 = fn(x, y, z) -> {:ok, [z, y, x]} end assert Tp.run([], [1, 2, 3]) == {:ok, [1, 2, 3]} assert Tp.run([f2], [7, 8, 9]) == {:ok, [9, 8, 7]} assert Tp.run([f1, f2], [2, 3, 5]) == {:ok, [25, 9, 4]} end
  49. Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} new_params -> run(t, new_params) end end
  50. Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} {:ok, new_params} -> run(t, new_params) end end
  51. Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} {:ok, new_params} -> run(t, new_params) end end
  52. Returns whether it succeeded or failed 7 lib/tp.ex def run([],

    params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
  53. Rolls back when it fails 8 test "rolls back when

    it fails" do State.start("initial") f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end test/tp_test.ex Test [4/4]
  54. Rolls back when it fails 8 test "rolls back when

    it fails" do State.start("initial") f1 = fn(x) -> state = State.get State.put(x) {:ok, [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end test/tp_test.ex Test [4/4]
  55. Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
  56. Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
  57. Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> rollback.() {:error, details} end end
  58. Rolls back when it fails 8 lib/tp.ex def run([], params),

    do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:ok, new_params, new_rollback} -> run(t, new_params) {:error, details} -> rollback.() {:error, details} end end
  59. Rolls back when it fails 8 lib/tp.ex def run([], params,

    _), do: {:ok, params} def run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
  60. Rolls back when it fails 8 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
  61. Rolls back when it fails 8 lib/tp.ex def run(functions, params)

    do default_rollback = fn -> nil end do_run(functions, params, default_rollback) end
  62. Rolls back when it fails 9 test/tp_test.ex Test [4/4] test

    "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get {State.put(x), [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end
  63. Rolls back when it fails 9 test/tp_test.ex Test [4/4] test

    "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get {State.put(x), [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f1, f2], [0]) assert State.get == "initial" State.stop end
  64. Rolls back when it fails 8 lib/tp.ex def run(functions, params)

    do default_rollback = fn -> nil end do_run(functions, params, default_rollback) end
  65. Rolls back when it fails 9 lib/tp.ex def run(functions, params)

    do do_run(functions, params, _rollback = []) end
  66. Rolls back when it fails 9 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
  67. Rolls back when it fails 9 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, [new_rollback | rollback]) {:error, details} -> rollback.() {:error, details} end end
  68. Rolls back when it fails 9 lib/tp.ex defp do_run([], params,

    _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, [new_rollback | rollback]) {:error, details} -> Enum.each(rollback, fn(f) -> f.() end) {:error, details} end end