Building a transactional pipeline with TDD and Elixir

We are used to work with database level transactions. But, what happens when the transactional limits go beyond the database? How do we handle a complex transaction that affects different systems? How would you solve this problem using a functional language?

In this talk we will describe a real problem that we faced at DNSimple. We will discuss the challenges that we faced and how we used TDD and Elixir to solve it.

Transcript

1. Transactional Pipeline function() Building a with TDD and Elixir #scpna17 #teamtinyp
2. jacegu jacegu javieracero.com
3. https://dnsimple.com
4. d s n http://howdns.works
5. pamplonaswcraft.com 192.30.252.153 192.30.252.154
6. pamplonaswcraft.com
7. https://help.github.com/articles/using-a-custom-domain-with-github-pages
8. https://help.github.com/articles/using-a-custom-domain-with-github-pages
11. domains services Connect to
12. api
13. api
14. connectors http://platform.dnsimple.com
16. transactional
18. if elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif elsif else end
19. connection |> add_alias_record(…) |> add_cname_record(…) |> configure_cname_file(…) |> save_connection(…)
20. WHERE ARE MY OBJECTS WHERE ARE MY OBJECTS
22. Runs anonymous functions without params 1
23. Runs anonymous functions without params defmodule TpTest do use ExUnit.Case test "runs anonymous functions without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end end 1 test/tp_test.ex Test [1/1]
24. test "runs anonymous functions without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end Runs anonymous functions without params 1 test/tp_test.ex Test [1/1]
25. test "runs anonymous functions without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end Runs anonymous functions without params 1 test/tp_test.ex Test [1/1]
26. Runs anonymous functions without params defmodule Tp do def run(_) do nil end end 1 lib/tp.ex
27. Runs anonymous functions without params def run(_) do nil end 1 lib/tp.ex
28. Runs anonymous functions without params def run(_) do 4 end 1 lib/tp.ex
29. Runs anonymous functions without params test "runs an anonymous function without params" do assert Tp.run(fn -> 2 + 2 end) == 4 end 1 test/tp_test.ex Test [1/1]
30. Runs anonymous functions without params test "runs an anonymous function without params" do assert Tp.run(fn -> 2 + 2 end) == 4 assert Tp.run(fn -> String.upcase("x") end) == "X" end 1 test/tp_test.ex Test [1/1]
31. Runs anonymous functions without params def run(_) do 4 end 1 lib/tp.ex
32. Runs anonymous functions without params def run(fun) do fun.() end 1 lib/tp.ex
33. Runs anonymous functions with 1 parameter 2
34. test "runs anonymous functions with 1 param" do assert Tp.run(fn(x) -> 2 + x end, 2) == 4 end Runs anonymous functions with 1 parameter 2 test/tp_test.ex Test [2/2]
35. Runs anonymous functions with 1 parameter def run(fun) do fun.() end 2 lib/tp.ex
36. Runs anonymous functions with 1 parameter def run(fun) do fun.() end def run(fun, param), do: fun.(param) 2 lib/tp.ex
37. Runs anonymous functions with 1 parameter def run(fun), do: fun.() def run(fun, param), do: fun.(param) 2 lib/tp.ex
38. Runs anonymous functions with 2 parameters 3
39. Runs anonymous functions with 2 parameters test "runs an anonymous function with 2 params" do assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3]
40. Runs anonymous functions with 2 parameters def run(fun), do: fun.() def run(fun, param), do: fun.(param) 3 lib/tp.ex
41. Runs anonymous functions with 2 parameters def run(fun), do: fun.() def run(fun, params) when is_list(params), do: apply(fun, params) def run(fun, param), do: fun.(param) 3 lib/tp.ex
42. Runs anonymous functions with 2 parameters test "runs an anonymous function with 2 params" do assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3]
43. test "runs an anonymous function with 2 params" do assert Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3] Runs anonymous functions with 2 parameters
44. test "runs a function" do assert Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end 3 test/tp_test.ex Test [3/3] Runs a function Test [1/1]
45. Runs a function def run(fun), do: fun.() def run(fun, params) when is_list(params), do: apply(fun, params) def run(fun, param), do: fun.(param) 3 lib/tp.ex
46. Runs a function def run(fun, params), do: apply(fun, params) 3 lib/tp.ex
47. Runs multiple functions 4
48. Runs multiple functions test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
49. Runs multiple functions def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
50. Runs multiple functions def run([fun1, fun2], params) do apply(fun2, apply(fun1, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
51. Runs multiple functions test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
52. Runs multiple functions test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2]
53. Runs multiple functions def run([fun1, fun2], params) do apply(fun2, apply(fun1, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
54. Runs multiple functions def run([first|rest], params) do apply(fun2, apply(fun1, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
55. Runs multiple functions def run([first|rest], params) do run(rest, apply(first, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
56. Runs multiple functions def run([], params), do: params def run([first|rest], params) do run(rest, apply(first, params)) end def run(fun, params), do: apply(fun, params) 4 lib/tp.ex
57. Runs multiple functions 4 lib/tp.ex def run([], params), do: params def run([h|t], params), do: run(t, apply(h, params)) def run(fun, params), do: apply(fun, params)
58. Runs multiple functions 4 test/tp_test.ex Test [1/1] test "runs a function" do assert Tp.run(fn -> 2 + 2 end, []) == 4 assert Tp.run(&String.upcase/1, ["x"]) == "X" assert Tp.run(fn(x, y) -> x+y end, [1, 2]) == 3 end
59. Runs multiple functions test "runs a function" do assert Tp.run([fn -> 2 + 2 end], []) == 4 assert Tp.run([&String.upcase/1], ["x"]) == "X" assert Tp.run([fn(x, y) -> x+y end], [1, 2]) == 3 end 4 test/tp_test.ex Test [1/1]
60. Runs multiple functions test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end 4 test/tp_test.ex Test [2/2] Test [1/1]
61. Runs multiple functions 4 lib/tp.ex def run([], params), do: params def run([h|t], params), do: run(t, apply(h, params)) def run(fun, params), do: apply(fun, params)
62. Runs multiple functions 4 lib/tp.ex def run([], params), do: params def run([h|t], params), do: run(t, apply(h, params))
66. Runs functions with side effects Runs “real world” functions 5
67. Runs “real world” functions defmodule State do def start(initial \\ nil), do: #... def stop, do: #... def get, do: #... def put(new), do: #... end 5 test/test_helper.ex
68. Runs “real world” functions 5 test/tp_test.ex test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end assert Tp.run([f1, f2], [3]) == [27] assert State.get == 3 State.stop end Test [2/2]
69. Returns whether it failed 6
70. ERR R HANDLING
71. http://elixir-lang.org/getting-started/try-catch-and-rescue.html
72. http://elixir-lang.org/getting-started/try-catch-and-rescue.html
73. try do result = this_can_fail!(params) continue(result) rescue error -> handle_error(error) end case this_can_fail(params) do {:ok, result} -> continue(result) {:error, error} -> handle_error(params) end vs
74. Returns whether it failed 6
75. Returns whether it failed 6 test "returns whether it failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, “KO"} end assert Tp.run([f1, f2], [3]) == [27] assert Tp.run([f3, f2], [3]) == {:error, “KO"} State.stop end test/tp_test.ex Test [3/3]
76. Returns whether it failed 6 lib/tp.ex def run([], params), do: params def run([h|t], params), do: run(t, apply(h, params))
77. Returns whether it failed 6 lib/tp.ex def run([], params), do: params def run([h|t], params) do run(t, apply(h, params)) end
78. Returns whether it failed 6 lib/tp.ex def run([], params), do: params def run([h|t], params) do case apply(h, params) do run(t,) end end
79. Returns whether it failed 6 lib/tp.ex def run([], params), do: params def run([h|t], params) do case apply(h, params) do {:error, details} -> {:error, details} run(t,) end end
80. Returns whether it failed 6 lib/tp.ex def run([], params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} new_params -> run(t, new_params) end end
81. Returns whether it succeeded or failed 7
82. Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3] test "returns whether it failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == [27] assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
83. Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3] test "returns whether it succeeded or failed" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
84. Returns whether it succeeded or failed 7 test/tp_test.ex Test [3/3] test "returns whether it succeeded or failed" do State.start f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(x) -> {:ok, [x*x*x]} end f3 = fn(_) -> {:error, "KO"} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert Tp.run([f3, f2], [3]) == {:error, "KO"} State.stop end
85. Returns whether it succeeded or failed 7 test/tp_test.ex Test [2/3] test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); [x] end f2 = fn(x) -> [x*x*x] end assert Tp.run([f1, f2], [3]) == [27] assert State.get == 3 State.stop end
86. Returns whether it succeeded or failed 7 test/tp_test.ex Test [2/3] test "runs functions with side effects" do State.start f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(x) -> {:ok, [x*x*x]} end assert Tp.run([f1, f2], [3]) == {:ok, [27]} assert State.get == 3 State.stop end
87. Returns whether it succeeded or failed 7 test/tp_test.ex Test [1/3] test "runs multiple functions" do f1 = fn(x, y, z) -> [x*x, y*y, z*z] end f2 = fn(x, y, z) -> [z, y, x] end assert Tp.run([], [1, 2, 3]) == [1, 2, 3] assert Tp.run([f2], [7, 8, 9]) == [9, 8, 7] assert Tp.run([f1, f2], [2, 3, 5]) == [25, 9, 4] end
88. Returns whether it succeeded or failed 7 test/tp_test.ex Test [1/3] test "runs multiple functions" do f1 = fn(x, y, z) -> {:ok, [x*x, y*y, z*z]} end f2 = fn(x, y, z) -> {:ok, [z, y, x]} end assert Tp.run([], [1, 2, 3]) == {:ok, [1, 2, 3]} assert Tp.run([f2], [7, 8, 9]) == {:ok, [9, 8, 7]} assert Tp.run([f1, f2], [2, 3, 5]) == {:ok, [25, 9, 4]} end
89. Returns whether it succeeded or failed 7 lib/tp.ex def run([], params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} new_params -> run(t, new_params) end end
90. Returns whether it succeeded or failed 7 lib/tp.ex def run([], params), do: params def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} {:ok, new_params} -> run(t, new_params) end end
91. Returns whether it succeeded or failed 7 lib/tp.ex def run([], params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:error, details} -> {:error, details} {:ok, new_params} -> run(t, new_params) end end
92. Returns whether it succeeded or failed 7 lib/tp.ex def run([], params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
93. Rolls back when it fails 8
94. Rolls back when it fails 8 test "rolls back when it fails" do State.start("initial") f1 = fn(x) -> State.put(x); {:ok, [x]} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end test/tp_test.ex Test [4/4]
95. Rolls back when it fails 8 test "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get State.put(x) {:ok, [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end test/tp_test.ex Test [4/4]
96. Rolls back when it fails 8 lib/tp.ex def run([], params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
97. Rolls back when it fails 8 lib/tp.ex def run([], params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> {:error, details} end end
98. Rolls back when it fails 8 lib/tp.ex def run([], params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:error, details} -> rollback.() {:error, details} end end
99. Rolls back when it fails 8 lib/tp.ex def run([], params), do: {:ok, params} def run([h | t], params) do case apply(h, params) do {:ok, new_params} -> run(t, new_params) {:ok, new_params, new_rollback} -> run(t, new_params) {:error, details} -> rollback.() {:error, details} end end
100. Rolls back when it fails 8 lib/tp.ex def run([], params, _), do: {:ok, params} def run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
101. Rolls back when it fails 8 lib/tp.ex defp do_run([], params, _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
102. Rolls back when it fails 8 lib/tp.ex def run(functions, params) do default_rollback = fn -> nil end do_run(functions, params, default_rollback) end
103. Rolls back when it fails 9
104. Rolls back when it fails 9 test/tp_test.ex Test [4/4] test "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get {State.put(x), [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f2], [0]) assert State.get == "initial" State.stop end
105. Rolls back when it fails 9 test/tp_test.ex Test [4/4] test "rolls back when it fails" do State.start("initial") f1 = fn(x) -> state = State.get {State.put(x), [x], fn() -> State.put(state) end} end f2 = fn(_) -> {:error, "BOOM!"} end Tp.run([f1, f1, f2], [0]) assert State.get == "initial" State.stop end
106. Rolls back when it fails 8 lib/tp.ex def run(functions, params) do default_rollback = fn -> nil end do_run(functions, params, default_rollback) end
107. Rolls back when it fails 9 lib/tp.ex def run(functions, params) do do_run(functions, params, _rollback = []) end
108. Rolls back when it fails 9 lib/tp.ex defp do_run([], params, _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, new_rollback) {:error, details} -> rollback.() {:error, details} end end
109. Rolls back when it fails 9 lib/tp.ex defp do_run([], params, _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, [new_rollback | rollback]) {:error, details} -> rollback.() {:error, details} end end
110. Rolls back when it fails 9 lib/tp.ex defp do_run([], params, _), do: {:ok, params} defp do_run([h | t], params, rollback) do case apply(h, params) do {:ok, new_params} -> run(t, new_params, rollback) {:ok, new_params, new_rollback} -> run(t, new_params, [new_rollback | rollback]) {:error, details} -> Enum.each(rollback, fn(f) -> f.() end) {:error, details} end end
111. it works!!
112. Tp.run([ &add_alias_record/2, &add_cname_record/2, &configure_cname_file/2, &save_connection/2, ], [connection, …])
113. what’s point your ?
114. practice
115. be smart
116. thank you
117. questions