CodeEurope 2017 – Monoliths to Services with Elixir and Phoenix

This talk was presented at CodeEurope 2017 by Lauren Tan. Recording available here - https://youtu.be/g2ATsmnkjbs.

Transitioning monolithic apps into a micro-service architecture isn't straightforward – in fact, it is often quite difficult. In this intermediate talk, we'll learn how Elixir umbrella apps and Phoenix utilize the Erlang VM (BEAM) to make building services less painful and more productive.

Transcript

1. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Monoliths to Services with Elixir & Phoenix PRESENTED BY Lauren Tan sugarpirate_ poteto Cinemagraph by /u/orbojunglist
2. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix bit.ly/elixir-microservices
3. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Lauren Tan sugarpirate_ poteto
4. EmberConf 2017 Confessions of an Ember Addon Author
5. EmberConf 2017 Confessions of an Ember Addon Author
6. EmberConf 2017 Confessions of an Ember Addon Author
7. From Front-End to Full Stack with Elixir & Phoenix ElixirConf 2016
8. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
9. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
10. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
11. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://github.com/Netflix/vizceral
12. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
13. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
14. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
15. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
16. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
17. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix http://worrydream.com/refs/Brooks-NoSilverBullet.pdf
18. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
19. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Microservices will improve our performance easily FALLACY #1
20. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Network latency
21. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix What's the bottleneck?
22. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix The code is cleaner FALLACY #2
23. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
24. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix You can write clean code without microservices
25. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix It's easier to write something that does one thing well FALLACY #3
26. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Application simplicity
27. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Infrastructure complexity
28. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://martinfowler.com/articles/microservice-trade-offs.html
29. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Embrace the monolith
30. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Microservices are an organizational optimization https://martinfowler.com/bliki/MicroservicePremium.html
31. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://www.slideshare.net/reed2001/culture-1798664/
32. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Conway's Law
33. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
34. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
35. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix The right tool for the job Cinemagraph by /u/smoothinto2nd
36. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Web API/apps ELIXIR IS GREAT FOR
37. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Soft real-time ELIXIR IS GREAT FOR
38. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix ELIXIR IS GREAT FOR High scalability and fault tolerance
39. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix ELIXIR IS GREAT FOR Concurrency
40. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix ELIXIR IS GREAT FOR Developer happiness
41. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Number crunching ELIXIR IS NOT GREAT FOR
42. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Anything CPU bound ELIXIR IS NOT GREAT FOR
43. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
44. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
45. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Highly-scalable, fault-tolerant systems ERLANG
46. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Pre-emptive scheduling THE ERLANG VM: BEAM BEAM SCHEDULER SCHEDULER SCHEDULER SCHEDULER Processes Processes Processes Processes OS Process OS Thread OS Thread OS Thread OS Thread https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html
47. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix 1. Everything is a process 2. Processes are strongly isolated 3. Process creation and destruction is a lightweight operation 4. Message passing is the only way for processes to interact 5. Processes have unique names 6. If you know the name of a process you can send it a message 7. Processes share no resources 8. Error handling is non-local 9. Processes do what they are supposed to do or fail http://erlang.org/download/armstrong_thesis_2003.pdf
48. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix JAVA Write once, run anywhere
49. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix ERLANG https://www.youtube.com/watch?v=u41GEwIq2mE&t=3m59s Write once, run forever
50. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Garbage Collection
51. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix http://www.erlang-factory.com/static/upload/media/1394350183453526efsf2014whatsappscaling.pdf
52. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://www.youtube.com/watch?v=5SbWapbXhKo
53. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Elixir?
54. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix 1 + 1 #!=> 2 {:ok, "hello"} [1, 2, 3] = [head | tail] head #!=> 1 tail #!=> [2,3]
55. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Math do def add(x, y), do: x + y def subtract(x, y) do x - y end end
56. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Collection do def sum([x, y, z]), do: x + y + z end
57. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(2)> Collection.sum([1, 2, 3]) 6 iex(3)> Collection.sum([1, 2, 3, 4]) !** (FunctionClauseError) no function clause matching in Collection.sum/1 iex:2: Collection.sum([1, 2, 3, 4])
58. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Pipe Operator
59. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix foo(bar(baz(123), 456), 789)
60. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix 123 !|> baz() !|> bar(456) !|> foo(123)
61. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def serialize(user) do user !|> serialize_attributes() !|> serialize_relationships() !|> to_json() end
62. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def validate_presence(input) do if input !== "" do "Can't be blank" end input end
63. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Pattern Matching
64. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def validate_presence(""), do: {:err, "Can't be blank"} def validate_presence(input), do: input
65. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix thing = %{foo: "bar", baz: 123} %{foo: foo, baz: baz} = thing; IO.inspect(foo) # "bar" IO.inspect(baz) # 123
66. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix thing = %{foo: "bar", baz: 123} %{foo: foo, baz: baz, qux: qux} = thing; !** (MatchError) no match of right hand side value: %{baz: 123, foo: "bar"}
67. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://fsharpforfunandprofit.com/rop/
68. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def save({:ok, %Response{body: %{"data" !=> _} = body, status_code: 200}}, schema_atom), do: do_save(body, schema_atom) defp do_save(body, schema_atom) do with {:ok, parsed} !<- parse_jsonapi(body), {:ok, normalized} !<- normalize(parsed), {:ok, _} !<- insert_all(normalized, schema_atom), do: {:ok, schema_atom} end
69. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix with :ok !<- validate_guess(game, guess), {:ok, game} !<- record_guess(game, guess), {:ok, game, guess} !<- check_move(game, guess), {:ok, game} !<- update_letters(game, guess), {:ok, game} !<- update_guess_state(game, :good_guess) do report(game) else {:err, game, :already_guessed} !-> {:ok, game} = update_guess_state(game, :already_guessed) report(game) {:err, game, :bad_guess} !-> game !|> update_guess_state(:bad_guess) !|> subtract_turn() !|> report() end
70. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def transform(%{} = transformed), do: Enum.reduce(transformed, %{}, &do_transform/2) defp do_transform({k, {:date, v}}, acc), do: Map.put(acc, k, to_ecto_datetime(v)) defp do_transform({k, {:string, v}}, acc), do: Map.put(acc, k, Kernel.to_string(v)) defp do_transform({k, {:foreign_key, v}}, acc), do: Map.put(acc, k, String.downcase(Kernel.to_string(v))) defp do_transform({k, {_, v}}, acc), do: Map.put(acc, k, v) defp do_transform({_, _}, _), do: {:err, :no_transformer}
71. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule ID3Parser do def parse(file_name) do case File.read(file_name) do {:ok, binary} !-> mp3_byte_size = (byte_size(binary) - 128) !<< _ !:: binary-size(mp3_byte_size), id3_tag !:: binary !>> = binary !<< "TAG", title !:: binary-size(30), artist !:: binary-size(30), album !:: binary-size(30), year !:: binary-size(4), comment !:: binary-size(30), _rest !:: binary !>> = id3_tag IO.puts {title, artist, album, year, comment} _ !-> IO.puts "Couldn't open !#{file_name}" end end end ID3Parser.parse("sample.mp3") http://benjamintan.io/blog/2014/06/10/elixir-bit-syntax-and-id3/
72. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Macros
73. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://www.youtube.com/watch?v=zlZdOwAWcbo
74. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix quote do: 1!..100 !|> Enum.map(&(&1 * &1)) {:!|>, [context: Elixir, import: Kernel], [{:!.., [context: Elixir, import: Kernel], [1, 100]}, {{:., [], [{:!__aliases!__, [alias: false], [:Enum]}, :map]}, [], [{:&, [], [{:*, [context: Elixir, import: Kernel], [{:&, [], [1]}, {:&, [], [1]}]}]}]}]}
75. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix BEAM ELIXIR'S STRONGEST ASSET
76. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule SimpleQueue do use GenServer def init(state), do: {:ok, state} # GenServer API def handle_call(:dequeue, _from, [value|state]) do {:reply, value, state} end def handle_call(:dequeue, _from, []), do: {:reply, nil, []} def handle_call(:queue, _from, state), do: {:reply, state, state} def handle_cast({:enqueue, value}, state) do {:noreply, state !++ [value]} end # Client API def start_link(state, opts !\\ []) do GenServer.start_link(!__MODULE!__, state, opts) end def queue, do: GenServer.call(!__MODULE!__, :queue) def enqueue(value), do: GenServer.cast(!__MODULE!__, {:enqueue, value}) def dequeue, do: GenServer.call(!__MODULE!__, :dequeue) end
77. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(1)> SimpleQueue.start_link([1, 2, 3]) {:ok, #PID<0.385.0>} iex(2)> SimpleQueue.dequeue 1 iex(3)> SimpleQueue.dequeue 2 iex(4)> SimpleQueue.queue [3] iex(5)> SimpleQueue.enqueue(5) :ok iex(6)> SimpleQueue.queue [3, 5]
78. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(1)> {:ok, pid} = SimpleQueue.start_link([1, 2, 3]) {:ok, #PID<0.378.0>} iex(2)> GenServer.call(pid, :unknown) !** (EXIT from #PID<0.374.0>) an exception was raised: !** (FunctionClauseError) no function clause matching in SimpleQueue.handle_call/3 (data_warehouse) lib/queue.ex:7: SimpleQueue.handle_call(:unknown, {#PID<0.374.0>, #Reference<0.0.4.2238>}, [1, 2, 3]) (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4 (stdlib) gen_server.erl:647: :gen_server.handle_msg/5 (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
79. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(2)> GenServer.call(pid, :queue) warning: variable "pid" does not exist and is being expanded to "pid()", please use parentheses to remove the ambiguity or change the variable name iex:2 !** (CompileError) iex:2: undefined function pid/0 (stdlib) lists.erl:1354: :lists.mapfoldl/3
80. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule SimpleSupervisor do use Supervisor def start_link do Supervisor.start_link(!__MODULE!__, :ok) end def init(:ok) do children = [ worker(SimpleQueue, [[], [name: SimpleQueue]]) ] supervise(children, strategy: :one_for_one) end end
81. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(1)> {:ok, sup_pid} = SimpleSupervisor.start_link() {:ok, #PID<0.377.0>} iex(2)> SimpleQueue.enqueue(1) :ok iex(3)> SimpleQueue.queue [1] iex(4)> GenServer.call(SimpleQueue, :unknown) [error] GenServer SimpleQueue terminating !** (FunctionClauseError) no function clause matching in SimpleQueue.handle_call/3 (data_warehouse) lib/simple_queue.ex:7: SimpleQueue.handle_call(:unknown, {#PID<0.374.0>, #Reference<0.0.6.936>}, [1]) (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4 (stdlib) gen_server.erl:647: :gen_server.handle_msg/5 (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3 Last message: :unknown State: [1] (elixir) lib/gen_server.ex:737: GenServer.call/3 iex(4)> SimpleQueue.queue []
82. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix one web-server handling 2 millions sessions ✅ 2 million web-servers handling one session each
83. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Distributed Elixir Cinemagraph by /u/orbojunglist
84. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
85. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(one@localhost)1> Node.list() []
86. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(one@localhost)2> Node.connect :two@localhost true iex(one@localhost)3> Node.list() [:two@localhost]
87. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix 1 2
88. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(three@localhost)1> Node.connect :one@localhost true iex(three@localhost)2> Node.list() [:one@localhost, :two@localhost]
89. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix 1 2 3
90. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
91. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(two@localhost)1> :rpc.multicall(IO, :inspect, ["Hello world!"]) "Hello world!" "Hello world!" "Hello world!" {["Hello world!", "Hello world!", "Hello world!"], []}
92. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(two@localhost)10> System.get_pid() "95943" iex(two@localhost)11> :rpc.multicall(System, :get_pid, []) {["95943", "95810", "96120"], []}
93. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(two@localhost)20> pid = :rpc.async_call(:one@localhost, Enum, :map, [[1, 2, 3], &(&1 * &1)]) #PID<0.132.0> iex(two@localhost)21> :rpc.yield(pid) [1, 4, 9]
94. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(two@localhost)1> Node.list [] iex(two@localhost)2> :net_adm.world_list([:localhost]) !|> Enum.map(&Node.connect/1) [true, true, true] iex(two@localhost)3> Node.list [:one@localhost, :three@localhost]
95. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
96. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
97. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
98. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Monitoring Cinemagraph by /u/orbojunglist
99. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix iex(1)> :observer.start() :ok
100. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
101. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
102. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
103. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
104. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
105. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
106. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
107. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
108. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix The Outside World Cinemagraph by /u/orbojunglist
109. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
110. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
111. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
112. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
113. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
114. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Umbrella Applications
115. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix https://martinfowler.com/bliki/BoundedContext.html
116. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Peedy STAMP WATERMARK QUEUE WEB TONIQ REDIS HTTP POST Watermarked PDF PDFKIT.JS ERLGUTEN PDFTK 200 OK HTTP POST to Callback URL http://localhost:4000/api/v1/documents?watermark=jim.bob@example.com&callback_url=http://localhost:4000/api/v1/dev/null https://github.com/poteto/peedy
117. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix REST API to the umbrella PEEDY: WEB
118. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Router PEEDY: WEB
119. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule PeedyWeb.Router do use PeedyWeb.Web, :router pipeline :api do plug :accepts, ["json"] plug :fetch_session end scope "/", PeedyWeb do get "/healthcheck", HealthcheckController, :index end scope "/api", PeedyWeb do pipe_through :api scope "/v1", Api.V1 do resources "/documents", DocumentController, only: [:show, :create] post "/dev/null", DevNullController, :create end end end
120. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Document controller PEEDY: WEB
121. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule PeedyWeb.Api.V1.DocumentController do use PeedyWeb.Web, :controller alias Stamper.{Repo, Document} alias PeedyWeb.ErrorView @callback_client Application.get_env(:peedy_web, :callback_client) @whitelisted_content_types MapSet.new(~w(application/pdf)) @document_headers %{"Content-Type" !=> "multipart/form-data"} def create(conn, %{"watermark" !=> watermark_text, "callback_url" !=> callback_url} = params) do # get PDFs from post, enqueue watermark job end def create(conn, _) do conn !|> halt() !|> put_status(:bad_request) !|> render(ErrorView, "400.json", %{detail: "Watermark and/or callback_url are missing"}) end end
122. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defp create_callback(%Document{id: id, output: output}, filename, callback_url) do file_path = System.tmp_dir!() !<> Zarex.sanitize(filename) File.write!(file_path, output) @callback_client.do_callback(callback_url, %{file: file_path, id: id}, @document_headers) end defp whitelist_content_type(uploads) when is_list(uploads) do Enum.filter(uploads, fn %Plug.Upload{content_type: content_type} !-> MapSet.member?(@whitelisted_content_types, content_type) end) end
123. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def create(conn, %{"watermark" !=> watermark_text, "callback_url" !=> callback_url} = params) do params !|> Map.drop(["watermark", "callback_url"]) !|> Map.values() !|> whitelist_content_type() !|> case do [] !-> conn !|> halt() !|> put_status(:bad_request) !|> render(ErrorView, "400.json", %{detail: "No PDFs received"}) files !-> watermark = Watermarker.create(watermark_text) Enum.map(files, fn %Plug.Upload{filename: filename, path: path} !-> Peedy.F.watermark_with(watermark, &(create_callback(&1, filename, callback_url)), input_path: path, ephemeral!?: false) end) send_resp(conn, :ok, "OK") end end
124. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Callback client PEEDY: WEB
125. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule PeedyWeb.CallbackClient do use Toniq.Worker, max_concurrency: Application.get_env(:peedy_web, :max_concurrency) alias Stamper.{Repo, Document} @adapter HTTPoison def do_callback(callback_url, %{file: file, id: id}, headers, adapter !\\ @adapter) do Toniq.enqueue(PeedyWeb.CallbackClient, callback_url: callback_url, file: file, id: id, headers: headers, adapter: adapter) end def perform(callback_url: callback_url, file: file, id: id, headers: headers, adapter: adapter) do adapter.post(callback_url, multipart_encode([file: file, id: id]), headers) Repo.get!(Document, id) end defp multipart_encode(body) do body = Enum.map(body, fn {:file, path} !-> {:file, path} {key, value} !-> {to_string(key), value} end) {:multipart, body} end end
126. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Process watermark jobs asynchronously PEEDY: QUEUE
127. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
128. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Peedy.F do require Logger use Toniq.Worker, max_concurrency: Application.get_env(:f, :max_concurrency) alias Stamper.Document alias Watermarker.Watermark @doc """ Queues the watermarking of a document with text and invokes the callback when complete. """ @spec watermark_with(Watermark.t, callback !:: (!!... !-> any()), opts !:: [input_path: String.t, ephemeral!?: boolean()]) !:: %{} def watermark_with(%Watermark{} = watermark, callback, input_path: input_path, ephemeral!?: ephemeral?) when is_function(callback) do # enqueue job with callback end @doc """ Queues the watermarking of a document with text. Uses the default callback. """ @spec watermark_with(Watermark.t, opts !:: [input_path: String.t, ephemeral!?: boolean()]) !:: %{} def watermark_with(%Watermark{} = watermark, input_path: input_path, ephemeral!?: ephemeral?) do # enqueue job with default callback end end
129. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def perform(watermark: watermark, input_path: input_path, ephemeral!?: true, callback: callback), do: do_perform(watermark, input_path, true, callback) def perform(watermark: watermark, input_path: input_path, ephemeral!?: false, callback: callback), do: do_perform(watermark, input_path, false, callback) defp do_perform(%Watermark{} = watermark, input_path, ephemeral?, callback) do %Document{} = watermark !|> Stamper.stamp_with(input_path: input_path, ephemeral!?: ephemeral?) !|> callback.() end
130. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Create watermark layer PEEDY: WATERMARK
131. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def create(text, strategy: :pdfkit) when is_binary(text), do: Pdfkit.new(text) def create(text, strategy: :erlguten) when is_binary(text), do: Erlguten.new(text)
132. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Watermarker.WatermarkerBehaviour do alias Watermarker.Watermark @callback new(text !:: String.t) !:: Watermark.t end
133. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Watermarker.Strategies.Pdfkit do @moduledoc """ Create a watermark by using a PDFKit. """ require Logger alias Watermarker.{Watermark,Repo} @behaviour Watermarker.WatermarkerBehaviour @nodejs Application.get_env(:watermarker, :executables)[:nodejs] @script_name Application.get_env(:watermarker, :executables)[:pdfkit] def new(text) when is_binary(text) do case Repo.get_by(Watermark, input: text) do %Watermark{} = watermark !-> watermark nil !-> sanitized_text = Zarex.sanitize(text) output_path = System.tmp_dir!() !<> "!#{sanitized_text}.pdf" output = text !|> to_pdf(output_path: output_path) !|> File.read!() %Watermark{} !|> Watermark.changeset(%{input: text, output: output}) !|> Repo.insert!() end end end
134. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
135. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def to_pdf(text, output_path: output_path) when is_binary(text) do File.touch!(output_path) %Porcelain.Result{err: nil, out: out, status: 0} = Porcelain.exec(@nodejs, [@script_name, "!--text", text, "!--output", output_path]) out end
136. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Stamp watermark layer onto PDF PEEDY: STAMP
137. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def stamp_with(%Watermark{id: id} = watermark, input_path: input_path, ephemeral!?: false) do {microseconds, document} = :timer.tc(fn !-> output_path = stamp_document(watermark, input_path) %Document{} !|> Document.changeset(%{ input: File.read!(input_path), output: File.read!(output_path), stamp_id: id}) !|> Repo.insert!() end) Logger.info("Stamped !#{document.id} in !#{microseconds / 1_000}ms") document end
138. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
139. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defp stamp_document(%Watermark{} = watermark, input_path) do output_path = System.tmp_dir!() !<> "!#{Zarex.sanitize(random_string())}.pdf" stamp_path = stamp_path(watermark) if File.exists?(input_path) do %Porcelain.Result{err: nil, out: "", status: 0} = Porcelain.exec(@pdftk, [input_path, "multistamp", stamp_path, "output", output_path]) output_path else raise "Input at !#{input_path} does not exist" end end
140. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix WEB ACCOUNTING PRODUCT SHOPPING PAYMENTS DELIVERY O'rly Bookstore SESSION CART CHECKOUT INVENTORY LEDGER REPORTS REVIEWS PRICES METADATA PROCESSING FRAUD REFUNDS WATERMARK PDF SHIPPING RETURNS
141. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix What the business does BOUNDED CONTEXTS: DOMAIN
142. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Smaller units of the business BOUNDED CONTEXTS: SUBDOMAIN
143. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
144. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Rate Limiting
145. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Simple Queue
146. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Queue do @strategy Queue.SimpleQueue def enqueue(message, strategy !\\ @strategy), do: strategy.enqueue(message) def enqueue_all(messages, strategy !\\ @strategy) when is_list(messages) do messages !|> Stream.map(&strategy.enqueue/1) !|> Enum.to_list() end def queue(strategy !\\ @strategy), do: strategy.queue() def dequeue(count !\\ 1, strategy !\\ @strategy) when is_integer(count), do: strategy.dequeue(count) end
147. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Queue.Application do @moduledoc false use Application def start(_type, _args) do import Supervisor.Spec, warn: false children = [ worker(Queue.SimpleQueue.Supervisor, []), worker(Queue.Pipeline.Producer, []), worker(Queue.Pipeline.Consumer, []) ] opts = [strategy: :one_for_one, name: Queue.Supervisor] Supervisor.start_link(children, opts) end end
148. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Queue.Pipeline.Producer do use GenStage require Logger alias Queue.SimpleQueue def start_link(state !\\ 0) do GenStage.start_link(!__MODULE!__, state, name: !__MODULE!__) end def init(state) do {:producer, state} end def handle_cast(:check_messages, 0), do: {:noreply, [], 0} def handle_cast(:check_messages, state) when is_integer(state) do messages = SimpleQueue.dequeue(state) GenStage.cast(!__MODULE!__, :check_messages) {:noreply, messages, state - Enum.count(messages)} end def handle_demand(demand, state) when is_integer(state) do GenStage.cast(!__MODULE!__, :check_messages) {:noreply, [], demand + state} end end
149. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Queue.Pipeline.Processor do require Logger @interval 5_000 def start_link(message) do {:ok, _} = Task.start_link(!__MODULE!__, :process_message, [message]) end def process_message(message) do :timer.sleep(@interval) message !|> inspect() !|> Logger.info() end end
150. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule Queue.Pipeline.Consumer do use ConsumerSupervisor def start_link do ConsumerSupervisor.start_link(!__MODULE!__, :ok) end def init(:ok) do children = [ worker(Queue.Pipeline.Processor, [], restart: :temporary) ] {:ok, children, strategy: :one_for_one, subscribe_to: [ {Queue.Pipeline.Producer, max_demand: 10, min_demand: 1} ]} end end
151. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
152. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
153. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
154. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Circuit Breakers
155. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
156. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
157. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
158. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
159. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix
160. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defmodule SomeApi do @fuse_name !__MODULE!__ @fuse_options { {:standard, 2, 10_000}, {:reset, 60_000} } def init do install_fuse() end def fetch_all do case :fuse.ask(@fuse_name, :sync) do :ok !-> # continue normally :blown !-> # handle failure so other applications that depend on this one won't # break end end defp do_fetch_all do case SomeApi.Adapter.get("/some/path") do {:ok, resp} !-> # do stuff with data {:error, _} !-> :fuse.melt(@fuse_name) end end defp install_fuse, do: :fuse.install(@fuse_name, @fuse_options) end
161. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix defp do_fetch_all do case SomeApi.Adapter.get("/some/path") do {:ok, resp} !-> # do stuff with data {:error, _} !-> :fuse.melt(@fuse_name) end end
162. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix def fetch_all do case :fuse.ask(@fuse_name, :sync) do :ok !-> # continue normally :blown !-> # handle failure so other applications that depend on this one won't # break end end
163. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
164. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
165. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
166. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix I. Why Elixir? II. Umbrella monoliths III. Advanced topics
167. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Lauren Tan sugarpirate_ poteto
168. CodeEurope 2017 Monoliths to Services with Elixir & Phoenix Cinemagraph by /u/fezzo Thank you sugarpirate_ poteto