Its all about the journey - ElixirDaze 2017

This talk shares some of the interesting ideas we have discovered whilst building the card generator too we used to create ElixirCards - the cards will be available on amazon soon. Info will be posted to www.elixircards.co.uk when we return to the UK.

Transcript

1. Its all about the Journey by Tanya and Keith ElixirDaze 2017 #elixirdaze2017
2. Chapter 1 Introduction ElixirDaze 2017 #elixirdaze2017
3. The journey of a thousand miles begins with one step Lao Tzu ElixirDaze 2017 #elixirdaze2017
4. Chapter 2 Written by hand ElixirDaze 2017 #elixirdaze2017
5. Southampton Elixir Meetup Nov 2015 ElixirDaze 2017 #elixirdaze2017
6. Chapter 3 First steps ElixirDaze 2017 #elixirdaze2017
7. ElixirDaze 2017 #elixirdaze2017
8. ElixirDaze 2017 #elixirdaze2017
9. The idea of a Card Generator was born ElixirDaze 2017 #elixirdaze2017
10. Chapter 4 mail merge ElixirDaze 2017 #elixirdaze2017
11. Inkskape and Ruby ElixirDaze 2017 #elixirdaze2017
12. Chapter 5 Python and HTML ElixirDaze 2017 #elixirdaze2017
13. ElixirDaze 2017 #elixirdaze2017
14. Chapter 6 Can we make it better? ElixirDaze 2017 #elixirdaze2017
15. Summary and limitations Generate from CSV is working Google spreadsheet works for now Generate HTML, not PDF, much faster! Building with EVERYTHING BUT(!) elixir Need to verify correctness ElixirDaze 2017 #elixirdaze2017
16. Things we want: Correct cards Fast Stable and reliable Scalable ElixirDaze 2017 #elixirdaze2017
17. Chapter 7 Elixir to the rescue \o/ ElixirDaze 2017 #elixirdaze2017
18. With Elixir we get: Tests with a bit of meta programming Fast with concurrency Reliable with processes and supervision Scalable with OTP applications ElixirDaze 2017 #elixirdaze2017
19. Let's get it started $ mix.new card_generator ElixirDaze 2017 #elixirdaze2017
20. Chapter 8 Architecture ElixirDaze 2017 #elixirdaze2017
21. ElixirDaze 2017 #elixirdaze2017
22. Chapter 9 Downloader Module ElixirDaze 2017 #elixirdaze2017
23. defmodule Downloader do def download(url, _file_path) do HTTPotion.get(url) |> Map.get(:body) |> save(file_path) end end What could go wrong with that? ElixirDaze 2017 #elixirdaze2017
24. defmodule Downloader do def download(url, file_path) do case HTTPotion.get(url) do %Response{body: content} -> save(content, file_path) %ErrorResponse{message: error} -> error end end end ElixirDaze 2017 #elixirdaze2017
25. def save(content, file_path) do {:ok, file} = File.open(file_path, [:write]) IO.binwrite(file, content) File.close(file) end So what could go wrong? ElixirDaze 2017 #elixirdaze2017
26. iex> h File.open^200 ... {:error, reason} - the file could not be opened. ElixirDaze 2017 #elixirdaze2017
27. def save(content, file_path) do {:ok, file} = File.open(file_path, [:write]) # <---- here IO.binwrite(file, content) File.close(file) end ElixirDaze 2017 #elixirdaze2017
28. ** (MatchError) no match of right hand side value: {:error, :eacces} ElixirDaze 2017 #elixirdaze2017
29. Erlang philosophy "Let it crash" doesn't mean don't handle errors ElixirDaze 2017 #elixirdaze2017
30. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> IO.binwrite(file, content) File.close(file) {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
31. iex> h IO.binwrite^200 blah blah... ElixirDaze 2017 #elixirdaze2017
32. @spec binwrite(device, iodata) :: :ok | {:error, term} def binwrite(device \\ :stdio, item) when is_iodata(item) do :file.write map_dev(device), item end ElixirDaze 2017 #elixirdaze2017
33. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
34. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> File.close(file) reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
35. iex> h File.close ... It mostly returns :ok, except for some severe errors such as *out of memory*. ElixirDaze 2017 #elixirdaze2017
36. @spec close(io_device) :: :ok | {:error, posix | :badarg | :terminated} ElixirDaze 2017 #elixirdaze2017
37. Back to our beautiful monster ElixirDaze 2017 #elixirdaze2017
38. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> File.close(file) reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
39. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) # <------------- Here :ok {:error, reason} -> File.close(file) reason # <----------------------- Here term end {:error, reason} -> reason # <--------- Here term end end ElixirDaze 2017 #elixirdaze2017
40. @spec save(binary(), binary()) :: :ok | term ElixirDaze 2017 #elixirdaze2017
41. @spec save(binary(), binary()) :: :ok | {:error, term} ElixirDaze 2017 #elixirdaze2017
42. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) # <------------------ Here :ok error -> File.close(file) error # <----------------------------- Here tuple end error -> error # <-------------------------- Here tuple end end ElixirDaze 2017 #elixirdaze2017
43. @spec download(binary(), binary()) :: :ok | {:error, term} def download(url, file_path) do case HTTPotion.get(url) do %HTTPotion.Response{body: body} -> case save(body, file_path) do :ok -> "Succes Yay! We saved the file here: #{file_path}" {:error, reason} -> # <------ here pattern match "I got 99 problems but... #{inspect reason}" end %HTTPotion.ErrorResponse{message: error} -> error end end ElixirDaze 2017 #elixirdaze2017
44. @spec download(binary(), binary()) :: :ok | {:error, term} def download(url, file_path) do case HTTPotion.get(url) do %Response{body: body} -> save(body, file_path) %ErrorResponse{message: error} -> {:error, error} end end ElixirDaze 2017 #elixirdaze2017
45. Once again back to our little monster ElixirDaze 2017 #elixirdaze2017
46. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
47. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
48. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> # <---- write file File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
49. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> # <---- write file File.close(file) # <---- close file error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
50. Use the force Luke Kernel.with Special Form!! Used to combine matching clauses ElixirDaze 2017 #elixirdaze2017
51. def save(content, file_path) do with {:ok, file} -> File.open(file_path, [:write]), # <--- note the commas :ok -> IO.binwrite(file, content), # <--- note the commas :ok -> File.close(file) do :ok -> {:ok, file_path} # <---- the happy case result else error -> error # <----- all errors pop out here end end ElixirDaze 2017 #elixirdaze2017
52. ElixirDaze 2017 #elixirdaze2017
53. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file} -> try do IO.binwrite(file, content) # <--- this returns result after File.close(file) end other -> other end end ElixirDaze 2017 #elixirdaze2017
54. def save(content, file_path) do File.open(file_path, [:write], fn(file) -> IO.binwrite(file, content) File.close(file) # <-- returns only the result of close/1 end) end ElixirDaze 2017 #elixirdaze2017
55. def save(content, file_path) do File.open(file_path, [:write], fn(file) -> try do IO.binwrite(file, content) # <-- returns result write/1 after File.close(file) end end) end ElixirDaze 2017 #elixirdaze2017
56. IO! IO! IO! Use a Monad! ElixirDaze 2017 #elixirdaze2017
57. # equivalent of bind (>>=) in Haskell defmacro left ~>> right do quote do (fn -> case unquote(left) do {:ok, x} -> x |> unquote(right) # <--- notice x! {:error, _} = expr -> expr end end).() end end ElixirDaze 2017 #elixirdaze2017
58. def save(content, file_path) do open_file(file_path) ~>> write_file(content) ~>> close_file(file_path) end ElixirDaze 2017 #elixirdaze2017
59. def save(content, file_path) do open_file(file_path) # <-- we need to lift into Either type ~>> write_file(content) ~>> close_file(file_path) end ElixirDaze 2017 #elixirdaze2017
60. def open_file(file_path) do # <—— here we need to lift our functions into "Either" :ok = File.mkdir_p(Path.dirname(file_path)) File.open(file_path, [:write]) # <- this returns {:ok, file} end def write_file(file, content) do # <— here case IO.binwrite(file, content) do :ok -> {:ok, file} # <- here we return {:ok, file} other -> other end end def close_file(file, file_path) do # <— here case File.close(file) do :ok -> {:ok, file_path} other -> other end end ElixirDaze 2017 #elixirdaze2017
61. {:ok, term} | {:error, term} ElixirDaze 2017 #elixirdaze2017
62. defmodule Bind do @moduledoc "Bind >>= or Flatmap" @doc "Compose functions which return {:ok, term} | {:error, term}" defmacro left >>= right do quote do (fn -> case unquote(left) do {:ok, x} -> x |> unquote(right) # <--- notice x! {:error, _} = expr -> expr end end).() end end defmacro __using__(_) do quote do import Kernel, except: [~>>: 2] end end end ElixirDaze 2017 #elixirdaze2017
63. Libraries? CrowdHailer/OK rob-brown/MonadEx many more... ElixirDaze 2017 #elixirdaze2017
64. Want to learn more? Go watch Scott Wlaschin wonderful talk "Railway Oriented Programming" Watch all of Scott's ^ talks - he knows a thing or two about functional Programming Watch Brian Lonsdorf's talks and screencasts - he knows his sh*t Learn about Monoids, Functors, Applicatives and Monads - it's fun! Listen to Lamdacast & Magic Read Along, Functional Geekery podcasts Listen to Elixir Fountain! - (ofc) I assume you already do! ElixirDaze 2017 #elixirdaze2017
65. $ iex -S mix iex> Downloader.download("elixir", "pack1") ElixirDaze 2017 #elixirdaze2017
66. remember iex history git clone git@github.com:ferd/erlang-history.git cd erlang-history make install ElixirDaze 2017 #elixirdaze2017
67. Let's create a mix task! \o/ ElixirDaze 2017 #elixirdaze2017
68. $ mix cards.download elixir pack1 ElixirDaze 2017 #elixirdaze2017
69. defmodule Mix.Tasks.Cards.Download do use Mix.Task def run(_) do IO.puts "Called with invalid parameters" IO.puts "Example: mix.cards.download elixir pack1" end end ElixirDaze 2017 #elixirdaze2017
70. $ mix compile ElixirDaze 2017 #elixirdaze2017
71. $ mix cards.download Called with invalid parameters Example: mix.cards.download elixir pack1 ElixirDaze 2017 #elixirdaze2017
72. $ mix -h ElixirDaze 2017 #elixirdaze2017
73. Why our task not in the list? ElixirDaze 2017 #elixirdaze2017
74. defmodule Mix.Tasks.Cards.Download do use Mix.Task @shortdoc "downloads cards from google spreadsheets" ... end ElixirDaze 2017 #elixirdaze2017
75. Changed mix task? Have you compiled? ElixirDaze 2017 #elixirdaze2017
76. ElixirDaze 2017 #elixirdaze2017
77. defmodule Mix.Tasks.Cards.Download do ... @spec run([String.t, String.t]) :: {:ok, String.t} | {:error, term} def run([lang, pack]) do Downloader.download(lang, pack) end def run(_) do ... end ElixirDaze 2017 #elixirdaze2017
78. defmodule Mix.Tasks.Cards.Download do ... @lang_packs [{"elixir", "pack1"}, ...] def run([lang, pack]) when {lang, pack} in @lang_packs do ... end put it in a list for multiple arguments. ElixirDaze 2017 #elixirdaze2017
79. $ mix compile $ mix cards.download elixir pack1 "Downloaded /files/elixir_cards/pack1/pack1.csv" ElixirDaze 2017 #elixirdaze2017
80. Chapter 10 Tester Module ElixirDaze 2017 #elixirdaze2017
81. csv file "`round 3.58`","`=> 4`" "What args elem/2 function accept?","tuple, index" ElixirDaze 2017 #elixirdaze2017
82. ElixirDaze 2017 #elixirdaze2017
83. iex> Tester.test({"elixir", "pack1"}) ElixirDaze 2017 #elixirdaze2017
84. defmodule Tester do def test({lang, pack}) do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" ... end end ElixirDaze 2017 #elixirdaze2017
85. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} = input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" ... end end ElixirDaze 2017 #elixirdaze2017
86. ElixirDaze 2017 #elixirdaze2017
87. ElixirDaze 2017 #elixirdaze2017
88. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} = input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) ... end end ElixirDaze 2017 #elixirdaze2017
89. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() ... end ElixirDaze 2017 #elixirdaze2017
90. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |> NimbleCSV.RFC4180.parse_stream() ... end ElixirDaze 2017 #elixirdaze2017
91. defmodule Tester.Mixfile do ... defp deps do [{:nimble_csv, ">= 0.1.0"}] end ... end ElixirDaze 2017 #elixirdaze2017
92. File.stream!(file) |> NimbleCSV.RFC4180.parse_stream() # Lazily parses CSV from a stream and returns a stream of rows. ElixirDaze 2017 #elixirdaze2017
93. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |> NimbleCSV.RFC4180.parse_stream() |> Stream.with_index(1) # <---- card id ... end ElixirDaze 2017 #elixirdaze2017
94. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |> NimbleCSV.RFC4180.parse_stream() |> Stream.with_index(2) |> Stream.flat_map(fn({[q, a], line}) -> parse(q, a, line) end) end ElixirDaze 2017 #elixirdaze2017
95. defmodule PrepareTests do ... defp parse(question, "`=> " <> << _code :: binary >> = answer, line) do q = extract_code(question) a = extract_code(answer) [{line, q, a}] end defp parse(_, _, _), do: [] ... ElixirDaze 2017 #elixirdaze2017
96. defmodule PrepareTests do ... q = extract_code(question) ... defp extract_code(data) do ~r/(\`(\`{2})?)((\n)*)?(=> )?(?>code>[^\`]*)(\`(\`{2})?)/ |> Regex.named_captures(data) |> Map.get("code") |> String.trim("\n") end ElixirDaze 2017 #elixirdaze2017
97. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} = input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) ... end end ElixirDaze 2017 #elixirdaze2017
98. ElixirDaze 2017 #elixirdaze2017
99. ElixirDaze 2017 #elixirdaze2017
100. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} = input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) TestIt.define(String.to_atom(pack), code_cards) ... end end ElixirDaze 2017 #elixirdaze2017
101. defmodule TestIt do ExUnit.start def define(mod, content) do ... Module.create(mod, body, __ENV__) end end ElixirDaze 2017 #elixirdaze2017
102. defmodule TestIt do ExUnit.start def define(mod, content) do body = quote do use ExUnit.Case unquote(generate_cases(content)) end Module.create(mod, body, __ENV__) end ... end ElixirDaze 2017 #elixirdaze2017
103. defmodule TestIt do ... defp generate_cases(content) do for {line, question, answer} <- content do ... end end end ElixirDaze 2017 #elixirdaze2017
104. defmodule TestIt do ... defp generate_cases(content) do for {line, question, answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) ... end end end ElixirDaze 2017 #elixirdaze2017
105. defmodule TestIt do ... defp generate_cases(content) do for {line, question, answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) quote do test "card#{unquote(line)}" do ... end end end end end ElixirDaze 2017 #elixirdaze2017
106. defmodule TestIt do ... defp generate_cases(content) do for {line, question, answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) quote do # <-- quote test "card#{unquote(line)}" do # <- unquote assert unquote(q) == unquote(a) # <- unquote end end end end end ElixirDaze 2017 #elixirdaze2017
107. defmodule TestIt do ExUnit.start def define(mod, content) do body = quote do use ExUnit.Case unquote(generate_cases(content)) end Module.create(mod, body, __ENV__) end ... end ElixirDaze 2017 #elixirdaze2017
108. ElixirDaze 2017 #elixirdaze2017
109. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} = input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) TestIt.define(String.to_atom(pack), code_cards) TestIt.run() end end ElixirDaze 2017 #elixirdaze2017
110. defmodule TestIt do ... def run() do Mix.env(:test) Mix.Tasks.Test.run([]) end end ElixirDaze 2017 #elixirdaze2017
111. ElixirDaze 2017 #elixirdaze2017
112. ElixirDaze 2017 #elixirdaze2017
113. ElixirDaze 2017 #elixirdaze2017
114. Chapter 11 Compiler Module ElixirDaze 2017 #elixirdaze2017
115. ElixirDaze 2017 #elixirdaze2017
116. iex> Generator.generate("elixir", "pack1") "elixircards/pack1.html" Generate an html containing 56 cards. Returns path where cards are saved. ElixirDaze 2017 #elixirdaze2017
117. Converting String of code to Formatted HTML Tried to solve using Regular Expressions Things got difficult with recursive regex Changed tack, tried Tokeniser/Parser/Compiler This proved much easier to write AND test!! ElixirDaze 2017 #elixirdaze2017
118. defmodule Generator do ... defp process(row, face) do row # string |> Tokenizer.tokenize # list of tokens |> Parser.parse # ast |> Compiler.compile # string html |> card(face) # card template end ElixirDaze 2017 #elixirdaze2017
119. Tokenizer # before "Given the function: ``` foo = fn(a, b) -> a + b end ``` How would you call `this` function?" # after [ {:text, "Given the function:\n"}, {:code, "foo = fn(a, b) ->\na + b\nend"}, {:text, "\nHow would you call `this` function?"} ] ElixirDaze 2017 #elixirdaze2017
120. Parser # before [ {:text, "Given the function:\n"}, {:code, "foo = fn(a, b) ->\na + b\nend"}, {:text, "\nHow would you call `this` function?"} ] # after {:card, [ {:text, "here is "}, {:code, "IO.puts"}, {:text, " text `inline code` more text"} ] } ElixirDaze 2017 #elixirdaze2017
121. Compiler # before {:card, [ {:text, "here is "}, {:code, "IO.puts"}, {:text, " text `inline code` more text"} ] } # after <p>here is </p> <pre><code>IO.puts</code></pre> <p> text <code>inline code</code> more text</p> ElixirDaze 2017 #elixirdaze2017
122. Mix task!! Love mix tasks! ElixirDaze 2017 #elixirdaze2017
123. Chapter 12 Concurrent Downloader ElixirDaze 2017 #elixirdaze2017
124. defmodule DownloaderBench do use Benchfella @lang "elixir" setup_all do HTTPotion.start end bench "download" do Downloader.not_concurrent(@lang) end bench "concurrent" do Downloader.concurrent(@lang) end end ElixirDaze 2017 #elixirdaze2017
125. def not_concurrent(lang) do for x <- 1..20 do download_pack(lang, "pack#{x}") end end def concurrent(lang) do for x <- 1..20 do spawn(fn -> send(self(), download_pack(lang, "pack#{x}") ) end) end end ElixirDaze 2017 #elixirdaze2017
126. $ mix.bench ElixirDaze 2017 #elixirdaze2017
127. ElixirDaze 2017 #elixirdaze2017
128. ElixirDaze 2017 #elixirdaze2017
129. What's next? Concurrent all the things! \o/ Pipeline of operations Keeping state in supervised process Admin UI Move from spreadsheet to a DB Public UIs - multiplayer games \o/ ElixirDaze 2017 #elixirdaze2017
130. It's good to have an end to journey toward, but it is the journey which matters at the end. Ernest Hemingway ElixirDaze 2017 #elixirdaze2017
131. Summary handle or not to handle this is a question... mix tasks are easy and fun testing on the fly concurrency is huge performance gain write small things, it works! journey is what matters ElixirDaze 2017 #elixirdaze2017
132. Questions ElixirDaze 2017 #elixirdaze2017