Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Its all about the journey - ElixirDaze 2017

1b7fef03e1f15f2cfd76eee5b0830d3f?s=47 Keith
March 03, 2017

Its all about the journey - ElixirDaze 2017

This talk shares some of the interesting ideas we have discovered whilst building the card generator too we used to create ElixirCards - the cards will be available on amazon soon. Info will be posted to www.elixircards.co.uk when we return to the UK.

1b7fef03e1f15f2cfd76eee5b0830d3f?s=128

Keith

March 03, 2017
Tweet

Transcript

  1. Its all about the Journey by Tanya and Keith ElixirDaze

    2017 #elixirdaze2017
  2. Chapter 1 Introduction ElixirDaze 2017 #elixirdaze2017

  3. The journey of a thousand miles begins with one step

    Lao Tzu ElixirDaze 2017 #elixirdaze2017
  4. Chapter 2 Written by hand ElixirDaze 2017 #elixirdaze2017

  5. Southampton Elixir Meetup Nov 2015 ElixirDaze 2017 #elixirdaze2017

  6. Chapter 3 First steps ElixirDaze 2017 #elixirdaze2017

  7. ElixirDaze 2017 #elixirdaze2017

  8. ElixirDaze 2017 #elixirdaze2017

  9. The idea of a Card Generator was born ElixirDaze 2017

    #elixirdaze2017
  10. Chapter 4 mail merge ElixirDaze 2017 #elixirdaze2017

  11. Inkskape and Ruby ElixirDaze 2017 #elixirdaze2017

  12. Chapter 5 Python and HTML ElixirDaze 2017 #elixirdaze2017

  13. ElixirDaze 2017 #elixirdaze2017

  14. Chapter 6 Can we make it better? ElixirDaze 2017 #elixirdaze2017

  15. Summary and limitations Generate from CSV is working Google spreadsheet

    works for now Generate HTML, not PDF, much faster! Building with EVERYTHING BUT(!) elixir Need to verify correctness ElixirDaze 2017 #elixirdaze2017
  16. Things we want: Correct cards Fast Stable and reliable Scalable

    ElixirDaze 2017 #elixirdaze2017
  17. Chapter 7 Elixir to the rescue \o/ ElixirDaze 2017 #elixirdaze2017

  18. With Elixir we get: Tests with a bit of meta

    programming Fast with concurrency Reliable with processes and supervision Scalable with OTP applications ElixirDaze 2017 #elixirdaze2017
  19. Let's get it started $ mix.new card_generator ElixirDaze 2017 #elixirdaze2017

  20. Chapter 8 Architecture ElixirDaze 2017 #elixirdaze2017

  21. ElixirDaze 2017 #elixirdaze2017

  22. Chapter 9 Downloader Module ElixirDaze 2017 #elixirdaze2017

  23. defmodule Downloader do def download(url, _file_path) do HTTPotion.get(url) |> Map.get(:body)

    |> save(file_path) end end What could go wrong with that? ElixirDaze 2017 #elixirdaze2017
  24. defmodule Downloader do def download(url, file_path) do case HTTPotion.get(url) do

    %Response{body: content} -> save(content, file_path) %ErrorResponse{message: error} -> error end end end ElixirDaze 2017 #elixirdaze2017
  25. def save(content, file_path) do {:ok, file} = File.open(file_path, [:write]) IO.binwrite(file,

    content) File.close(file) end So what could go wrong? ElixirDaze 2017 #elixirdaze2017
  26. iex> h File.open^200 ... {:error, reason} - the file could

    not be opened. ElixirDaze 2017 #elixirdaze2017
  27. def save(content, file_path) do {:ok, file} = File.open(file_path, [:write]) #

    <---- here IO.binwrite(file, content) File.close(file) end ElixirDaze 2017 #elixirdaze2017
  28. ** (MatchError) no match of right hand side value: {:error,

    :eacces} ElixirDaze 2017 #elixirdaze2017
  29. Erlang philosophy "Let it crash" doesn't mean don't handle errors

    ElixirDaze 2017 #elixirdaze2017
  30. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> IO.binwrite(file, content) File.close(file) {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  31. iex> h IO.binwrite^200 blah blah... ElixirDaze 2017 #elixirdaze2017

  32. @spec binwrite(device, iodata) :: :ok | {:error, term} def binwrite(device

    \\ :stdio, item) when is_iodata(item) do :file.write map_dev(device), item end ElixirDaze 2017 #elixirdaze2017
  33. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  34. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> File.close(file) reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  35. iex> h File.close ... It mostly returns :ok, except for

    some severe errors such as *out of memory*. ElixirDaze 2017 #elixirdaze2017
  36. @spec close(io_device) :: :ok | {:error, posix | :badarg |

    :terminated} ElixirDaze 2017 #elixirdaze2017
  37. Back to our beautiful monster ElixirDaze 2017 #elixirdaze2017

  38. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> File.close(file) reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  39. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) # <------------- Here :ok {:error, reason} -> File.close(file) reason # <----------------------- Here term end {:error, reason} -> reason # <--------- Here term end end ElixirDaze 2017 #elixirdaze2017
  40. @spec save(binary(), binary()) :: :ok | term ElixirDaze 2017 #elixirdaze2017

  41. @spec save(binary(), binary()) :: :ok | {:error, term} ElixirDaze 2017

    #elixirdaze2017
  42. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) # <------------------ Here :ok error -> File.close(file) error # <----------------------------- Here tuple end error -> error # <-------------------------- Here tuple end end ElixirDaze 2017 #elixirdaze2017
  43. @spec download(binary(), binary()) :: :ok | {:error, term} def download(url,

    file_path) do case HTTPotion.get(url) do %HTTPotion.Response{body: body} -> case save(body, file_path) do :ok -> "Succes Yay! We saved the file here: #{file_path}" {:error, reason} -> # <------ here pattern match "I got 99 problems but... #{inspect reason}" end %HTTPotion.ErrorResponse{message: error} -> error end end ElixirDaze 2017 #elixirdaze2017
  44. @spec download(binary(), binary()) :: :ok | {:error, term} def download(url,

    file_path) do case HTTPotion.get(url) do %Response{body: body} -> save(body, file_path) %ErrorResponse{message: error} -> {:error, error} end end ElixirDaze 2017 #elixirdaze2017
  45. Once again back to our little monster ElixirDaze 2017 #elixirdaze2017

  46. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  47. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  48. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> # <---- write file File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  49. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> # <---- write file File.close(file) # <---- close file error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  50. Use the force Luke Kernel.with Special Form!! Used to combine

    matching clauses ElixirDaze 2017 #elixirdaze2017
  51. def save(content, file_path) do with {:ok, file} -> File.open(file_path, [:write]),

    # <--- note the commas :ok -> IO.binwrite(file, content), # <--- note the commas :ok -> File.close(file) do :ok -> {:ok, file_path} # <---- the happy case result else error -> error # <----- all errors pop out here end end ElixirDaze 2017 #elixirdaze2017
  52. ElixirDaze 2017 #elixirdaze2017

  53. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> try do IO.binwrite(file, content) # <--- this returns result after File.close(file) end other -> other end end ElixirDaze 2017 #elixirdaze2017
  54. def save(content, file_path) do File.open(file_path, [:write], fn(file) -> IO.binwrite(file, content)

    File.close(file) # <-- returns only the result of close/1 end) end ElixirDaze 2017 #elixirdaze2017
  55. def save(content, file_path) do File.open(file_path, [:write], fn(file) -> try do

    IO.binwrite(file, content) # <-- returns result write/1 after File.close(file) end end) end ElixirDaze 2017 #elixirdaze2017
  56. IO! IO! IO! Use a Monad! ElixirDaze 2017 #elixirdaze2017

  57. # equivalent of bind (>>=) in Haskell defmacro left ~>>

    right do quote do (fn -> case unquote(left) do {:ok, x} -> x |> unquote(right) # <--- notice x! {:error, _} = expr -> expr end end).() end end ElixirDaze 2017 #elixirdaze2017
  58. def save(content, file_path) do open_file(file_path) ~>> write_file(content) ~>> close_file(file_path) end

    ElixirDaze 2017 #elixirdaze2017
  59. def save(content, file_path) do open_file(file_path) # <-- we need to

    lift into Either type ~>> write_file(content) ~>> close_file(file_path) end ElixirDaze 2017 #elixirdaze2017
  60. def open_file(file_path) do # <—— here we need to lift

    our functions into "Either" :ok = File.mkdir_p(Path.dirname(file_path)) File.open(file_path, [:write]) # <- this returns {:ok, file} end def write_file(file, content) do # <— here case IO.binwrite(file, content) do :ok -> {:ok, file} # <- here we return {:ok, file} other -> other end end def close_file(file, file_path) do # <— here case File.close(file) do :ok -> {:ok, file_path} other -> other end end ElixirDaze 2017 #elixirdaze2017
  61. {:ok, term} | {:error, term} ElixirDaze 2017 #elixirdaze2017

  62. defmodule Bind do @moduledoc "Bind >>= or Flatmap" @doc "Compose

    functions which return {:ok, term} | {:error, term}" defmacro left >>= right do quote do (fn -> case unquote(left) do {:ok, x} -> x |> unquote(right) # <--- notice x! {:error, _} = expr -> expr end end).() end end defmacro __using__(_) do quote do import Kernel, except: [~>>: 2] end end end ElixirDaze 2017 #elixirdaze2017
  63. Libraries? CrowdHailer/OK rob-brown/MonadEx many more... ElixirDaze 2017 #elixirdaze2017

  64. Want to learn more? Go watch Scott Wlaschin wonderful talk

    "Railway Oriented Programming" Watch all of Scott's ^ talks - he knows a thing or two about functional Programming Watch Brian Lonsdorf's talks and screencasts - he knows his sh*t Learn about Monoids, Functors, Applicatives and Monads - it's fun! Listen to Lamdacast & Magic Read Along, Functional Geekery podcasts Listen to Elixir Fountain! - (ofc) I assume you already do! ElixirDaze 2017 #elixirdaze2017
  65. $ iex -S mix iex> Downloader.download("elixir", "pack1") ElixirDaze 2017 #elixirdaze2017

  66. remember iex history git clone git@github.com:ferd/erlang-history.git cd erlang-history make install

    ElixirDaze 2017 #elixirdaze2017
  67. Let's create a mix task! \o/ ElixirDaze 2017 #elixirdaze2017

  68. $ mix cards.download elixir pack1 ElixirDaze 2017 #elixirdaze2017

  69. defmodule Mix.Tasks.Cards.Download do use Mix.Task def run(_) do IO.puts "Called

    with invalid parameters" IO.puts "Example: mix.cards.download elixir pack1" end end ElixirDaze 2017 #elixirdaze2017
  70. $ mix compile ElixirDaze 2017 #elixirdaze2017

  71. $ mix cards.download Called with invalid parameters Example: mix.cards.download elixir

    pack1 ElixirDaze 2017 #elixirdaze2017
  72. $ mix -h ElixirDaze 2017 #elixirdaze2017

  73. Why our task not in the list? ElixirDaze 2017 #elixirdaze2017

  74. defmodule Mix.Tasks.Cards.Download do use Mix.Task @shortdoc "downloads cards from google

    spreadsheets" ... end ElixirDaze 2017 #elixirdaze2017
  75. Changed mix task? Have you compiled? ElixirDaze 2017 #elixirdaze2017

  76. ElixirDaze 2017 #elixirdaze2017

  77. defmodule Mix.Tasks.Cards.Download do ... @spec run([String.t, String.t]) :: {:ok, String.t}

    | {:error, term} def run([lang, pack]) do Downloader.download(lang, pack) end def run(_) do ... end ElixirDaze 2017 #elixirdaze2017
  78. defmodule Mix.Tasks.Cards.Download do ... @lang_packs [{"elixir", "pack1"}, ...] def run([lang,

    pack]) when {lang, pack} in @lang_packs do ... end put it in a list for multiple arguments. ElixirDaze 2017 #elixirdaze2017
  79. $ mix compile $ mix cards.download elixir pack1 "Downloaded /files/elixir_cards/pack1/pack1.csv"

    ElixirDaze 2017 #elixirdaze2017
  80. Chapter 10 Tester Module ElixirDaze 2017 #elixirdaze2017

  81. csv file "`round 3.58`","`=> 4`" "What args elem/2 function accept?","tuple,

    index" ElixirDaze 2017 #elixirdaze2017
  82. ElixirDaze 2017 #elixirdaze2017

  83. iex> Tester.test({"elixir", "pack1"}) ElixirDaze 2017 #elixirdaze2017

  84. defmodule Tester do def test({lang, pack}) do file = "files/#{lang}_cards/#{pack}/#{pack}.csv"

    ... end end ElixirDaze 2017 #elixirdaze2017
  85. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" ... end end ElixirDaze 2017 #elixirdaze2017
  86. ElixirDaze 2017 #elixirdaze2017

  87. ElixirDaze 2017 #elixirdaze2017

  88. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) ... end end ElixirDaze 2017 #elixirdaze2017
  89. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() ...

    end ElixirDaze 2017 #elixirdaze2017
  90. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |>

    NimbleCSV.RFC4180.parse_stream() ... end ElixirDaze 2017 #elixirdaze2017
  91. defmodule Tester.Mixfile do ... defp deps do [{:nimble_csv, ">= 0.1.0"}]

    end ... end ElixirDaze 2017 #elixirdaze2017
  92. File.stream!(file) |> NimbleCSV.RFC4180.parse_stream() # Lazily parses CSV from a stream

    and returns a stream of rows. ElixirDaze 2017 #elixirdaze2017
  93. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |>

    NimbleCSV.RFC4180.parse_stream() |> Stream.with_index(1) # <---- card id ... end ElixirDaze 2017 #elixirdaze2017
  94. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |>

    NimbleCSV.RFC4180.parse_stream() |> Stream.with_index(2) |> Stream.flat_map(fn({[q, a], line}) -> parse(q, a, line) end) end ElixirDaze 2017 #elixirdaze2017
  95. defmodule PrepareTests do ... defp parse(question, "`=> " <> <<

    _code :: binary >> = answer, line) do q = extract_code(question) a = extract_code(answer) [{line, q, a}] end defp parse(_, _, _), do: [] ... ElixirDaze 2017 #elixirdaze2017
  96. defmodule PrepareTests do ... q = extract_code(question) ... defp extract_code(data)

    do ~r/(\`(\`{2})?)((\n)*)?(=> )?(?>code>[^\`]*)(\`(\`{2})?)/ |> Regex.named_captures(data) |> Map.get("code") |> String.trim("\n") end ElixirDaze 2017 #elixirdaze2017
  97. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) ... end end ElixirDaze 2017 #elixirdaze2017
  98. ElixirDaze 2017 #elixirdaze2017

  99. ElixirDaze 2017 #elixirdaze2017

  100. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) TestIt.define(String.to_atom(pack), code_cards) ... end end ElixirDaze 2017 #elixirdaze2017
  101. defmodule TestIt do ExUnit.start def define(mod, content) do ... Module.create(mod,

    body, __ENV__) end end ElixirDaze 2017 #elixirdaze2017
  102. defmodule TestIt do ExUnit.start def define(mod, content) do body =

    quote do use ExUnit.Case unquote(generate_cases(content)) end Module.create(mod, body, __ENV__) end ... end ElixirDaze 2017 #elixirdaze2017
  103. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do ... end end end ElixirDaze 2017 #elixirdaze2017
  104. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) ... end end end ElixirDaze 2017 #elixirdaze2017
  105. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) quote do test "card#{unquote(line)}" do ... end end end end end ElixirDaze 2017 #elixirdaze2017
  106. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) quote do # <-- quote test "card#{unquote(line)}" do # <- unquote assert unquote(q) == unquote(a) # <- unquote end end end end end ElixirDaze 2017 #elixirdaze2017
  107. defmodule TestIt do ExUnit.start def define(mod, content) do body =

    quote do use ExUnit.Case unquote(generate_cases(content)) end Module.create(mod, body, __ENV__) end ... end ElixirDaze 2017 #elixirdaze2017
  108. ElixirDaze 2017 #elixirdaze2017

  109. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) TestIt.define(String.to_atom(pack), code_cards) TestIt.run() end end ElixirDaze 2017 #elixirdaze2017
  110. defmodule TestIt do ... def run() do Mix.env(:test) Mix.Tasks.Test.run([]) end

    end ElixirDaze 2017 #elixirdaze2017
  111. ElixirDaze 2017 #elixirdaze2017

  112. ElixirDaze 2017 #elixirdaze2017

  113. ElixirDaze 2017 #elixirdaze2017

  114. Chapter 11 Compiler Module ElixirDaze 2017 #elixirdaze2017

  115. ElixirDaze 2017 #elixirdaze2017

  116. iex> Generator.generate("elixir", "pack1") "elixircards/pack1.html" Generate an html containing 56 cards.

    Returns path where cards are saved. ElixirDaze 2017 #elixirdaze2017
  117. Converting String of code to Formatted HTML Tried to solve

    using Regular Expressions Things got difficult with recursive regex Changed tack, tried Tokeniser/Parser/Compiler This proved much easier to write AND test!! ElixirDaze 2017 #elixirdaze2017
  118. defmodule Generator do ... defp process(row, face) do row #

    string |> Tokenizer.tokenize # list of tokens |> Parser.parse # ast |> Compiler.compile # string html |> card(face) # card template end ElixirDaze 2017 #elixirdaze2017
  119. Tokenizer # before "Given the function: ``` foo = fn(a,

    b) -> a + b end ``` How would you call `this` function?" # after [ {:text, "Given the function:\n"}, {:code, "foo = fn(a, b) ->\na + b\nend"}, {:text, "\nHow would you call `this` function?"} ] ElixirDaze 2017 #elixirdaze2017
  120. Parser # before [ {:text, "Given the function:\n"}, {:code, "foo

    = fn(a, b) ->\na + b\nend"}, {:text, "\nHow would you call `this` function?"} ] # after {:card, [ {:text, "here is "}, {:code, "IO.puts"}, {:text, " text `inline code` more text"} ] } ElixirDaze 2017 #elixirdaze2017
  121. Compiler # before {:card, [ {:text, "here is "}, {:code,

    "IO.puts"}, {:text, " text `inline code` more text"} ] } # after <p>here is </p> <pre><code>IO.puts</code></pre> <p> text <code>inline code</code> more text</p> ElixirDaze 2017 #elixirdaze2017
  122. Mix task!! Love mix tasks! ElixirDaze 2017 #elixirdaze2017

  123. Chapter 12 Concurrent Downloader ElixirDaze 2017 #elixirdaze2017

  124. defmodule DownloaderBench do use Benchfella @lang "elixir" setup_all do HTTPotion.start

    end bench "download" do Downloader.not_concurrent(@lang) end bench "concurrent" do Downloader.concurrent(@lang) end end ElixirDaze 2017 #elixirdaze2017
  125. def not_concurrent(lang) do for x <- 1..20 do download_pack(lang, "pack#{x}")

    end end def concurrent(lang) do for x <- 1..20 do spawn(fn -> send(self(), download_pack(lang, "pack#{x}") ) end) end end ElixirDaze 2017 #elixirdaze2017
  126. $ mix.bench ElixirDaze 2017 #elixirdaze2017

  127. ElixirDaze 2017 #elixirdaze2017

  128. ElixirDaze 2017 #elixirdaze2017

  129. What's next? Concurrent all the things! \o/ Pipeline of operations

    Keeping state in supervised process Admin UI Move from spreadsheet to a DB Public UIs - multiplayer games \o/ ElixirDaze 2017 #elixirdaze2017
  130. It's good to have an end to journey toward, but

    it is the journey which matters at the end. Ernest Hemingway ElixirDaze 2017 #elixirdaze2017
  131. Summary handle or not to handle this is a question...

    mix tasks are easy and fun testing on the fly concurrency is huge performance gain write small things, it works! journey is what matters ElixirDaze 2017 #elixirdaze2017
  132. Questions ElixirDaze 2017 #elixirdaze2017