Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Its all about the journey - ElixirDaze 2017

Keith
March 03, 2017

Its all about the journey - ElixirDaze 2017

This talk shares some of the interesting ideas we have discovered whilst building the card generator too we used to create ElixirCards - the cards will be available on amazon soon. Info will be posted to www.elixircards.co.uk when we return to the UK.

Keith

March 03, 2017
Tweet

More Decks by Keith

Other Decks in Programming

Transcript

  1. The journey of a thousand miles begins with one step

    Lao Tzu ElixirDaze 2017 #elixirdaze2017
  2. Summary and limitations Generate from CSV is working Google spreadsheet

    works for now Generate HTML, not PDF, much faster! Building with EVERYTHING BUT(!) elixir Need to verify correctness ElixirDaze 2017 #elixirdaze2017
  3. With Elixir we get: Tests with a bit of meta

    programming Fast with concurrency Reliable with processes and supervision Scalable with OTP applications ElixirDaze 2017 #elixirdaze2017
  4. defmodule Downloader do def download(url, _file_path) do HTTPotion.get(url) |> Map.get(:body)

    |> save(file_path) end end What could go wrong with that? ElixirDaze 2017 #elixirdaze2017
  5. defmodule Downloader do def download(url, file_path) do case HTTPotion.get(url) do

    %Response{body: content} -> save(content, file_path) %ErrorResponse{message: error} -> error end end end ElixirDaze 2017 #elixirdaze2017
  6. def save(content, file_path) do {:ok, file} = File.open(file_path, [:write]) IO.binwrite(file,

    content) File.close(file) end So what could go wrong? ElixirDaze 2017 #elixirdaze2017
  7. iex> h File.open^200 ... {:error, reason} - the file could

    not be opened. ElixirDaze 2017 #elixirdaze2017
  8. def save(content, file_path) do {:ok, file} = File.open(file_path, [:write]) #

    <---- here IO.binwrite(file, content) File.close(file) end ElixirDaze 2017 #elixirdaze2017
  9. ** (MatchError) no match of right hand side value: {:error,

    :eacces} ElixirDaze 2017 #elixirdaze2017
  10. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> IO.binwrite(file, content) File.close(file) {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  11. @spec binwrite(device, iodata) :: :ok | {:error, term} def binwrite(device

    \\ :stdio, item) when is_iodata(item) do :file.write map_dev(device), item end ElixirDaze 2017 #elixirdaze2017
  12. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  13. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> File.close(file) reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  14. iex> h File.close ... It mostly returns :ok, except for

    some severe errors such as *out of memory*. ElixirDaze 2017 #elixirdaze2017
  15. @spec close(io_device) :: :ok | {:error, posix | :badarg |

    :terminated} ElixirDaze 2017 #elixirdaze2017
  16. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) {:error, reason} -> File.close(file) reason end {:error, reason} -> reason end end ElixirDaze 2017 #elixirdaze2017
  17. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> case IO.binwrite(file, content) do :ok -> File.close(file) # <------------- Here :ok {:error, reason} -> File.close(file) reason # <----------------------- Here term end {:error, reason} -> reason # <--------- Here term end end ElixirDaze 2017 #elixirdaze2017
  18. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) # <------------------ Here :ok error -> File.close(file) error # <----------------------------- Here tuple end error -> error # <-------------------------- Here tuple end end ElixirDaze 2017 #elixirdaze2017
  19. @spec download(binary(), binary()) :: :ok | {:error, term} def download(url,

    file_path) do case HTTPotion.get(url) do %HTTPotion.Response{body: body} -> case save(body, file_path) do :ok -> "Succes Yay! We saved the file here: #{file_path}" {:error, reason} -> # <------ here pattern match "I got 99 problems but... #{inspect reason}" end %HTTPotion.ErrorResponse{message: error} -> error end end ElixirDaze 2017 #elixirdaze2017
  20. @spec download(binary(), binary()) :: :ok | {:error, term} def download(url,

    file_path) do case HTTPotion.get(url) do %Response{body: body} -> save(body, file_path) %ErrorResponse{message: error} -> {:error, error} end end ElixirDaze 2017 #elixirdaze2017
  21. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> case IO.binwrite(file, content) do :ok -> File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  22. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  23. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> # <---- write file File.close(file) error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  24. @spec save(binary(), binary()) :: :ok | {:error, term} def save(content,

    file_path) do case File.open(file_path, [:write]) do {:ok, file} -> # <---- open file case IO.binwrite(file, content) do :ok -> # <---- write file File.close(file) # <---- close file error -> File.close(file) error end error -> error end end ElixirDaze 2017 #elixirdaze2017
  25. Use the force Luke Kernel.with Special Form!! Used to combine

    matching clauses ElixirDaze 2017 #elixirdaze2017
  26. def save(content, file_path) do with {:ok, file} -> File.open(file_path, [:write]),

    # <--- note the commas :ok -> IO.binwrite(file, content), # <--- note the commas :ok -> File.close(file) do :ok -> {:ok, file_path} # <---- the happy case result else error -> error # <----- all errors pop out here end end ElixirDaze 2017 #elixirdaze2017
  27. def save(content, file_path) do case File.open(file_path, [:write]) do {:ok, file}

    -> try do IO.binwrite(file, content) # <--- this returns result after File.close(file) end other -> other end end ElixirDaze 2017 #elixirdaze2017
  28. def save(content, file_path) do File.open(file_path, [:write], fn(file) -> IO.binwrite(file, content)

    File.close(file) # <-- returns only the result of close/1 end) end ElixirDaze 2017 #elixirdaze2017
  29. def save(content, file_path) do File.open(file_path, [:write], fn(file) -> try do

    IO.binwrite(file, content) # <-- returns result write/1 after File.close(file) end end) end ElixirDaze 2017 #elixirdaze2017
  30. # equivalent of bind (>>=) in Haskell defmacro left ~>>

    right do quote do (fn -> case unquote(left) do {:ok, x} -> x |> unquote(right) # <--- notice x! {:error, _} = expr -> expr end end).() end end ElixirDaze 2017 #elixirdaze2017
  31. def save(content, file_path) do open_file(file_path) # <-- we need to

    lift into Either type ~>> write_file(content) ~>> close_file(file_path) end ElixirDaze 2017 #elixirdaze2017
  32. def open_file(file_path) do # <—— here we need to lift

    our functions into "Either" :ok = File.mkdir_p(Path.dirname(file_path)) File.open(file_path, [:write]) # <- this returns {:ok, file} end def write_file(file, content) do # <— here case IO.binwrite(file, content) do :ok -> {:ok, file} # <- here we return {:ok, file} other -> other end end def close_file(file, file_path) do # <— here case File.close(file) do :ok -> {:ok, file_path} other -> other end end ElixirDaze 2017 #elixirdaze2017
  33. defmodule Bind do @moduledoc "Bind >>= or Flatmap" @doc "Compose

    functions which return {:ok, term} | {:error, term}" defmacro left >>= right do quote do (fn -> case unquote(left) do {:ok, x} -> x |> unquote(right) # <--- notice x! {:error, _} = expr -> expr end end).() end end defmacro __using__(_) do quote do import Kernel, except: [~>>: 2] end end end ElixirDaze 2017 #elixirdaze2017
  34. Want to learn more? Go watch Scott Wlaschin wonderful talk

    "Railway Oriented Programming" Watch all of Scott's ^ talks - he knows a thing or two about functional Programming Watch Brian Lonsdorf's talks and screencasts - he knows his sh*t Learn about Monoids, Functors, Applicatives and Monads - it's fun! Listen to Lamdacast & Magic Read Along, Functional Geekery podcasts Listen to Elixir Fountain! - (ofc) I assume you already do! ElixirDaze 2017 #elixirdaze2017
  35. defmodule Mix.Tasks.Cards.Download do use Mix.Task def run(_) do IO.puts "Called

    with invalid parameters" IO.puts "Example: mix.cards.download elixir pack1" end end ElixirDaze 2017 #elixirdaze2017
  36. defmodule Mix.Tasks.Cards.Download do ... @spec run([String.t, String.t]) :: {:ok, String.t}

    | {:error, term} def run([lang, pack]) do Downloader.download(lang, pack) end def run(_) do ... end ElixirDaze 2017 #elixirdaze2017
  37. defmodule Mix.Tasks.Cards.Download do ... @lang_packs [{"elixir", "pack1"}, ...] def run([lang,

    pack]) when {lang, pack} in @lang_packs do ... end put it in a list for multiple arguments. ElixirDaze 2017 #elixirdaze2017
  38. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" ... end end ElixirDaze 2017 #elixirdaze2017
  39. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) ... end end ElixirDaze 2017 #elixirdaze2017
  40. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |>

    NimbleCSV.RFC4180.parse_stream() ... end ElixirDaze 2017 #elixirdaze2017
  41. File.stream!(file) |> NimbleCSV.RFC4180.parse_stream() # Lazily parses CSV from a stream

    and returns a stream of rows. ElixirDaze 2017 #elixirdaze2017
  42. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |>

    NimbleCSV.RFC4180.parse_stream() |> Stream.with_index(1) # <---- card id ... end ElixirDaze 2017 #elixirdaze2017
  43. defmodule PrepareTests do def get_code_cards(file) do file |> File.stream!() |>

    NimbleCSV.RFC4180.parse_stream() |> Stream.with_index(2) |> Stream.flat_map(fn({[q, a], line}) -> parse(q, a, line) end) end ElixirDaze 2017 #elixirdaze2017
  44. defmodule PrepareTests do ... defp parse(question, "`=> " <> <<

    _code :: binary >> = answer, line) do q = extract_code(question) a = extract_code(answer) [{line, q, a}] end defp parse(_, _, _), do: [] ... ElixirDaze 2017 #elixirdaze2017
  45. defmodule PrepareTests do ... q = extract_code(question) ... defp extract_code(data)

    do ~r/(\`(\`{2})?)((\n)*)?(=> )?(?>code>[^\`]*)(\`(\`{2})?)/ |> Regex.named_captures(data) |> Map.get("code") |> String.trim("\n") end ElixirDaze 2017 #elixirdaze2017
  46. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) ... end end ElixirDaze 2017 #elixirdaze2017
  47. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) TestIt.define(String.to_atom(pack), code_cards) ... end end ElixirDaze 2017 #elixirdaze2017
  48. defmodule TestIt do ExUnit.start def define(mod, content) do ... Module.create(mod,

    body, __ENV__) end end ElixirDaze 2017 #elixirdaze2017
  49. defmodule TestIt do ExUnit.start def define(mod, content) do body =

    quote do use ExUnit.Case unquote(generate_cases(content)) end Module.create(mod, body, __ENV__) end ... end ElixirDaze 2017 #elixirdaze2017
  50. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do ... end end end ElixirDaze 2017 #elixirdaze2017
  51. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) ... end end end ElixirDaze 2017 #elixirdaze2017
  52. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) quote do test "card#{unquote(line)}" do ... end end end end end ElixirDaze 2017 #elixirdaze2017
  53. defmodule TestIt do ... defp generate_cases(content) do for {line, question,

    answer} <- content do q = Code.string_to_quoted!(question) a = Code.string_to_quoted!(answer) quote do # <-- quote test "card#{unquote(line)}" do # <- unquote assert unquote(q) == unquote(a) # <- unquote end end end end end ElixirDaze 2017 #elixirdaze2017
  54. defmodule TestIt do ExUnit.start def define(mod, content) do body =

    quote do use ExUnit.Case unquote(generate_cases(content)) end Module.create(mod, body, __ENV__) end ... end ElixirDaze 2017 #elixirdaze2017
  55. defmodule Tester do @packs [{"elixir", "pack1"}] def test({lang, pack} =

    input) when input in @packs do file = "files/#{lang}_cards/#{pack}/#{pack}.csv" code_cards = PrepareTests.get_code_cards(file) TestIt.define(String.to_atom(pack), code_cards) TestIt.run() end end ElixirDaze 2017 #elixirdaze2017
  56. iex> Generator.generate("elixir", "pack1") "elixircards/pack1.html" Generate an html containing 56 cards.

    Returns path where cards are saved. ElixirDaze 2017 #elixirdaze2017
  57. Converting String of code to Formatted HTML Tried to solve

    using Regular Expressions Things got difficult with recursive regex Changed tack, tried Tokeniser/Parser/Compiler This proved much easier to write AND test!! ElixirDaze 2017 #elixirdaze2017
  58. defmodule Generator do ... defp process(row, face) do row #

    string |> Tokenizer.tokenize # list of tokens |> Parser.parse # ast |> Compiler.compile # string html |> card(face) # card template end ElixirDaze 2017 #elixirdaze2017
  59. Tokenizer # before "Given the function: ``` foo = fn(a,

    b) -> a + b end ``` How would you call `this` function?" # after [ {:text, "Given the function:\n"}, {:code, "foo = fn(a, b) ->\na + b\nend"}, {:text, "\nHow would you call `this` function?"} ] ElixirDaze 2017 #elixirdaze2017
  60. Parser # before [ {:text, "Given the function:\n"}, {:code, "foo

    = fn(a, b) ->\na + b\nend"}, {:text, "\nHow would you call `this` function?"} ] # after {:card, [ {:text, "here is "}, {:code, "IO.puts"}, {:text, " text `inline code` more text"} ] } ElixirDaze 2017 #elixirdaze2017
  61. Compiler # before {:card, [ {:text, "here is "}, {:code,

    "IO.puts"}, {:text, " text `inline code` more text"} ] } # after <p>here is </p> <pre><code>IO.puts</code></pre> <p> text <code>inline code</code> more text</p> ElixirDaze 2017 #elixirdaze2017
  62. defmodule DownloaderBench do use Benchfella @lang "elixir" setup_all do HTTPotion.start

    end bench "download" do Downloader.not_concurrent(@lang) end bench "concurrent" do Downloader.concurrent(@lang) end end ElixirDaze 2017 #elixirdaze2017
  63. def not_concurrent(lang) do for x <- 1..20 do download_pack(lang, "pack#{x}")

    end end def concurrent(lang) do for x <- 1..20 do spawn(fn -> send(self(), download_pack(lang, "pack#{x}") ) end) end end ElixirDaze 2017 #elixirdaze2017
  64. What's next? Concurrent all the things! \o/ Pipeline of operations

    Keeping state in supervised process Admin UI Move from spreadsheet to a DB Public UIs - multiplayer games \o/ ElixirDaze 2017 #elixirdaze2017
  65. It's good to have an end to journey toward, but

    it is the journey which matters at the end. Ernest Hemingway ElixirDaze 2017 #elixirdaze2017
  66. Summary handle or not to handle this is a question...

    mix tasks are easy and fun testing on the fly concurrency is huge performance gain write small things, it works! journey is what matters ElixirDaze 2017 #elixirdaze2017