Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's ahead for Elixir: v1.2 and GenRouter

What's ahead for Elixir: v1.2 and GenRouter

With Elixir v1.1 just released, José Valim explains what is ahead for Elixir 1.2 and discuss what we may see in Elixir v1.3 and later.

Plataformatec

October 10, 2015
Tweet

More Decks by Plataformatec

Other Decks in Programming

Transcript

  1. @elixirlang / elixir-lang.org

    View full-size slide

  2. Elixir v1.0
    • September/2014
    • >180 contributors
    • 3 books out!
    • First Elixirconf!

    View full-size slide

  3. Extensibility
    • Web infrastructure
    • Embedded systems
    • Financial/Video platforms
    • Graphical User Interfaces

    View full-size slide

  4. Elixir v1.1
    • September/2015
    • >295 contributors
    • ~ 1000 packages on hex.pm
    • > 6.5million downloads

    View full-size slide

  5. Application.fetch_env!/2
    Keyword.get_and_update/3
    Keyword.get_lazy/3
    Keyword.pop_lazy/3
    Keyword.put_new_lazy/3
    Map.get_and_update/3
    Map.get_lazy/3
    Map.pop_lazy/3
    Map.put_new_lazy/3
    Enum.dedup/1
    Enum.dedup_by/2
    Enum.min_max/1
    Enum.min_max_by/2
    Enum.random/1
    Enum.reduce_while/3
    Enum.reverse_slice/3
    Enum.take_random/2
    More functions
    File.lstat/1
    File.lstat!/1
    File.rename/2
    Integer.digits/2
    Integer.undigits/2
    GenServer.whereis/1
    Process.hibernate/3
    Stream.dedup/1
    Stream.dedup_by/2
    Stream.transform/4
    String.jaro_distance/2
    String.splitter/3
    StringIO.flush/1
    Task.yield/2
    Task.shutdown/2
    Tuple.append/2
    URI.to_string/1

    View full-size slide

  6. Enum
    iex> Enum.dedup([1, 1, 2, 2, 1, 1])
    [1, 2, 1]
    iex> Enum.random 1..100
    51
    iex> Enum.take_random 1..100, 3
    [14, 3, 72]

    View full-size slide

  7. iex> String.jaro_distance "helo", "help"
    0.8333333333334
    String
    $ mix helo
    ** (Mix) The task "helo" could not be found.
    Did you mean "help"?

    View full-size slide

  8. # v1.0+
    iex> task = Task.async(fn -> ... end)
    %Task{}
    iex> Task.await(task, 5_000)
    # v1.1+
    iex> Task.yield(task, 5_000)
    {:ok, result} | nil
    iex> Task.shutdown(task, :brutal_kill)
    {:ok, result} | nil
    Task

    View full-size slide

  9. Capture Log
    # v1.0+
    import ExUnit.CaptureIO
    capture_io fn ->
    IO.puts "hello"
    end
    # v1.1+
    import ExUnit.CaptureLog
    capture_log fn ->
    Logger.info "hello"
    end

    View full-size slide

  10. # Per test
    @tag :capture_log
    test "something with logging" do
    ...
    end
    # Per suite
    ExUnit.configure capture_log: true
    Capture Log

    View full-size slide

  11. Capture Log
    1) test something with logging
    test/plug/error.exs:59
    ** (RuntimeError) oops
    stacktrace:
    test/plug/error.exs:59
    The following output was logged:
    20:27:52.443 [info] log message

    View full-size slide

  12. Not implemented tests
    # @tag :not_implemented
    test "eventually will do X"
    $ mix test --exclude not_implemented
    $ mix test --only not_implemented

    View full-size slide

  13. • stacktrace depth is now configurable
    and has been increased to 20
    • More detailed errors
    • Ability to skip tests
    • Proper line number in doctests failures
    More improvements

    View full-size slide

  14. mix profile.fprof
    # CNT ACC (ms) OWN (ms)
    Total 200279 1972.188 1964.579
    Test.run/0 1 1972.166 0.007
    Test.do_something/1 3 1972.131 0.040
    Test.bottleneck/0 1 1599.490 0.007
    ...
    $ mix profile.fprof my_script.exs

    View full-size slide

  15. [build_embedded: Mix.env == :prod,
    start_permanent: Mix.env == :prod]
    Closer to releases

    View full-size slide

  16. Start Permanent (true)
    • Shuts the node down if your
    application crashes

    View full-size slide

  17. Build Embedded (false)
    /_build
    ├── dev
    │ └── lib
    │ ├── cowboy
    │ │ └── ebin -> ../../../../deps/cowboy/ebin
    │ ├── cowlib
    │ │ ├── ebin -> ../../../../deps/cowlib/ebin
    │ ├── phoenix
    │ │ ├── ebin
    │ │ │ ├── Elixir.Access.Phoenix.Socket.beam

    View full-size slide

  18. Build embedded (true)
    • Packages your whole application
    into _build (no symlinks)
    • Enables protocol consolidation
    • No longer automatically compiles

    View full-size slide

  19. config/config.exs
    # v1.0: config/config.exs
    config :my_app, :key, :value
    # v1.1: this will warn
    config :unknown_app, :key, :value

    View full-size slide

  20. • "mix compile" tracks only compile-time
    dependencies for Elixir
    • "recompile()" inside IEx
    • Use the safer https protocol instead of
    git for :github dependencies
    More improvements

    View full-size slide

  21. Deprecations

    View full-size slide

  22. Elixir v1.1
    # v1.1
    iex> "Hello \u2661"
    "Hello ♡"
    # v1.0
    iex> "Hello \x{2661}”
    "Hello ♡"
    SOFT DEPRECATED

    View full-size slide

  23. Elixir v1.1
    # v1.0
    defmodule Calculator do
    use Behaviour
    defcallback add(number, number) :: number
    end
    SOFT DEPRECATED
    # v1.1
    defmodule Calculator do
    @callback add(number, number) :: number
    end

    View full-size slide

  24. Elixir v1.1
    defmodule Calculator do
    @typedoc "A complex number"
    @opaque complex :: ...
    @doc "Adds two numbers"
    @callback add(number, number) :: ...
    end
    Code.get_docs(Calculator, :type_docs)

    View full-size slide

  25. Elixir v1.1
    DEPRECATED
    defimpl Access, for: X do
    ...
    end

    View full-size slide

  26. Access
    opts[key] opts[key] opts[key] opts[key]
    Code
    Server

    View full-size slide

  27. • opts[key] still works
    • Access defines a subset of the Dict
    behaviour
    • The bottleneck does not exist in prod
    • Common hot-paths have been inlined
    Access

    View full-size slide

  28. Migration
    • Elixir v1.0, v1.1

    Support Erlang 17 & 18+

    • Elixir v1.2

    Supports Erlang 18+

    View full-size slide

  29. Erlang 18+
    • Variables in maps: %{key => value}
    • Official support for large maps

    View full-size slide

  30. Dicts in Elixir v1.1
    • Keyword, Map, HashDict
    • Unified via the Dict API
    • Too many options. Often confusing.

    View full-size slide

  31. Dicts in Elixir v1.2
    • Keyword lists - used as options, allow
    duplicate keys, user ordered
    • Maps - pattern matching, fast &
    scalable, term ordered

    View full-size slide

  32. Dict & Set
    • HashDict will be soft deprecated

    Use Map
    • HashSet will be soft deprecated

    Use MapSet
    • Dict and Set will be soft deprecated

    View full-size slide

  33. Multi-aliases
    alias MyApp.Foo
    alias MyApp.Bar
    alias MyApp.Baz
    alias MyApp.{Foo, Bar, Baz}

    View full-size slide

  34. December/2015

    View full-size slide

  35. Meanwhile,
    upgrade to
    Erlang 18+

    View full-size slide

  36. Collections
    sink (collectable)
    widgets
    |> Enum.filter(fn b -> b.color == :red end)
    |> Enum.map(fn b -> {b.title, b.height} end)
    |> Enum.into(%{})
    source (enumerable)

    View full-size slide

  37. Collections
    CSV.parse(path)
    |> Enum.filter(fn b -> b.color == :red end)
    |> Enum.map(fn b -> {b.title, b.height} end)
    |> Enum.into(IO.stream(:stdio, :inspect))

    View full-size slide

  38. Collections + Laziness
    CSV.parse(path)
    |> Stream.filter(fn b -> b.color == :red end)
    |> Stream.map(fn b -> {b.title, b.height} end)
    |> Stream.into(IO.stream(:stdio, :inspect))
    |> Stream.run()

    View full-size slide

  39. Pipeline parallelism
    CSV.parse(path)
    |> Stream.async()
    |> Stream.filter(fn b -> b.color == :red end)
    |> Stream.map(fn b -> {b.title, b.height} end)
    |> Stream.into(IO.stream(:stdio, :inspect))
    |> Stream.run()

    View full-size slide

  40. Pipeline parallelism
    CSV.parse(path)
    |> ...
    |> Stream.async()
    |> ...
    |> Stream.async()
    |> ...
    |> Stream.async()
    |> ...
    |> Stream.run()

    View full-size slide

  41. Pipeline parallelism
    async
    CSV async async run

    View full-size slide

  42. Pipeline parallelism
    • Each async stage is a process
    • What happens if one of them crash?
    • How to provide back-pressure?

    View full-size slide

  43. Pipeline parallelism
    • How to make streams supervised
    processes?
    • Pipeline parallelism isn’t even a good
    strategy

    View full-size slide

  44. How to make supervised
    processes stream data?

    View full-size slide

  45. GenRouter
    • Connects sources to sinks
    • Demand-driven

    View full-size slide

  46. GenRouter
    Router

    View full-size slide

  47. GenRouter
    Sink

    View full-size slide

  48. GenRouter
    Source

    View full-size slide

  49. GenRouter
    in | out

    View full-size slide

  50. GenRouter
    GenRouter.start_link(
    InPart, in_args,
    OutPart, out_args
    )

    View full-size slide

  51. GenRouter.start_link(
    GenRouter.SingleIn, [],
    GenRouter.BroadcastOut, []
    )
    GenRouter

    View full-size slide

  52. SingleIn - BroadcastOut
    router

    View full-size slide

  53. SingleIn - BroadcastOut
    router
    1
    1
    1
    1
    1

    View full-size slide

  54. SingleIn - BroadcastOut
    router
    1,2
    1,2
    1,2
    1,2
    1,2

    View full-size slide

  55. SingleIn - BroadcastOut
    router
    1,2,3
    1,2,3
    1,2,3
    1,2,3
    1,2,3

    View full-size slide

  56. DynamicIn - BroadcastOut
    router
    1,2,3
    1,2,3
    1,2,3
    1,2,3
    2
    1
    3

    View full-size slide

  57. DynamicIn - MessageQueueOut
    router
    1
    2
    3
    4
    2
    1
    3

    View full-size slide

  58. Some Use Cases
    • A GenEvent replacement that is
    process-based
    • A way to load-balance jobs across a
    pool of processes

    View full-size slide

  59. Demand-driven

    View full-size slide

  60. Demand-driven
    B C
    A Source | Sink Source | Sink

    View full-size slide

  61. Demand-driven
    B C
    A
    Ask 10
    Ask 10
    Sends max 10 Sends max 10

    View full-size slide

  62. Demand-driven
    • It is a message contract
    • It pushes back-pressure to the boundary
    • GenRouter is one impl of this contract
    • Inspired by Akka Streams

    View full-size slide

  63. Supervised TCP Acceptor
    # Source
    {:ok, acceptor} =
    GenRouter.TCPAcceptor.start_link(...)
    # Sink
    {:ok, supervisor} =
    GenRouter.Supervisor.start_link(...)
    # Source <-> Sink
    GenRouter.subscribe(supervisor, to: acceptor)

    View full-size slide

  64. The path forward
    • Define the demand-driven
    message contract
    • Implement GenRouter and related
    abstractions
    • Integrate with streams

    View full-size slide

  65. consulting and software engineering

    View full-size slide

  66. @elixirlang / elixir-lang.org

    View full-size slide