What's ahead for Elixir: v1.2 and GenRouter

With Elixir v1.1 just released, José Valim explains what is ahead for Elixir 1.2 and discuss what we may see in Elixir v1.3 and later.

Transcript

1. @elixirlang / elixir-lang.org
2. Elixir v1.0 • September/2014 • >180 contributors • 3 books out! • First Elixirconf!
3. Extensibility • Web infrastructure • Embedded systems • Financial/Video platforms • Graphical User Interfaces
4. Elixir v1.1
5. Elixir v1.1 • September/2015 • >295 contributors • ~ 1000 packages on hex.pm • > 6.5million downloads
7. Application.fetch_env!/2 Keyword.get_and_update/3 Keyword.get_lazy/3 Keyword.pop_lazy/3 Keyword.put_new_lazy/3 Map.get_and_update/3 Map.get_lazy/3 Map.pop_lazy/3 Map.put_new_lazy/3 Enum.dedup/1 Enum.dedup_by/2 Enum.min_max/1 Enum.min_max_by/2 Enum.random/1 Enum.reduce_while/3 Enum.reverse_slice/3 Enum.take_random/2 More functions File.lstat/1 File.lstat!/1 File.rename/2 Integer.digits/2 Integer.undigits/2 GenServer.whereis/1 Process.hibernate/3 Stream.dedup/1 Stream.dedup_by/2 Stream.transform/4 String.jaro_distance/2 String.splitter/3 StringIO.flush/1 Task.yield/2 Task.shutdown/2 Tuple.append/2 URI.to_string/1
8. Enum iex> Enum.dedup([1, 1, 2, 2, 1, 1]) [1, 2, 1] iex> Enum.random 1..100 51 iex> Enum.take_random 1..100, 3 [14, 3, 72]
9. iex> String.jaro_distance "helo", "help" 0.8333333333334 String $ mix helo ** (Mix) The task "helo" could not be found. Did you mean "help"?
10. # v1.0+ iex> task = Task.async(fn -> ... end) %Task{} iex> Task.await(task, 5_000) # v1.1+ iex> Task.yield(task, 5_000) {:ok, result} | nil iex> Task.shutdown(task, :brutal_kill) {:ok, result} | nil Task
11. ExUnit v1.1
12. Capture Log # v1.0+ import ExUnit.CaptureIO capture_io fn -> IO.puts "hello" end # v1.1+ import ExUnit.CaptureLog capture_log fn -> Logger.info "hello" end
13. # Per test @tag :capture_log test "something with logging" do ... end # Per suite ExUnit.configure capture_log: true Capture Log
14. Capture Log 1) test something with logging test/plug/error.exs:59 ** (RuntimeError) oops stacktrace: test/plug/error.exs:59 The following output was logged: 20:27:52.443 [info] log message
15. Not implemented tests # @tag :not_implemented test "eventually will do X" $ mix test --exclude not_implemented $ mix test --only not_implemented
16. • stacktrace depth is now configurable and has been increased to 20 • More detailed errors • Ability to skip tests • Proper line number in doctests failures More improvements
17. Mix v1.1
18. mix profile.fprof # CNT ACC (ms) OWN (ms) Total 200279 1972.188 1964.579 Test.run/0 1 1972.166 0.007 Test.do_something/1 3 1972.131 0.040 Test.bottleneck/0 1 1599.490 0.007 ... $ mix profile.fprof my_script.exs
19. [build_embedded: Mix.env == :prod, start_permanent: Mix.env == :prod] Closer to releases
20. Start Permanent (true) • Shuts the node down if your application crashes
21. Build Embedded (false) /_build ├── dev │ └── lib │ ├── cowboy │ │ └── ebin -> ../../../../deps/cowboy/ebin │ ├── cowlib │ │ ├── ebin -> ../../../../deps/cowlib/ebin │ ├── phoenix │ │ ├── ebin │ │ │ ├── Elixir.Access.Phoenix.Socket.beam
22. Build embedded (true) • Packages your whole application into _build (no symlinks) • Enables protocol consolidation • No longer automatically compiles
23. config/config.exs # v1.0: config/config.exs config :my_app, :key, :value # v1.1: this will warn config :unknown_app, :key, :value
24. • "mix compile" tracks only compile-time dependencies for Elixir • "recompile()" inside IEx • Use the safer https protocol instead of git for :github dependencies More improvements
25. Deprecations
26. Elixir v1.1 # v1.1 iex> "Hello \u2661" "Hello ♡" # v1.0 iex> "Hello \x{2661}” "Hello ♡" SOFT DEPRECATED
27. Elixir v1.1 # v1.0 defmodule Calculator do use Behaviour defcallback add(number, number) :: number end SOFT DEPRECATED # v1.1 defmodule Calculator do @callback add(number, number) :: number end
28. Elixir v1.1 defmodule Calculator do @typedoc "A complex number" @opaque complex :: ... @doc "Adds two numbers" @callback add(number, number) :: ... end Code.get_docs(Calculator, :type_docs)
29. Elixir v1.1 DEPRECATED defimpl Access, for: X do ... end
30. Access opts[key] opts[key] opts[key] opts[key] Code Server
31. • opts[key] still works • Access defines a subset of the Dict behaviour • The bottleneck does not exist in prod • Common hot-paths have been inlined Access
32. Community
35. Elixir v1.2
36. Migration • Elixir v1.0, v1.1
 Support Erlang 17 & 18+
 • Elixir v1.2
 Supports Erlang 18+
37. Erlang 18+ • Variables in maps: %{key => value} • Official support for large maps
38. Dicts in Elixir v1.1 • Keyword, Map, HashDict • Unified via the Dict API • Too many options. Often confusing.
39. Dicts in Elixir v1.2 • Keyword lists - used as options, allow duplicate keys, user ordered • Maps - pattern matching, fast & scalable, term ordered
40. Dict & Set • HashDict will be soft deprecated
 Use Map • HashSet will be soft deprecated
 Use MapSet • Dict and Set will be soft deprecated
41. Multi-aliases alias MyApp.Foo alias MyApp.Bar alias MyApp.Baz alias MyApp.{Foo, Bar, Baz}
42. December/2015
43. Meanwhile, upgrade to Erlang 18+
44. Elixir v1.3
45. Collections sink (collectable) widgets |> Enum.filter(fn b -> b.color == :red end) |> Enum.map(fn b -> {b.title, b.height} end) |> Enum.into(%{}) source (enumerable)
46. Collections CSV.parse(path) |> Enum.filter(fn b -> b.color == :red end) |> Enum.map(fn b -> {b.title, b.height} end) |> Enum.into(IO.stream(:stdio, :inspect))
47. Collections + Laziness CSV.parse(path) |> Stream.filter(fn b -> b.color == :red end) |> Stream.map(fn b -> {b.title, b.height} end) |> Stream.into(IO.stream(:stdio, :inspect)) |> Stream.run()
48. Pipeline parallelism CSV.parse(path) |> Stream.async() |> Stream.filter(fn b -> b.color == :red end) |> Stream.map(fn b -> {b.title, b.height} end) |> Stream.into(IO.stream(:stdio, :inspect)) |> Stream.run()
49. Pipeline parallelism CSV.parse(path) |> ... |> Stream.async() |> ... |> Stream.async() |> ... |> Stream.async() |> ... |> Stream.run()
50. Pipeline parallelism async CSV async async run
51. Pipeline parallelism • Each async stage is a process • What happens if one of them crash? • How to provide back-pressure?
52. Pipeline parallelism • How to make streams supervised processes? • Pipeline parallelism isn’t even a good strategy
53. How to make supervised processes stream data?
54. GenRouter
55. GenRouter • Connects sources to sinks • Demand-driven
56. GenRouter Router
57. GenRouter Sink
58. GenRouter Source
59. GenRouter in | out
60. GenRouter GenRouter.start_link( InPart, in_args, OutPart, out_args )
61. GenRouter.start_link( GenRouter.SingleIn, [], GenRouter.BroadcastOut, [] ) GenRouter
62. SingleIn - BroadcastOut router
63. SingleIn - BroadcastOut router 1 1 1 1 1
64. SingleIn - BroadcastOut router 1,2 1,2 1,2 1,2 1,2
65. SingleIn - BroadcastOut router 1,2,3 1,2,3 1,2,3 1,2,3 1,2,3
66. DynamicIn - BroadcastOut router 1,2,3 1,2,3 1,2,3 1,2,3 2 1 3
67. DynamicIn - MessageQueueOut router 1 2 3 4 2 1 3
68. Some Use Cases • A GenEvent replacement that is process-based • A way to load-balance jobs across a pool of processes
69. Demand-driven
70. Demand-driven B C A Source | Sink Source | Sink
71. Demand-driven B C A Ask 10 Ask 10 Sends max 10 Sends max 10
72. Demand-driven • It is a message contract • It pushes back-pressure to the boundary • GenRouter is one impl of this contract • Inspired by Akka Streams
73. Supervised TCP Acceptor # Source {:ok, acceptor} = GenRouter.TCPAcceptor.start_link(...) # Sink {:ok, supervisor} = GenRouter.Supervisor.start_link(...) # Source <-> Sink GenRouter.subscribe(supervisor, to: acceptor)
74. The path forward • Define the demand-driven message contract • Implement GenRouter and related abstractions • Integrate with streams
75. consulting and software engineering
76. @elixirlang / elixir-lang.org