Elixir Stream by Examples

Presented at ElixirCamp 3 (March 16–19, 2018) in Lord Somers Camp, Somers, VIC

1. Elixir Stream by Examples Qing Wu
3. Service S Integration
4. Data source: API endpoint • millions of records involved • pagination that maps to id ranges
5. Data source @spec fetch_data(integer) :: [Map.t] def fetch_data(page_number) do ... end
6. Sending to Service S • Generate CSV files • Upload to an FTP server
7. Sending data @spec handle_data([Map.t]) :: :ok | {:error, String.t} def handle_data(data) do ... end
8. Solution page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data |> handle_data end)
10. Enum Module • Provide functions to work with collections that can be enumerated over • Functions in Enum always produce immediate results
11. Enum examples iex(1)> result1 = 1..10 |> Enum.map(&(&1 + 20)) [21, 22, 23, 24, 25, 26, 27, 28, 29, 30] iex(2)> result2 = result1 |> Enum.filter(&(rem(&1, 2) == 0)) [22, 24, 26, 28, 30] iex(3)> result2 |> Enum.reduce(0, &+/2) 130
12. Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data |> handle_data end)
14. Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data |> handle_data end)
15. Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |> Enum.each(&handle_data/1)
16. Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |> Enum.each(&handle_data/1)
17. Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |> Enum.each(&handle_data/1)
18. Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |> Enum.each(&handle_data/1)
20. Stream module iex(1)> result1 = 1..10 |> Stream.map(&(&1 + 20)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>]]> iex(2)> result2 = result1 |> Stream.filter(&(rem(&1, 2) == 0)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>, #Function<39.51599720/1 in Stream.filter/2>]]> iex(3)> result2 |> Enum.reduce(0, &+/2) 130
21. Stream Solution page_range |> Stream.map(&fetch_data/1) |> Stream.concat |> Stream.chunk_every(10_000) |> Enum.each(&handle_data/1)
22. Summary • Stream module is very handy for lazy evaluation • saves unnecessary iterations and memory • determine when to trigger evaluation
23. Further reading http://elixir-lang.org/getting-started/ enumerables-and-streams.html Offical Docs https://hexdocs.pm/elixir/Stream.html