Practical Elixir Streams — To Infinity and Beyond

By using Elixir Streams, you can make your application more responsive and consume fewer resources. Within this session, I shall explain how Streams work, and provide a few practical uses of Streams, with a live demonstration.

Transcript

1. Practical Elixir Streams:
   To Infinity and Beyond
   Evadne Wu
   Occasional Fixer-Upper
   ev@radi.ws
   twitter.com/evadne
   

   View Slide

2. Myself
   Another Elixir Programmer
   Inquiries: ev@radi.ws
   Arguments: twitter.com/evadne
   

   View Slide

3. 1
   Basics
   

   View Slide

4. Enumeration
   Enum: reduce, each, map, etc.
   for … in
   for … in, into:
   for … in, reduce:
   

   View Slide

5. Finite Enumerables
   [1, 2, 3]
   %{a: 1, b: 2}
   [a: 1, b: 2]
   1 .. 10
   

   View Slide

6. Finite Enumerables
   Lists, Maps, Keywords, etc. are finite Enumerables, with
   contents generated ahead of time
   All of them can be enumerated in the same manner — in
   linear or constant time, depending on the structure —
   because they all implement the Enumerable protocol
   

   View Slide

7. Lazy Enumerables
   Any data type which has an implementation of the
   Enumerable protocol can be enumerated.
   This attribute can in turn be used to create lazily
   evaluated Enumerables.
   

   View Slide

8. Lazy Enumerables
   Lazy Enumerables can act as placeholders for actual
   content which minimises blocking
   Computation is spread throughout the lifetime of the
   enumerable and can be halted if the results are no
   longer required
   

   View Slide

9. Lazy Enumerables
   Stream.map(1..3, &munge/1)
   File.stream!(path)
   Task.async_stream(paths, &{&1,
   File.read(&1)})
   

   View Slide

10. Using Lazy Enumerables
    Enum.to_list/1: to return a List
    Stream.run/1: to consume the Stream
    NB: Calling Enum functions on Streams forces evaluation
    

    View Slide

11. Anything is Enumerable
    > Since enumerables can have different shapes (structs,
    anonymous functions, and so on), the functions in this
    module may return any of those shapes and this may
    change at any time.
    

    View Slide

12. View Slide

13. 2
    Reduction
    

    View Slide

14. Enum.reduce/3
    The core concept which drives all enumeration
    Can be used to implement all other Enum functions
    Can be replaced with optimisation paths
    Calls Enumerable.reduce/3
    

    View Slide

15. Enumerable Protocol
    Defines a series of functions to be implemented by any
    data type that wishes to become enumerable
    Includes reduce/3, count/1, member?/2, slice/1
    with reduce/3 providing core functionality
    

    View Slide

16. View Slide

17. Enumerable.reduce/3
    The core building block of enumeration, which is
    implemented for all enumerable types
    Continuable, i.e. can be suspended
    

    View Slide

18. View Slide

19. Enumerable.reduce/3
    @type acc :: {:cont | :halt | :suspend, term}
    @type reducer :: (acc -> result)
    @spec reduce(t, acc, reducer) :: result
    

    View Slide

20. Infinite Enumerables
    Any data type that implements Enumerable, and
    always returns new items in its reduce/3
    implementation, is infinite.
    The Stream module provides convenience functions to
    construct such Enumerables!
    

    View Slide

21. Infinite Enumerables
    Stream.cycle([1, 2, 3])
    Stream.repeatedly(fn -> :hi end)
    

    View Slide

22. Infinite Enumerables
    Stream.unfold({0, 0}, fn
    {0, 0} -> {1, {1, 0}}
    {1, 0} -> {1, {1, 1}}
    {c, p} -> {c + p, {c + p, c}}
    end)
    

    View Slide

23. Stream Module = Convenience
    iex> Stream.map([1, 2, 3], & &1)
    #Stream<[
    enum: [1, 2, 3],
    funs: [
    #Function<48.68317796/1 in Stream.map/2>
    ]
    ]>
    

    View Slide

24. Stream Module = Convenience
    iex(4)> Stream.cycle([1, 2, 3])
    #Function<64.68317796/2 in Stream.unfold/2>
    

    View Slide

25. Incremental Peeling
    Abuse of Enumerable.reduce/3 allows incremental
    consumption of any Enumerable, including Streams!
    Trick: Start the enumeration in a suspended state,
    accumulate into nil, but keep returning items
    Note: Streams might be non-reentrant
    

    View Slide

26. Incremental Peeling
    reduce_fun = fn item, _ -> {:suspend, item} end
    acc = {:suspend, nil}
    result = Enumerable.reduce(enum, acc, reduce_fun)
    {:suspended, nil, next_fun} = result
    {:suspended, _, next_fun} = next_fun.({:cont, nil})
    …
    {:done, nil} = next_fun.({:cont, nil})
    

    View Slide

27. Note
    Streams are pull-based. No computation is done unless
    if new values are pulled from it.
    Collections are push-based. They can be used to collect
    output (e.g. File.Stream!/1)
    

    View Slide

28. 3
    Convenience
    

    View Slide

29. Stream.transform/3
    Similar to flat map
    Returns results
    

    View Slide

30. Stream.transform/3
    iex> enum = 1001..9999
    iex> n = 3
    iex> stream = Stream.transform(enum, 0, fn i, acc ->
    ...> if acc < n, do: {[i], acc + 1}, else: {:halt, acc}
    ...> end)
    iex> Enum.to_list(stream)
    [1001, 1002, 1003]
    

    View Slide

31. Stream.resource/3
    Creates initial accumulator in start_fun
    Calls next_fun repeatedly until completion
    Calls after_fun at the end of enumeration
    Can be used for single-pass (streaming) file generation
    

    View Slide

32. Stream.resource/3
    Stream.resource(
    fn -> File.open!("sample") end,
    fn file ->
    case IO.read(file, :line) do
    data when is_binary(data) -> {[data], file}
    _ -> {:halt, file}
    end
    end,
    fn file -> File.close(file) end
    )
    

    View Slide

33. Stream.into/2
    Used to redirect output of a stream into a Collectable
    Useful when the Collectable represents the outside
    world (e.g. a File Handle)
    

    View Slide

34. Stream.into/2
    File.stream!(path_input)
    |> Stream.map(&String.replace(&1, "#", "%"))
    |> Stream.into(File.stream!(path_output))
    |> Stream.run()
    

    View Slide

35. Task.async_stream/3
    Emits a Stream which runs the given function once for
    each element in the Enumerable
    With options: max_concurrency, ordered
    Executed with Stream.run/1
    

    View Slide

36. Task.async_stream/3
    1..100
    |> Stream.take(10)
    |> Task.async_stream(&IO.puts(to_string(&1)))
    |> Stream.run()
    

    View Slide

37. Plug.Conn.chunk/2
    Sends the response to the client incrementally
    Requires a connection which has been configured with
    send_chunked/2
    Leaves enumeration to the programmer
    

    View Slide

38. Plug.Conn.chunk/2
    chunked_conn = conn
    |> put_resp_content_type(type)
    |> send_chunked(200)
    

    View Slide

39. Plug.Conn.chunk/2
    Enum.reduce_while(enum, conn, fn (chunk, conn) ->
    case Plug.Conn.chunk(conn, chunk) do
    {:ok, conn} -> {:cont, conn}
    {:error, :closed} -> {:halt, conn}
    end
    end)
    

    View Slide

40. Ecto.Repo.stream/2
    Returns all matching entries from the database
    Lazily enumerated but may require a transaction
    

    View Slide

41. Ecto.Repo.stream/2
    Repo.transaction(fn ->
    query = from p in Post, select: p.title
    stream = Repo.stream()
    end)
    

    View Slide

42. 4
    Projection
    

    View Slide

43. View Slide

44. StreamData
    Provides value generators in the form of Streams
    Can be used for data generation
    Very helpful for property-based testing
    

    View Slide

45. StreamData
    StreamData.integer()
    |> Stream.map(&abs/1)
    |> Enum.take(3)
    #=> [1, 0, 2]
    

    View Slide

46. StreamData
    use ExUnitProperties
    property "bin1 <> bin2 always starts with bin1" do
    check all bin1 <- binary(), bin2 <- binary() do
    assert String.starts_with?(bin1 <> bin2, bin1)
    end
    end
    

    View Slide

47. View Slide

48. Packmatic
    Generates a Zip64 archive stream
    Comes with Plug integration to enable fast download
    Added refinements based on prior community work
    

    View Slide

49. Packmatic
    entries = [
    [source: {:file, "/tmp/foo.pdf"}, path: "foo/bar.pdf"],
    [source: {:url, "https://example.com/baz.pdf"}, path:
    "baz.pdf"]
    ]
    Packmatic.build_stream(entries)
    |> Packmatic.Conn.send_chunked(conn, "download.zip")
    

    View Slide

50. 5
    Conclusion
    

    View Slide

51. Why Use Streams?
    Minimise latency: Reduce time-to-first-byte and
    minimise jank risk (system saturation)
    Reduce resource usage: Eliminate peaks and avoid
    wasted work
    Reduce complexity: Leverage composition for shorter
    and more succinct programs
    

    View Slide