Elixir GenStage & Flow

Transcript

1. 1.

   GenStage & Flow Poznań Elixir Meetup 25.05.2017

2. 2.

   None
3. 3.

   Hi! Tymon Tobolski • GitHub: teamon • Twitter: @iteamon •

   Blog: teamon.eu • 8+ years with Ruby, 2+ years with Elixir • Currently Elixir+Dev+Ops @ Recruitee.com • Hex: tesla, mix_docker
4. 4.

   None
5. 5.

   The Job

6. 6.

   Library defmodule Poz do def select do items = Enum.to_list

   (1..100) :timer.sleep(:rand.uniform(100)) Progress.incr(:select, length(items)) items end def download(record) do :timer.sleep(:rand.uniform(100)) Progress.incr(:download) {:file, record} end
7. 7.

   Library cont. def extract(file) do :timer.sleep(:rand.uniform(10)) Progress.incr(:extract) {:text, file} end

   def index(texts) do :timer.sleep(:rand.uniform(1000)) Progress.incr(:index, length(texts)) :ok end end
8. 8.

   Example blueprint defmodule Example do import Poz def run do

   Progress.start_link([:select, :download, :extract, :index]) # work work work Progress.stop() end end
9. 9.

   Example 1 - Enum select() |> Enum.map(&download/1) |> Enum.map(&extract/1) |>

   index()
10. 10.

    Example 1 - Enum

11. 11.

    Example 2 - Stream select() |> Stream.concat([]) |> Stream.map(&download/1) |>

    Stream.map(&extract/1) |> Enum.to_list() |> index()
12. 12.

    Example 2 - Stream

13. 13.

    Example 3 - Task.async select() |> Enum.map(fn e -> Task.async(Poz,

    :download, [e]) end) |> Enum.map(&Task.await/1) |> Enum.map(fn e -> Task.async(Poz, :extract, [e]) end) |> Enum.map(&Task.await/1) |> index()
14. 14.

    Example 3 - Task.async

15. 15.

    Stages

16. 16.

    GenStage explained

17. 17.

    Example 4 - GenStage - SELECT defmodule Select do use

    GenStage def init(_), do: {:producer, :ok} def handle_demand(demand, :ok) do items = Poz.select() {:noreply, items, :empty} end def handle_demand(_demand, :empty) do {:noreply, [], :empty} end end
18. 18.

    Example 4 - GenStage - DOWNLOAD defmodule Download do use

    GenStage def init(_), do: {:producer_consumer, :ok} def handle_events(items, _from, state) do files = Enum.map(items, &Poz.download/1) {:noreply, files, state} end end
19. 19.

    Example 4 - GenStage - EXTRACT defmodule Extract do use

    GenStage def init(_), do: {:producer_consumer, :ok} def handle_events(files, _from, state) do texts = Enum.map(files, &Poz.extract/1) {:noreply, files, texts} end end
20. 20.

    Example 4 - GenStage - INDEX defmodule Index do use

    GenStage def init(_), do: {:consumer, :ok} def handle_events(texts, _from, state) do Poz.index(texts) {:noreply, [], state} end end
21. 21.

    Example 4 - GenStage {:ok, select} = GenStage.start_link(Select, :ok) {:ok,

    download} = GenStage.start_link(Download, :ok) {:ok, extract} = GenStage.start_link(Extract, :ok) {:ok, index} = GenStage.start_link(Index, :ok) GenStage.sync_subscribe(download, to: select) GenStage.sync_subscribe(extract, to: download) GenStage.sync_subscribe(index, to: extract) :timer.sleep(:infinity)
22. 22.

    Example 4 - GenStage (defaults)

23. 23.

    Example 5 - GenStage (custom demand) {:ok, select} = GenStage.start_link(Select,

    :ok) {:ok, download} = GenStage.start_link(Download, :ok) {:ok, extract} = GenStage.start_link(Extract, :ok) {:ok, index} = GenStage.start_link(Index, :ok) GenStage.sync_subscribe(download, to: select, max_demand: 20) GenStage.sync_subscribe(extract, to: download, max_demand: 20) GenStage.sync_subscribe(index, to: extract, max_demand: 100) :timer.sleep(:infinity)
24. 24.

    Example 5 - GenStage (tuned demand)

25. 25.

    Example 6 - GenStage (multiprocess) {:ok, select} = GenStage.start_link(Select, :ok)

    {:ok, extract} = GenStage.start_link(Extract, :ok) {:ok, index} = GenStage.start_link(Index, :ok) for i <- (1..10) do {:ok, download} = GenStage.start_link(Download, i) GenStage.sync_subscribe(download, to: select, max_demand: 20) GenStage.sync_subscribe(extract, to: download, max_demand: 20) end GenStage.sync_subscribe(index, to: extract, max_demand: 100)
26. 26.

    None
27. 27.

    Example 7 - Flow select() |> Flow.from_enumerable(max_demand: 20) |> Flow.partition(max_demand:

    20, stages: 5) |> Flow.map(&download/1) |> Flow.partition(max_demand: 20, stages: 2) |> Flow.map(&extract/1) |> Flow.partition(window: Flow.Window.count(50), stages: 1) |> Flow.reduce(fn -> [] end, fn item, list -> [item | list] end) |> Flow.emit(:state) |> Flow.partition(stages: 2) |> Flow.map(&index/1) |> Flow.run()
28. 28.

    Example 7 - Flow

29. 29.

    Real World

30. 30.

    Real World - optimised

31. 31.

    References • https:/ /github.com/teamon/poz • https:/ /elixir-lang.org/blog/2016/07/14/announcing-genstage/ • https:/ /hexdocs.pm/gen_stage/GenStage.html

    • https:/ /hexdocs.pm/flow/Flow.html • http:/ /teamon.eu/2016/tuning-elixir-genstage-flow-pipeline- processing/ • http:/ /teamon.eu/2016/measuring-visualizing-genstage-flow- with-gnuplot/
32. 32.

    Thanks! Questions?