A brief introduction to GenStage

Transcript

1. A brief introduction to GenStage A mechanism to build an on-demand data processing pipeline
2. Let’s begin with a story
3. Alice Bob Clair
4. Donut store
5. Making
6. Baking
7. Packing
10. Until one day…
12. Slower
20. Slower
21. They come up with an idea…
23. Hey, I want a donut
24. Bob, one donut, plz.
25. Alice, one donut, plz
26. Ok, here you are
28. Baking
29. Clair, take this one
30. Let me pack it!
33. Revisit the story
34. Initial workflow
38. Slower
39. How they improve it?
45. Just a story?
46. Two ways to process data
47. 1. Batch
48. Batch
49. Batch • Each step works as an independent unit • Data are processed in a batch (ex. every 5 min) • Producer decides the amount of data passed to consumer • Slower Consumer will overload the system
50. 2. On-Demand
51. On-Demand
52. On-Demand • Data are demand from the consumer • Producer passes demanded amount of data to consumer • Consumer controls the processing rate of the pipeline
53. GenStage https:/ /github.com/elixir-lang/gen_stage
54. GenStage • Exchanging events with back-pressure • Provide On-Demand mechanism • Supports multiple Dispatchers
55. Demand-driven
56. A B Subscribes Producer Consumer Asks demands Sends events GenStage: Demand-driven
57. A B Producer ProducerConsumer Asks demands Sends events C Consumer Asks demands Sends events GenStage: Demand-driven
58. Dispatcher
59. A 1 Producer 2, 5 3 4 DemandDisaptcher 1, 2, 3, 4, 5
60. BroadcastDispatcher A 1, 2 Producer 1, 2 1, 2 1, 2 1, 2
61. PartitionDispatcher A 1, 5 Producer 2, 6 3 4 1, 2, 3, 4, 5, 6 rem(x, 4)
62. Use cases
63. Use cases • Data Transformation Pipeline — We could produce events from a database or even another source like Apache’s Kafka. Process, sort, catalog, and store metrics as they become available. • Work Queue — We could produce works of unit to be completed by a series of consumers. • Event Processing — Similar to a data pipeline, we could receive, process, sort, and take action on events emitted in real time from our sources.
64. Let’s build a donut store https:/ /github.com/linjunpop/donut_store
65. defmodule DonutStore.Staff.Alice do alias DonutStore.Donut def make_donuts(n) do Stream.repeatedly(&Donut.build/0) |> Enum.take(n) end end Staff: Alice
66. defmodule DonutStore.Staff.Bob do alias DonutStore.Donut def bake_donut(%Donut{} = donut, color) do %{donut | color: color} end end Staff: Bob
67. defmodule DonutStore.Staff.Clair do alias DonutStore.Donut alias DonutStore.Package def pack_donut(%Donut{} = donut) do %Package{donut: donut} end end Staff: Clair
68. The workflow a.k.a Stages
69. defmodule DonutStore.Workflow.Making do use GenStage require Logger alias DonutStore.Staff.Alice def start_link(args) do GenStage.start_link(__MODULE__, args, name: __MODULE__) end @impl GenStage def init(_state) do {:producer, %{events: []}} end def handle_demand(demand, state) when is_integer(demand) and demand > 0 do events = Alice.make_donuts(demand) {:noreply, events, state} end end Workflow: Making
70. defmodule DonutStore.Workflow.Baking do use GenStage require Logger alias DonutStore.Staff.Bob alias DonutStore.Workflow.Making def start_link(args) do GenStage.start_link(__MODULE__, args, name: __MODULE__) end def init(_) do {:producer_consumer, :state, subscribe_to: [{Making, min_demand: 2, max_demand: 20}]} end def handle_events(events, _from, state) do events = events |> Enum.map(&Bob.bake_donut(&1, :yellow)) {:noreply, events, state} end end Workflow: Baking
71. defmodule DonutStore.Workflow.Packing do use GenStage alias DonutStore.Workflow.Baking alias DonutStore.Staff.Clair def start_link(args) do GenStage.start_link(__MODULE__, args, name: __MODULE__) end def init(_) do {:consumer, :state, subscribe_to: [{Baking, min_demand: 2, max_demand: 5}]} end def handle_events(events, _from, state) do events |> Enum.map(&Clair.pack_donut/1) {:noreply, [], state} end end Workflow: Packing
72. defmodule DonutStore.Workflow do use Supervisor alias DonutStore.Workflow.Making alias DonutStore.Workflow.Baking alias DonutStore.Workflow.Packing def start_link(arg) do Supervisor.start_link(__MODULE__, arg, name: __MODULE__) end @impl Supervisor def init(_arg) do children = [ {Making, []}, {Baking, []}, {Packing, []} ] Supervisor.init(children, strategy: :one_for_one) end end Workflow: Supervisor
73. Demo https:/ /github.com/linjunpop/donut_store
74. Recap
75. GenStage • Exchanging events with back-pressure • Provide On-Demand mechanism • Supports multiple Dispatchers
76. Archiver
77. Archiver A 1, 2 Producer 1, 2 1, 2 1, 2 items Google Drive Box Dropbox One Drive
78. Archiver Producer Google Drive items Box Text processor Video processor
79. Q&A
80. Java • Akka Stream https:/ /doc.akka.io/docs/akka/2.5/stream/ • RxJava https:/ /github.com/ReactiveX/RxJava/wiki/Backpressure
81. Thanks
82. Credits • http:/ /openmoji.org • https:/ /elixirschool.com • https:/ /elixir-lang.org