ElixirConf 2019 - Build Efficient Data Processing Pipelines in Elixir Using Broadway

Announced in February 2019, Broadway is a new open-source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir. It allows developers to build complex GenStage topologies that can consume and process data efficiently from different sources, including Amazon SQS and RabbitMQ. In this talk, we'll discuss some of the main concepts behind Broadway, how it leverages OTP to achieve fault-tolerance, sharing implementation details and architectural aspects. You'll learn how to build efficient data processing pipelines and how to optimize them using real-time metrics based on Telemetry events.

Transcript

1. Broadway Building efficient data pipelines in Elixir Marlus Saraiva @msaraiva @MarlusSaraiva
2. Agenda • Why Broadway? • How does it work? • Implementation details • Fault tolerance • Graceful shutdown • Telemetry integration
3. Broadway An open source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir
4. Data pipelines “A set of data processing elements connected in series, where the output of one element is the input of the next one” - Wikipedia
5. Data pipelines (example) User API SQS S3
6. Data pipelines (example) User API SQS S3
7. User API SQS S3 Broadway Data pipelines (example)
8. Why Broadway? Many companies are using Elixir to building data processing pipelines
9. Why Broadway? • Reimplementing the same features • Assembling complex GenStage topologies • Running into the same common pitfalls Many companies are using Elixir to building data processing pipelines
10. Desired Features • Concurrency • Back-pressure • Batching • Graceful shutdowns • Consume data from different sources like SQS, RabbitMQ, … • More…
11. Desired Features • Concurrency • Back-pressure • Batching • Graceful shutdowns • Consume data from different sources like SQS, RabbitMQ, … • More…
12. Desired Features • Concurrency • Back-pressure • Batching • Graceful shutdowns • Consume data from different sources like SQS, RabbitMQ, … • More…
13. Desired Features • Concurrency • Back-pressure • Batching • Graceful shutdowns • Consume data from different sources like SQS, RabbitMQ, … • More…
14. Desired Features • Concurrency • Back-pressure • Batching • Graceful shutdowns • Consume data from different sources like SQS, RabbitMQ, … • More…
15. Complex GenStage pipelines • How to define the right topology for the pipeline? • How to structure the supervision tree correctly? • How to handle graceful shutdown without data loss? ?
16. Complex GenStage pipelines • How to define the right topology for the pipeline? • How to structure the supervision tree correctly? • How to handle graceful shutdown without data loss? ?
17. Broadway Configuration def handle_message(…) do ... end def handle_batch(…) do ... end Behaviour + = Pipeline
18. Broadway defmodule MyBroadway do use Broadway def start_link(_opts) do Broadway.start_link(MyBroadway, producers: [...], processors: [...], batchers: [...] ) end def handle_message(...) do ... end def handle_batch(...) do ... end end Behaviour
19. Broadway defmodule MyBroadway do use Broadway def start_link(_opts) do Broadway.start_link(MyBroadway, producers: [...], processors: [...], batchers: [...] ) end def handle_message(...) do ... end def handle_batch(...) do ... end end Configuration
20. Broadway defmodule MyBroadway do use Broadway def start_link(_opts) do Broadway.start_link(MyBroadway, producers: [...], processors: [...], batchers: [...] ) end def handle_message(...) do ... end def handle_batch(...) do ... end end Callbacks
21. Configuration (defines the topology) producers: [ default: [ module: {Counter, []}, stages: 2 ] ], processors: [ default: [stages: 3] ] Producers Processors
22. producers: [ default: [ module: {Counter, []}, stages: 2 ] ], processors: [ default: [stages: 3] ] Producers Processors Configuration (defines the topology)
23. producers: [ default: [ module: {Counter, []}, stages: 1 ] ], processors: [ default: [stages: 2] ], batchers: [ sqs: [stages: 2], s3: [stages: 1] ] S3 SQS Producers Processors Batchers Batch Processors Configuration (defines the topology)
24. producers: [ default: [ module: {Counter, []}, stages: 1 ] ], processors: [ default: [stages: 2] ], batchers: [ sqs: [stages: 2], s3: [stages: 1] ] S3 SQS Producers Processors Batchers Batch Processors Configuration (defines the topology)
25. producers: [ default: [ module: {Counter, []}, stages: 1 ] ], processors: [ default: [stages: 2] ], batchers: [ sqs: [stages: 2], s3: [stages: 1] ] S3 SQS Producers Processors Batchers Batch Processors Configuration (defines the topology)
26. producers: [ default: [ module: {Counter, []}, stages: 1 ] ], processors: [ default: [stages: 2] ], batchers: [ sqs: [stages: 2], s3: [stages: 1] ] S3 SQS Producers Processors Batchers Batch Processors Configuration (defines the topology)
27. Broadway (Configuration) DEMO
28. Callbacks (define the custom code that runs inside the pipeline) def handle_message(...) do ... if sqs?(message) do message |> Message.put_batcher(:sqs) else message |> Message.put_batcher(:s3) end end def handle_batch(:sqs, ...) do ... end def handle_batch(:s3, ...) do ... end S3 SQS Producers Processors Batchers Batch Processors
29. Callbacks (define the custom code that runs inside the pipeline) def handle_message(...) do ... if sqs?(message) do message |> Message.put_batcher(:sqs) else message |> Message.put_batcher(:s3) end end def handle_batch(:sqs, ...) do ... end def handle_batch(:s3, ...) do ... end S3 SQS Producers Processors Batchers Batch Processors
30. def handle_message(...) do ... if sqs?(message) do message |> Message.put_batcher(:sqs) else message |> Message.put_batcher(:s3) end end def handle_batch(:sqs, ...) do ... end def handle_batch(:s3, ...) do ... end S3 SQS Callbacks (define the custom code that runs inside the pipeline) Producers Processors Batchers Batch Processors
31. def handle_message(...) do ... if sqs?(message) do message |> Message.put_batcher(:sqs) else message |> Message.put_batcher(:s3) end end def handle_batch(:sqs, ...) do ... end def handle_batch(:s3, ...) do ... end S3 SQS Callbacks (define the custom code that runs inside the pipeline) Producers Processors Batchers Batch Processors
32. def handle_message(...) do ... if sqs?(message) do message |> Message.put_batcher(:sqs) else message |> Message.put_batcher(:s3) end end def handle_batch(:sqs, ...) do ... end def handle_batch(:s3, ...) do ... end S3 SQS Callbacks (define the custom code that runs inside the pipeline) Producers Processors Batchers Batch Processors
33. def handle_message(...) do ... if sqs?(message) do message |> Message.put_batcher(:sqs) else message |> Message.put_batcher(:s3) end end def handle_batch(:sqs, ...) do ... end def handle_batch(:s3, ...) do ... end S3 SQS Callbacks (define the custom code that runs inside the pipeline) Producers Processors Batchers Batch Processors
34. Broadway (callbacks) DEMO
35. Complex GenStage pipelines • How to define the right topology for the pipeline? • How to structure the supervision tree correctly? • How to handle graceful shutdown without data loss? ?
36. Complex GenStage pipelines • How to define the right topology for the pipeline? • How to structure the supervision tree correctly? • How to handle graceful shutdown without data loss? ?
37. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
38. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
39. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
40. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
41. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
42. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
43. ProducerSupervisor :one_for_one Terminator Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
44. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
45. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
46. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision
47. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
48. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
49. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
50. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
51. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
52. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
53. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
54. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
55. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2
56. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
57. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
58. ProducerSupervisor :one_for_one Terminator Producer_1 Processor_2 Processor_1 Consumer_1 Batcher Consumer_2 Batcher Consumer_1 Broadway Pipeline Supervisor :rest_for_one Broadway GenServer ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one ConsumerSupervisor :one_for_all ConsumerSupervisor :one_for_all BatcherConsumerSuperv_1 :rest_for_one BatcherConsumerSuperv_2 :rest_for_one Supervision Producer_2 ( max_restarts = 0 )
59. Complex GenStage pipelines ? • How to define the right topology for the pipeline? • How to structure supervisions trees correctly? • How to handle graceful shutdown without data loss?
60. Complex GenStage pipelines ? • How to define the right topology for the pipeline? • How to structure supervisions trees correctly? • How to handle graceful shutdown without data loss?
61. Graceful Shutdown ProducerSupervisor :one_for_one Terminator Broadway Pipeline Supervisor :rest_for_one ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one
62. Producers Processors Batchers Batch Processors Graceful Shutdown Step 1: Tell processors to stop subscribing to producers Messages Terminator
63. Producers Processors Batchers Batch Processors Graceful Shutdown Step 1: Tell processors to stop subscribing to producers Messages Terminator
64. Producers Processors Batchers Batch Processors Graceful Shutdown Step 1: Tell processors to stop subscribing to producers Messages Terminator
65. Producers Processors Batchers Batch Processors Graceful Shutdown Messages Terminator Step 2: Cancel and shutdown producers • Tell producers to: • Stop receiving messages • Stop accepting demand • Flush all events in the buffer • Cancel consumers
66. Processors Batchers Batch Processors Graceful Shutdown Step 3: Monitors consumers and wait until they’re dead Terminator monitor and wait Producers Messages
67. Processors Batchers Batch Processors Graceful Shutdown Step 3: Monitors consumers and wait until they’re dead Terminator monitor and wait Producers Messages
68. Batchers Batch Processors Graceful Shutdown Step 3: Monitors consumers and wait until they’re dead Terminator monitor and wait Producers Messages
69. Batch Processors Graceful Shutdown Step 3: Monitors consumers and wait until they’re dead Terminator monitor and wait Producers Messages
70. Graceful Shutdown Step 3: Monitors consumers and wait until they’re dead Terminator Producers Messages
71. Graceful Shutdown Step 3: Monitors consumers and wait until they’re dead Producers Messages
72. Graceful Shutdown ProducerSupervisor :one_for_one Terminator Broadway Pipeline Supervisor :rest_for_one ProcessorSupervisor :one_for_all BatcherPartitionSupervisor :one_for_one
73. DEMO Graceful Shutdown
74. Complex GenStage pipelines ? • How to define the right topology for the pipeline? • How to structure supervisions trees correctly? • How to handle graceful shutdown without data loss?
75. Metrics and Statistics :telemetry “If You Can't Measure It, You Can't Improve It!”
76. Metrics and Statistics DEMO
77. Whats next? •Metrics and Statistics (WIP) •Multiple processors •Kafka connector (WIP) •Rate limiter •More…
78. @msaraiva @MarlusSaraiva Thank you! https://github.com/plataformatec/broadway