Stream Driven Development - Design your data pipeline with Akka Streams

Akka Streams is a well-known Reactive Streams implementation that helps you build asynchronous, data-intensive applications with no predetermined data volumes. But how would you leverage its full power to design complex, Akka-backed reactive pipelines? At HomeAway.com we devised an approach to tackle this problem, combining elements of Domain-Driven Design with the abstraction power of the Akka Streams model. In this talk we’ll present our approach by example, discussing useful patterns to

- reason about your streaming application and identify its building blocks
- type-drive the implementation
- handle failure
- instrument your application for logging and monitoring purposes

Transcript

1. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE SUMMIT 2017
2. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STREAM DRIVEN DEVELOPMENT Design your data pipeline with Akka Streams
3. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 3 © 2017 IKEA
4. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 4 © RonTech2000/iStock
5. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz
6. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. THE PROBLEM 6
7. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 7
8. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 8 AD PLATFORMS
9. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 9
10. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 10 Pricing Images Property Details ...
11. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 11
12. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 12 RESILIENT
13. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 13 RESILIENT SCALABLE
14. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 14 RESILIENT SCALABLE REACTIVE
15. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 15 RESILIENT SCALABLE streams
16. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 16 PROCESSING
17. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 17 PROCESSING
18. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. THE APPROACH 18
19. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 19 STAGES TYPES CODE
20. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. TYPES 20 STAGES CODE
21. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 21 SOURCE (1 output) FAN-IN (n inputs, 1 output) FAN-OUT (1 input, n outputs) RUNNABLEGRAPH (no input or output) STREAMS STAGES FLOW (1 input, 1 output) SINK (1 input) ... CUSTOM
22. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 22 Source ~> Flow = Source Flow ~> Sink = Sink Flow ~> Flow = Flow Broadcast ~> Merge = Flow Source ~> Sink = RunnableGraph Source ~> Flow ~> Sink = RunnableGraph STAGE ARITHMETIC
23. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 23
24. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 24 Source Sink
25. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 25 Flow Source Sink
26. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 26 Flow RunnableGraph Source Sink
27. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 27 TYPES STAGES CODE 27
28. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 28 Flow[In, Out, Mat]
29. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 29 Flow[In, Out, Mat] PORTS
30. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 30 Flow[In, Out, Mat] MATERIALIZED VALUE
31. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 31 Flow RunnableGraph Source Sink
32. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 32 Flow RunnableGraph Source[PropertyChange, NotUsed] Sink
33. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 33 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph Source[PropertyChange, NotUsed] Sink
34. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 34 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]]
35. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 35 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph[Future[Done]] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]]
36. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 36 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph[Future[Done]] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]]
37. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 37 SIDE EFFECTS Error Handling Logging Monitoring ...
38. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 38 SIDE EFFECTS Error Handling Logging Monitoring EVENTS Errors Audit ... ...
39. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 39 SIDE EFFECTS Error Handling Logging Monitoring EVENTS Errors Audit ... ... REFERENTIALLY TRANSPARENT STAGES Flow[In, Either[Error, Out], Mat] Flow[In, (Seq[Audit], Out), Mat] ...
40. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
41. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]
42. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. CODE TYPES 42 STAGES 42
43. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]
44. def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE
45. def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA
46. def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA
47. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]
48. def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val service: PropertyProcessingService = ??? Flow.fromFunction(service.process) } PROCESSING FLOW
49. def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val validationService : PropertyValidationService = ??? val transformationService: PropertyTransformationService = ??? Flow.fromFunction(validationService.validate) .map(transformationService.transform) } PROCESSING FLOW
50. def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val validationService : PropertyValidationService = ??? val transformationService: PropertyTransformationService = ??? Flow.fromFunction(validationService.validate) .async .map(transformationService.transform) } PROCESSING FLOW
51. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]
52. STORING FLOW def adwordsFlow(config: AdwordsConfig): Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] = { val service: AdwordsService = ??? Flow[AdwordsChange].mapAsync(config.parallelism)(service.store) }
53. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]
54. ERROR SINK def errorSink(cfg: ErrorConfig): Sink[AdwordsStreamError, NotUsed] = { val service: AdwordsErrorService = ??? Sink.foreach[AdwordsStreamError](service.handle) .mapMaterializedValue(_ ⇒ NotUsed) }
55. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]
56. def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH
57. def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH
58. def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH
59. def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH
60. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 60 Flow[Either[E,T], T, M] Sink[E, M] Either[E, T] T E DIVERT ERRORS
61. def divertErrors[T, E, M](to: Sink[E, M]): Flow[Either[E, T], T, M] = { Flow.fromGraph(GraphDSL.create(to) { implicit b ⇒ sink ⇒ val partition = b.add(Partition[Either[E, T]](2, _.fold(_ ⇒ 0, _ ⇒ 1))) val left = b.add(Flow[Either[E, T]].map (_.left.get)) val right = b.add(Flow[Either[E, T]].map (_.right.get)) partition ~> left ~> sink partition ~> right FlowShape(partition.in, right.out) }) } GRAPH - DIVERT ERRORS
62. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 62 PropertyProcessingService def process(p: PropertyChange): Either[ValidationError, AdwordsChange] PropertyChange AdwordsService def store(p: AdwordsChange): Future[Either[StorageError, Stored]] DOMAIN ValidationError AdwordsChange Stored StorageError AdwordsErrorService def handle(p: AdwordsError): Unit
63. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 63 APPLICATION Failure Handling Config Management Materialization Stage Creation
64. val cfg: ApplicationConfig = ??? val source = propertySource(cfg.kafka) val pFlow = processingFlow() val sFlow = adwordsFlow(cfg.adwords) val eSink = errorSink(cfg.error) val graph = AdwordsGraph.graph(source, pFlow, sFlow, eSink) graph.run() APPLICATION
65. final case class ApplicationConfig(kafka : KafkaConfig, adwords : AdwordsConfig, error : ErrorConfig) APPLICATION - CONFIGURATION
66. kafka { bootstrapServers = "my.kafka" consumerGroup = "my-consumer" topic = "properties" } adwords { parallelism = 5 endpoint = "http://adwords.google.com/properties" } error { loggerName = "adwordsErrors" } APPLICATION - CONFIGURATION
67. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 67 Application layer Graph Events Services Domain layer Repositories Factories Sources Flows Sinks Other Reactive layer
68. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 68 STAGES TYPES CODE 68
69. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. TYPES 69 STAGES CODE 69
70. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 70 TYPES STAGES CODE 70
71. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. CODE TYPES 71 STAGES 71
72. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 72 STAGES TYPES CODE 72
73. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz Thank you.