Stream Driven Development - Design your data pipeline with Akka Streams

Reactive Streams are the key to build asynchronous, data-intensive applications with no predetermined data volumes. Akka Streams is a well-known Reactive Streams implementation offering a powerful DSL to build complex, Akka-backed reactive pipelines.

At HomeAway we devised an approach to roll out reactive data pipelines, by combining elements of Domain-Driven Design with the abstraction power of the Akka Streams model.

In this talk we’ll

- briefly introduce Reactive Streams and Akka Streams
- present our approach to pipeline design by example, discussing useful patterns to reason about your streaming application and type-drive its implementation

Transcript

1. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. CODEMOTION MADRID 2017
2. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STREAM DRIVEN DEVELOPMENT Design your data pipeline with Akka Streams
3. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz
4. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 4 EXAMPLE PROBLEM
5. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 5 EXAMPLE PROBLEM
6. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 6
7. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 7 AD PLATFORMS
8. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 8
9. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 9 Pricing Images Property Details ...
10. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 10
11. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 11 RESILIENT
12. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 12 RESILIENT SCALABLE
13. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 13 RESILIENT SCALABLE REACTIVE
14. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 14 EXAMPLE PROBLEM
15. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 15 PUBLISHER SUBSCRIBER
16. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 16 PUBLISHER SUBSCRIBER PUSH
17. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 17 PUBLISHER SUBSCRIBER PUSH
18. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 18 PUSH PUBLISHER SUBSCRIBER
19. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 19 PUSH PUBLISHER SUBSCRIBER
20. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 20 PUBLISHER SUBSCRIBER SOLUTION 1 pull()
21. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 21 SOLUTION 2 PUBLISHER SUBSCRIBER
22. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 22 SOLUTION 3 PUBLISHER SUBSCRIBER
23. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 23 SOLUTION 3 PUBLISHER SUBSCRIBER
24. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 24 SOLUTION 3 PUBLISHER SUBSCRIBER
25. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 25 REACTIVE STREAMS PUBLISHER SUBSCRIBER requests events
26. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 26 REACTIVE STREAMS request(2) PUBLISHER SUBSCRIBER
27. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 27 REACTIVE STREAMS PUBLISHER SUBSCRIBER
28. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 28 REACTIVE STREAMS STOP PUBLISHER SUBSCRIBER
29. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. package org.reactivestreams; public interface Processor<T, R> extends Subscriber<T>, Publisher<R> { } public interface Publisher<T> { public void subscribe(Subscriber<? super T> s); } public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete(); } public interface Subscription { public void request(long n); public void cancel(); } 29
30. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 30 WHY REACTIVE STREAMS • flow control with asynchronous backpressure • interoperability between tools and libraries
31. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 31 WHY REACTIVE STREAMS • flow control with asynchronous backpressure • interoperability between tools and libraries
32. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 32 EXAMPLE PROBLEM
33. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. package org.reactivestreams; public interface Processor<T, R> extends Subscriber<T>, Publisher<R> { } public interface Publisher<T> { public void subscribe(Subscriber<? super T> s); } public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete(); } public interface Subscription { public void request(long n); public void cancel(); } 33
34. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 34 IMPLEMENTATIONS REACTOR MONIX
35. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 35 IMPLEMENTATIONS
36. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 36 WHY AKKA STREAMS • higher-level DSL • based on akka actor • Scala native with Java and Scala DSLs • 3rd party integrations / connectors
37. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 37 WHY AKKA STREAMS • higher-level DSL • based on akka actor • Scala native with Java and Scala DSLs • 3rd party integrations / connectors
38. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 38 WHY AKKA STREAMS • higher-level DSL • based on akka actor • Scala native with Java and Scala DSLs • 3rd party integrations / connectors
39. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 39 WHY AKKA STREAMS • higher-level DSL • based on akka actor • Scala native with Java and Scala DSLs • 3rd party integrations / connectors
40. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 40 SOURCE (1 output) FAN-IN (n inputs, 1 output) FAN-OUT (1 input, n outputs) RUNNABLEGRAPH (no input or output) STREAMS STAGES FLOW (1 input, 1 output) SINK (1 input) ... CUSTOM
41. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 41 Flow RunnableGraph Source Sink
42. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 42 Flow[In, Out, Mat]
43. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 43 Flow[In, Out, Mat] PORTS
44. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 44 Flow[In, Out, Mat] MATERIALIZED VALUE
45. val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS
46. val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS
47. val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS
48. val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS
49. val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) val eventualCompletion: Future[Done] = source.via(flow).runWith(sink) eventualCompletion.onComplete { case Success(_) ⇒ println("Stream completed!") case Failure(ex) ⇒ println(s"Stream failed with error $ex") } AKKA STREAMS
50. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 50 EXAMPLE PROBLEM
51. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 51 PROCESSING
52. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 52 PROCESSING
54. Source[PropertyChange, NotUsed]
55. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed]
56. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] Sink[ValidationError, NotUsed]
57. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
58. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
59. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
60. def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE
61. def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA
62. def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA
63. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
64. def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val service: PropertyProcessingService = ??? Flow.fromFunction(service.process) } PROCESSING FLOW
65. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
66. STORING SINK def adwordsSink(config: AdwordsConfig): Sink[AdwordsChange, Future[Done]] = { val service: AdwordsService = ??? Flow[AdwordsChange].mapAsync(config.parallelism)(service.store) .toMat(Sink.ignore)(Keep.right) }
67. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
68. ERROR SINK def errorSink(cfg: ErrorConfig): Sink[ValidationError, NotUsed] = { val service: AdwordsErrorService = ??? Sink.foreach[ValidationError](service.handle) .mapMaterializedValue(_ ⇒ NotUsed) }
69. Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]
70. GRAPH def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Sink[AdwordsChange, Future[Done]], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { RunnableGraph.fromGraph(GraphDSL.create(store) { implicit builder ⇒ store ⇒ val p = builder.add(PartitionEither.apply()[ValidationError, AdwordsChange]) p.out0 ~> errorSink source ~> process ~> p.in p.out1 ~> store ClosedShape }) }
71. GRAPH def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Sink[AdwordsChange, Future[Done]], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { RunnableGraph.fromGraph(GraphDSL.create(store) { implicit builder ⇒ store ⇒ val p = builder.add(PartitionEither.apply()[ValidationError, AdwordsChange]) p.out0 ~> errorSink source ~> process ~> p.in p.out1 ~> store ClosedShape }) }
72. def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Sink[AdwordsChange, Future[Done]], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { RunnableGraph.fromGraph(GraphDSL.create(store) { implicit builder ⇒ store ⇒ val p = builder.add(PartitionEither.apply()[ValidationError, AdwordsChange]) p.out0 ~> errorSink source ~> process ~> p.in p.out1 ~> store ClosedShape }) } GRAPH
73. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 73 PropertyProcessingService def process(p: PropertyChange): Either[ValidationError, AdwordsChange] PropertyChange AdwordsService def store(p: AdwordsChange): Future[Either[StorageError, Stored]] DOMAIN ValidationError AdwordsErrorService def handle(p: AdwordsError): Unit AdwordsChange
74. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 74 APPLICATION Failure Handling Config Management Materialization Stage Creation
75. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 75 Application layer Graph Events Services Domain layer Repositories Factories Sources Flows Sinks Other Reactive layer
76. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. 76 Reactive layer Application layer Domain layer Reactive layer Application layer Domain layer Reactive layer Application layer Domain layer Reactive layer Application layer Domain layer
77. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 77 EXAMPLE PROBLEM
78. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 78 EXAMPLE PROBLEM
79. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 79 EXAMPLE PROBLEM
80. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 80 EXAMPLE PROBLEM
81. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 81 EXAMPLE PROBLEM
82. © 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz Thank you. homeaway.com/careers