Tweet
Tweet

Slide 1

Slide 1 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. CODEMOTION MADRID 2017

Slide 2

Slide 2 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STREAM DRIVEN DEVELOPMENT Design your data pipeline with Akka Streams

Slide 3

Slide 3 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz

Slide 4

Slide 4 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 4 EXAMPLE PROBLEM

Slide 5

Slide 5 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 5 EXAMPLE PROBLEM

Slide 6

Slide 6 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 6

Slide 7

Slide 7 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 7 AD PLATFORMS

Slide 8

Slide 8 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 8

Slide 9

Slide 9 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 9 Pricing Images Property Details ...

Slide 10

Slide 10 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 10

Slide 11

Slide 11 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 11 RESILIENT

Slide 12

Slide 12 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 12 RESILIENT SCALABLE

Slide 13

Slide 13 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 13 RESILIENT SCALABLE REACTIVE

Slide 14

Slide 14 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 14 EXAMPLE PROBLEM

Slide 15

Slide 15 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 15 PUBLISHER SUBSCRIBER

Slide 16

Slide 16 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 16 PUBLISHER SUBSCRIBER PUSH

Slide 17

Slide 17 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 17 PUBLISHER SUBSCRIBER PUSH

Slide 18

Slide 18 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 18 PUSH PUBLISHER SUBSCRIBER

Slide 19

Slide 19 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 19 PUSH PUBLISHER SUBSCRIBER

Slide 20

Slide 20 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 20 PUBLISHER SUBSCRIBER SOLUTION 1 pull()

Slide 21

Slide 21 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 21 SOLUTION 2 PUBLISHER SUBSCRIBER

Slide 22

Slide 22 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 22 SOLUTION 3 PUBLISHER SUBSCRIBER

Slide 23

Slide 23 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 23 SOLUTION 3 PUBLISHER SUBSCRIBER

Slide 24

Slide 24 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 24 SOLUTION 3 PUBLISHER SUBSCRIBER

Slide 25

Slide 25 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 25 REACTIVE STREAMS PUBLISHER SUBSCRIBER requests events

Slide 26

Slide 26 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 26 REACTIVE STREAMS request(2) PUBLISHER SUBSCRIBER

Slide 27

Slide 27 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 27 REACTIVE STREAMS PUBLISHER SUBSCRIBER

Slide 28

Slide 28 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 28 REACTIVE STREAMS STOP PUBLISHER SUBSCRIBER

Slide 29

Slide 29 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. package org.reactivestreams; public interface Processor extends Subscriber, Publisher { } public interface Publisher { public void subscribe(Subscriber s); } public interface Subscriber { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete(); } public interface Subscription { public void request(long n); public void cancel(); } 29

Slide 30

Slide 30 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 30 WHY REACTIVE STREAMS ● flow control with asynchronous backpressure ● interoperability between tools and libraries

Slide 31

Slide 31 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 31 WHY REACTIVE STREAMS ● flow control with asynchronous backpressure ● interoperability between tools and libraries

Slide 32

Slide 32 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 32 EXAMPLE PROBLEM

Slide 33

Slide 33 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. package org.reactivestreams; public interface Processor extends Subscriber, Publisher { } public interface Publisher { public void subscribe(Subscriber s); } public interface Subscriber { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete(); } public interface Subscription { public void request(long n); public void cancel(); } 33

Slide 34

Slide 34 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 34 IMPLEMENTATIONS REACTOR MONIX

Slide 35

Slide 35 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 35 IMPLEMENTATIONS

Slide 36

Slide 36 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 36 WHY AKKA STREAMS ● higher-level DSL ● based on akka actor ● Scala native with Java and Scala DSLs ● 3rd party integrations / connectors

Slide 37

Slide 37 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 37 WHY AKKA STREAMS ● higher-level DSL ● based on akka actor ● Scala native with Java and Scala DSLs ● 3rd party integrations / connectors

Slide 38

Slide 38 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 38 WHY AKKA STREAMS ● higher-level DSL ● based on akka actor ● Scala native with Java and Scala DSLs ● 3rd party integrations / connectors

Slide 39

Slide 39 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 39 WHY AKKA STREAMS ● higher-level DSL ● based on akka actor ● Scala native with Java and Scala DSLs ● 3rd party integrations / connectors

Slide 40

Slide 40 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 40 SOURCE (1 output) FAN-IN (n inputs, 1 output) FAN-OUT (1 input, n outputs) RUNNABLEGRAPH (no input or output) STREAMS STAGES FLOW (1 input, 1 output) SINK (1 input) ... CUSTOM

Slide 41

Slide 41 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 41 Flow RunnableGraph Source Sink

Slide 42

Slide 42 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 42 Flow[In, Out, Mat]

Slide 43

Slide 43 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 43 Flow[In, Out, Mat] PORTS

Slide 44

Slide 44 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 44 Flow[In, Out, Mat] MATERIALIZED VALUE

Slide 45

Slide 45 text

val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS

Slide 46

Slide 46 text

val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS

Slide 47

Slide 47 text

val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS

Slide 48

Slide 48 text

val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) source.via(flow).runWith(sink) AKKA STREAMS

Slide 49

Slide 49 text

val source: Source[String, NotUsed] = Source.single("World") val flow : Flow[String, String, NotUsed] = Flow[String].map(x ⇒ s"Hello, $x!") val sink : Sink[String, Future[Done]] = Sink.foreach[String](println) val eventualCompletion: Future[Done] = source.via(flow).runWith(sink) eventualCompletion.onComplete { case Success(_) ⇒ println("Stream completed!") case Failure(ex) ⇒ println(s"Stream failed with error $ex") } AKKA STREAMS

Slide 50

Slide 50 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 50 EXAMPLE PROBLEM

Slide 51

Slide 51 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 51 PROCESSING

Slide 52

Slide 52 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 52 PROCESSING

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Source[PropertyChange, NotUsed]

Slide 55

Slide 55 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed]

Slide 56

Slide 56 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] Sink[ValidationError, NotUsed]

Slide 57

Slide 57 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 58

Slide 58 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 59

Slide 59 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 60

Slide 60 text

def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE

Slide 61

Slide 61 text

def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA

Slide 62

Slide 62 text

def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA

Slide 63

Slide 63 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 64

Slide 64 text

def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val service: PropertyProcessingService = ??? Flow.fromFunction(service.process) } PROCESSING FLOW

Slide 65

Slide 65 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 66

Slide 66 text

STORING SINK def adwordsSink(config: AdwordsConfig): Sink[AdwordsChange, Future[Done]] = { val service: AdwordsService = ??? Flow[AdwordsChange].mapAsync(config.parallelism)(service.store) .toMat(Sink.ignore)(Keep.right) }

Slide 67

Slide 67 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 68

Slide 68 text

ERROR SINK def errorSink(cfg: ErrorConfig): Sink[ValidationError, NotUsed] = { val service: AdwordsErrorService = ??? Sink.foreach[ValidationError](service.handle) .mapMaterializedValue(_ ⇒ NotUsed) }

Slide 69

Slide 69 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 70

Slide 70 text

GRAPH def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Sink[AdwordsChange, Future[Done]], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { RunnableGraph.fromGraph(GraphDSL.create(store) { implicit builder ⇒ store ⇒ val p = builder.add(PartitionEither.apply()[ValidationError, AdwordsChange]) p.out0 ~> errorSink source ~> process ~> p.in p.out1 ~> store ClosedShape }) }

Slide 71

Slide 71 text

GRAPH def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Sink[AdwordsChange, Future[Done]], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { RunnableGraph.fromGraph(GraphDSL.create(store) { implicit builder ⇒ store ⇒ val p = builder.add(PartitionEither.apply()[ValidationError, AdwordsChange]) p.out0 ~> errorSink source ~> process ~> p.in p.out1 ~> store ClosedShape }) }

Slide 72

Slide 72 text

def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Sink[AdwordsChange, Future[Done]], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { RunnableGraph.fromGraph(GraphDSL.create(store) { implicit builder ⇒ store ⇒ val p = builder.add(PartitionEither.apply()[ValidationError, AdwordsChange]) p.out0 ~> errorSink source ~> process ~> p.in p.out1 ~> store ClosedShape }) } GRAPH

Slide 73

Slide 73 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 73 PropertyProcessingService def process(p: PropertyChange): Either[ValidationError, AdwordsChange] PropertyChange AdwordsService def store(p: AdwordsChange): Future[Either[StorageError, Stored]] DOMAIN ValidationError AdwordsErrorService def handle(p: AdwordsError): Unit AdwordsChange

Slide 74

Slide 74 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 74 APPLICATION Failure Handling Config Management Materialization Stage Creation

Slide 75

Slide 75 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 75 Application layer Graph Events Services Domain layer Repositories Factories Sources Flows Sinks Other Reactive layer

Slide 76

Slide 76 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 76 Reactive layer Application layer Domain layer Reactive layer Application layer Domain layer Reactive layer Application layer Domain layer Reactive layer Application layer Domain layer

Slide 77

Slide 77 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 77 EXAMPLE PROBLEM

Slide 78

Slide 78 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 78 EXAMPLE PROBLEM

Slide 79

Slide 79 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 79 EXAMPLE PROBLEM

Slide 80

Slide 80 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 80 EXAMPLE PROBLEM

Slide 81

Slide 81 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE STREAMS AKKA STREAMS 81 EXAMPLE PROBLEM

Slide 82

Slide 82 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz Thank you. homeaway.com/careers