Tweet
Tweet

Slide 1

Slide 1 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. REACTIVE SUMMIT 2017

Slide 2

Slide 2 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STREAM DRIVEN DEVELOPMENT Design your data pipeline with Akka Streams

Slide 3

Slide 3 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 3 © 2017 IKEA

Slide 4

Slide 4 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 4 © RonTech2000/iStock

Slide 5

Slide 5 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz

Slide 6

Slide 6 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. THE PROBLEM 6

Slide 7

Slide 7 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 7

Slide 8

Slide 8 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 8 AD PLATFORMS

Slide 9

Slide 9 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 9

Slide 10

Slide 10 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 10 Pricing Images Property Details ...

Slide 11

Slide 11 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 11

Slide 12

Slide 12 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 12 RESILIENT

Slide 13

Slide 13 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 13 RESILIENT SCALABLE

Slide 14

Slide 14 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 14 RESILIENT SCALABLE REACTIVE

Slide 15

Slide 15 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. OBJECTIVE Quickly roll out pipelines which are 15 RESILIENT SCALABLE streams

Slide 16

Slide 16 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 16 PROCESSING

Slide 17

Slide 17 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. EXAMPLE 17 PROCESSING

Slide 18

Slide 18 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. THE APPROACH 18

Slide 19

Slide 19 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 19 STAGES TYPES CODE

Slide 20

Slide 20 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. TYPES 20 STAGES CODE

Slide 21

Slide 21 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 21 SOURCE (1 output) FAN-IN (n inputs, 1 output) FAN-OUT (1 input, n outputs) RUNNABLEGRAPH (no input or output) STREAMS STAGES FLOW (1 input, 1 output) SINK (1 input) ... CUSTOM

Slide 22

Slide 22 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 22 Source ~> Flow = Source Flow ~> Sink = Sink Flow ~> Flow = Flow Broadcast ~> Merge = Flow Source ~> Sink = RunnableGraph Source ~> Flow ~> Sink = RunnableGraph STAGE ARITHMETIC

Slide 23

Slide 23 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 23

Slide 24

Slide 24 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 24 Source Sink

Slide 25

Slide 25 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 25 Flow Source Sink

Slide 26

Slide 26 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 26 Flow RunnableGraph Source Sink

Slide 27

Slide 27 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 27 TYPES STAGES CODE 27

Slide 28

Slide 28 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 28 Flow[In, Out, Mat]

Slide 29

Slide 29 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 29 Flow[In, Out, Mat] PORTS

Slide 30

Slide 30 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. STAGE SIGNATURE 30 Flow[In, Out, Mat] MATERIALIZED VALUE

Slide 31

Slide 31 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 31 Flow RunnableGraph Source Sink

Slide 32

Slide 32 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 32 Flow RunnableGraph Source[PropertyChange, NotUsed] Sink

Slide 33

Slide 33 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 33 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph Source[PropertyChange, NotUsed] Sink

Slide 34

Slide 34 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 34 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]]

Slide 35

Slide 35 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 35 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph[Future[Done]] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]]

Slide 36

Slide 36 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 36 Flow[PropertyChange, AdwordsChange, NotUsed] RunnableGraph[Future[Done]] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]]

Slide 37

Slide 37 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 37 SIDE EFFECTS Error Handling Logging Monitoring ...

Slide 38

Slide 38 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 38 SIDE EFFECTS Error Handling Logging Monitoring EVENTS Errors Audit ... ...

Slide 39

Slide 39 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 39 SIDE EFFECTS Error Handling Logging Monitoring EVENTS Errors Audit ... ... REFERENTIALLY TRANSPARENT STAGES Flow[In, Either[Error, Out], Mat] Flow[In, (Seq[Audit], Out), Mat] ...

Slide 40

Slide 40 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Sink[AdwordsChange, Future[Done]] RunnableGraph[Future[Done]] Sink[ValidationError, NotUsed]

Slide 41

Slide 41 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]

Slide 42

Slide 42 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. CODE TYPES 42 STAGES 42

Slide 43

Slide 43 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]

Slide 44

Slide 44 text

def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE

Slide 45

Slide 45 text

def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA

Slide 46

Slide 46 text

def propertySource(config: KafkaConfig): Source[PropertyChange, NotUsed] = { def settings(config: KafkaConfig): ConsumerSettings[String, PropertyChange] = ??? val kafkaSrc: Source[ConsumerRecord[String, PropertyChange], Control] = Consumer.plainSource( settings(config), Subscriptions.topics(config.topic) ) kafkaSrc .map(_.value) .mapMaterializedValue { _ ⇒ NotUsed } } SOURCE - REACTIVE KAFKA

Slide 47

Slide 47 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]

Slide 48

Slide 48 text

def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val service: PropertyProcessingService = ??? Flow.fromFunction(service.process) } PROCESSING FLOW

Slide 49

Slide 49 text

def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val validationService : PropertyValidationService = ??? val transformationService: PropertyTransformationService = ??? Flow.fromFunction(validationService.validate) .map(transformationService.transform) } PROCESSING FLOW

Slide 50

Slide 50 text

def processingFlow(): Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] = { val validationService : PropertyValidationService = ??? val transformationService: PropertyTransformationService = ??? Flow.fromFunction(validationService.validate) .async .map(transformationService.transform) } PROCESSING FLOW

Slide 51

Slide 51 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]

Slide 52

Slide 52 text

STORING FLOW def adwordsFlow(config: AdwordsConfig): Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] = { val service: AdwordsService = ??? Flow[AdwordsChange].mapAsync(config.parallelism)(service.store) }

Slide 53

Slide 53 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]

Slide 54

Slide 54 text

ERROR SINK def errorSink(cfg: ErrorConfig): Sink[AdwordsStreamError, NotUsed] = { val service: AdwordsErrorService = ??? Sink.foreach[AdwordsStreamError](service.handle) .mapMaterializedValue(_ ⇒ NotUsed) }

Slide 55

Slide 55 text

Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed] Source[PropertyChange, NotUsed] Flow[AdwordsChange, Either[StorageError, Stored], NotUsed] RunnableGraph[Future[Done]] Sink[Error, NotUsed] Sink[Stored, Future[Done]]

Slide 56

Slide 56 text

def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH

Slide 57

Slide 57 text

def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH

Slide 58

Slide 58 text

def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH

Slide 59

Slide 59 text

def graph(source : Source[PropertyChange, NotUsed], process : Flow[PropertyChange, Either[ValidationError, AdwordsChange], NotUsed], store : Flow[AdwordsChange, Either[StorageError, Stored], NotUsed], errorSink: Sink[AdwordsStreamError, NotUsed]): RunnableGraph[Future[Done]] = { val processAndDivert: Flow[PropertyChange, AdwordsChange, NotUsed] = process via DivertErrors(to = errorSink) val storeAndDivert: Flow[AdwordsChange, Stored, NotUsed] = store via DivertErrors(to = errorSink) source .via(processAndDivert) .via(storeAndDivert) .toMat(Sink.ignore)(Keep.right) } GRAPH

Slide 60

Slide 60 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 60 Flow[Either[E,T], T, M] Sink[E, M] Either[E, T] T E DIVERT ERRORS

Slide 61

Slide 61 text

def divertErrors[T, E, M](to: Sink[E, M]): Flow[Either[E, T], T, M] = { Flow.fromGraph(GraphDSL.create(to) { implicit b ⇒ sink ⇒ val partition = b.add(Partition[Either[E, T]](2, _.fold(_ ⇒ 0, _ ⇒ 1))) val left = b.add(Flow[Either[E, T]].map (_.left.get)) val right = b.add(Flow[Either[E, T]].map (_.right.get)) partition ~> left ~> sink partition ~> right FlowShape(partition.in, right.out) }) } GRAPH - DIVERT ERRORS

Slide 62

Slide 62 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 62 PropertyProcessingService def process(p: PropertyChange): Either[ValidationError, AdwordsChange] PropertyChange AdwordsService def store(p: AdwordsChange): Future[Either[StorageError, Stored]] DOMAIN ValidationError AdwordsChange Stored StorageError AdwordsErrorService def handle(p: AdwordsError): Unit

Slide 63

Slide 63 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 63 APPLICATION Failure Handling Config Management Materialization Stage Creation

Slide 64

Slide 64 text

val cfg: ApplicationConfig = ??? val source = propertySource(cfg.kafka) val pFlow = processingFlow() val sFlow = adwordsFlow(cfg.adwords) val eSink = errorSink(cfg.error) val graph = AdwordsGraph.graph(source, pFlow, sFlow, eSink) graph.run() APPLICATION

Slide 65

Slide 65 text

final case class ApplicationConfig(kafka : KafkaConfig, adwords : AdwordsConfig, error : ErrorConfig) APPLICATION - CONFIGURATION

Slide 66

Slide 66 text

kafka { bootstrapServers = "my.kafka" consumerGroup = "my-consumer" topic = "properties" } adwords { parallelism = 5 endpoint = "http://adwords.google.com/properties" } error { loggerName = "adwordsErrors" } APPLICATION - CONFIGURATION

Slide 67

Slide 67 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 67 Application layer Graph Events Services Domain layer Repositories Factories Sources Flows Sinks Other Reactive layer

Slide 68

Slide 68 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 68 STAGES TYPES CODE 68

Slide 69

Slide 69 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. TYPES 69 STAGES CODE 69

Slide 70

Slide 70 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 70 TYPES STAGES CODE 70

Slide 71

Slide 71 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. CODE TYPES 71 STAGES 71

Slide 72

Slide 72 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. 72 STAGES TYPES CODE 72

Slide 73

Slide 73 text

© 2017 HOMEAWAY. ALL RIGHTS RESERVED. Stefano Bonetti Software Engineer @svezfaz @svez_faz Thank you.