A dive into Akka Streams: from the basics to a real-world scenario

Reactive streaming is becoming the best approach to handle data flows across asynchronous boundaries. Here, we present the implementation of a real-world application based on Akka Streams. After reviewing the basics, we will discuss the development of a data processing pipeline that collects real-time sensor data and sends it to a Kinesis stream. There are various possible point of failures in this architecture. What should happen when Kinesis is unavailable? If the data flow is not handled in the correct way, some information may get lost. Akka Streams are the tools that enabled us to build a reliable processing logic for the pipeline that avoids data losses and maximizes the robustness of the entire system.

Transcript

1. A dive into Akka Streams From the basics to a real-world scenario Gioia Ballin, Simone Fabiano - SW developers
2. Who are these guys? Gioia Ballin Twitter / @GioiaBallin Simone Fabiano Twitter / @mone_tpatpc R&D Data Analysis Microservices Infrastructure Web Blog
3. Agenda The Problem Akka Streams Basics The Collector Service Recap and Learnings
4. Data processing pipeline COLLECTOR SVC AMAZON KINESIS WI-FI/3G SENSORS
5. The Collector Service: two months ago... AMAZON KINESIS INBOX ACTOR TCP Kinesis Client INBOX ACTOR Kinesis Client INBOX ACTOR Kinesis Client TCP TCP chunks chunks chunks events events events put put put
7. Data may be lost COLLECTOR SVC AMAZON KINESIS WI-FI/3G SENSORS CONNECTION ISSUES MESSAGE OVERLOAD
8. Not REALLY reactive: why? Lack of... RECOVERY MECHANISMS BACKPRESSURE
9. Recovery mechanisms Upload failure? Retry Kinesis unavailable? Buffer
10. Ingestion speed under control Take advantage of sensors buffer Sensors switch from busier to less busy services
11. Welcome Akka Streams! Backpressure on the network + Easy fit for stream paradigm Typed flow & Increased Readability
12. Akka Streams Basics - 3 2 1 Go!
13. SOURCE FLOW SINK
14. val source = Source(1 to 42) val flow = Flow[Int].map(_ + 1) val sink = Sink.foreach(println) val graph = source.via(flow).to(sink) graph.run()
15. Flows are collections (almost) OK map, filter, fold, … No flatMap! (there’s mapConcat though) Extras mapAsync, mapAsyncUnordered, buffer..
16. How backpressure works Source Flow Sink pull pull push push
17. Collector with Akka Streams
18. path("data") { post { extractRequest { request => val source = request.entity.dataBytes val flow = processing() val sink = Sink.ignore source.via(flow).runWith(sink) … } }
19. Flow[ByteString] .via(chunkBuffer)
20. Transformation tools: Framing.delimiter val chunkBuffer = Framing.delimiter( ByteString("\n"), maxPresenceEventBytes, false ) .map(_.dropRight(1)) .map(_.utf8String)
21. Flow[ByteString] .via(chunkBuffer) .via(presenceEventFactory.asStream)
22. Flow[ByteString] .via(chunkBuffer) .via(presenceEventFactory.asStream) .via(persistentBuffer)
23. class PersistentBuffer[A] (...) extends GraphStage[FlowShape[A, A]] { val in = Inlet[A]("PersistentBuffer.in") val out = Outlet[A]("PersistentBuffer.out") override val shape = FlowShape.of(in,out) When the available stages are not enough, write your own
24. Custom stages: State override def createLogic( attr: Attributes ): GraphStageLogic = new GraphStageLogic(shape) { var state: StageState[A] = initialState[A]
25. val cb = getAsyncCallback[Try[A]] { case Success(elements:A) => state = state.copy(...) ... } queue.getCurrentQueue .onComplete(cb.invoke)
26. Custom Stages: ports setHandler(in, new InHandler { override def onPush() = { val element = grab(in) pull(in) ... } ... }) setHandler(out, new OutHandler { override def onPull(): Unit = { push(out, something) } ... })
27. Custom Stages: Start and stop override def postStop(): Unit = { ... } override def preStart(): Unit = { ... }
28. Flow[ByteString] .via(chunkBuffer) .via(presenceEventFactory.asStream) .via(persistentBuffer) .via(kinesisPublisher)
29. Delegating work to actors Flow[KinesisEvent] .mapAsync(1) { event => (publisher ? Publish(event)) .mapTo[PublishCompleted] }
30. Flow[KinesisEvent] .mapAsync(1) { event => (publisher ? Publish(event)) .mapTo[Future[PublishCompleted]] .flatten }
31. Flow[ByteString] .via(chunkBuffer) .via(presenceEventFactory.asStream) .via(persistentBuffer) .via(kinesisPublisher) .alsoTo(countEvents)
32. Extra side-effects val countEvents = Sink.foreach[Event] { _ => metricEventsPerTimeUnitOfSensor.mark() metricEventsPerTimeUnit.mark() }
33. Automatic Fusion and Async Stages A stream is handled by 1 thread NO CROSS-THREAD COMMUNICATIONS NO PARALLELISM FASTER CODE! SLOWER CODE! A boundary will split your stream on different threads source .via(doSomething).async .via(doSomething).async .runWith(Sink.foreach(println))
34. Materialized Values val sink: Sink[Any, Future[Done]] = Sink.ignore .runWith(sink) is a shortcut for .toMat(sink)(Keep.Right).run STREAM RUN Obtain a materialized value per each stage
35. Materialized Values Example: TestKit val myFlow = Flow[String].map { v => v.take(1) } val (pub, sub) = TestSource.probe[String] .via(myFlow) .toMat(TestSink.probe[Any])(Keep.both) .run() sub.request(1) pub.sendNext("Gathering") sub.expectNext("G")
36. Stream Ending: Supervisioning val flow = otherFlow .withAttributes ActorAttributes.supervisionStrategy { case _ => Supervision.resume } )
37. Stream Ending: Supervisioning RESUME RESET STOP > drop failing elem > keep going > drop failing elem > reset stage state > keep going fail all the things!
38. Other shapes FAN OUT FAN IN WHATEVER
39. Finally Got Reactive!
40. In two months to production... Memory Leaks Data Losses Backpressure on the network Recovery mechanisms
41. Key learnings Actors aren’t sufficient to be reactive Backpressure as a building block Akka Streams API is rich
42. THANK YOU! Sleep more worry less