Monitoring Reactive Streams

Reactive Streams are the key to build asynchronous, data-intensive applications with no predetermined data volumes. By enabling non-blocking backpressure, they boost the resiliency of your systems by design. But how do you tune and debug such applications? When productionizing Reactive Streams, the same backpressure that preserves the safety of your pipeline can get in the way of effectively monitoring its status. In this talk we’ll present a line of action to

- measure the throughput of your pipeline
- identify its bottlenecks and look at possible tuning counteractions
- diagnose liveness issues.

Examples will be in Scala and Akka Streams, however these patterns are generic and applicable to any Reactive Streams implementation out there.

Transcript

1. Scale by the Bay 2018 Stefano Bonetti - @svez_faz
2. Monitoring Reactive Streams Stefano Bonetti - @svez_faz
4. Agenda Throughput Backpressure
5. Agenda Throughput Backpressure
6. Throughput Web Service req res
7. Topology
8. Topology
9. I/O Correlation web service
10. I/O Correlation Processor[T, S]
11. Transparent Processors Processor[T, S] Processor[T, S] Subscriber[S] Publisher[T]
12. Transparent Processors Processor[T, S] Processor[T, S] Subscriber[S] Publisher[T] c1 c2 c3 c4 Processor[T, T] Processor[S, S]
13. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val evenOnly = Flow[Int].filter(_ % 2 == 0) val print = Sink.foreach(println) randomInt .via(evenOnly) .runWith(print) AKKA STREAMS - THROUGHPUT
14. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val evenOnly = Flow[Int].filter(_ % 2 == 0) val print = Sink.foreach(println) randomInt .via(evenOnly) .runWith(print) AKKA STREAMS - THROUGHPUT
15. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val evenOnly = Flow[Int].filter(_ % 2 == 0) val print = Sink.foreach(println) def meter[T](name: String): Flow[T, T, NotUsed] = { val msgCounter = Kamon.metrics.counter(name) Flow[T].map { x ⇒ msgCounter.increment(); x } } randomInt .via(meter("produced")) .via(evenOnly) .via(meter("processed")) .runWith(print) AKKA STREAMS - THROUGHPUT
16. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val evenOnly = Flow[Int].filter(_ % 2 == 0) val print = Sink.foreach(println) def meter[T](name: String): Flow[T, T, NotUsed] = { val msgCounter = Kamon.metrics.counter(name) Flow[T].map { x ⇒ msgCounter.increment(); x } } randomInt .via(meter("produced")) .via(evenOnly) .via(meter("processed")) .runWith(print) AKKA STREAMS - THROUGHPUT
17. Throughput dashboard (/s) randomInt .via(meter("produced")) .via(heavyEvenOnly) .via(meter("processed")) .runWith(print)
18. Agenda Throughput Backpressure
19. Agenda Throughput Backpressure
20. Example Validation/ Filtering Commit Kafka offset valid invalid Store via API
21. Example 100/s 25/s 75/s 75/s Validation/ Filtering Commit Kafka offset valid invalid Store via API
22. Example 100/s 25/s 75/s 75/s Validation/ Filtering Commit Kafka offset valid invalid Store via API 20000/s lag
23. Backpressure
24. Reactive Streams 101 Processor Publisher Subscriber request request emit emit
25. Reactive Streams 101 Processor Publisher Subscriber request request emit emit
26. Reactive Streams 101 Processor Publisher Subscriber request request emit emit
27. Reactive Streams 101 Processor Publisher Subscriber request request emit emit
28. Reactive Streams 101 Processor Publisher Subscriber request request emit emit
29. Backpressure Processor Publisher Subscriber request ? request ? emit emit
30. Backpressure Processor Publisher Subscriber request request emit ? emit ?
31. Backpressure Processor Publisher Subscriber request request emit emit request emit request request emit
32. Akka Streams - Graphstage API
33. final case class Map[In, Out](f: In ⇒ Out) extends GraphStage[FlowShape[In, Out]] { val in = Inlet[In]("Map.in") val out = Outlet[Out]("Map.out") override val shape = FlowShape(in, out) override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with InHandler with OutHandler { override def onPush(): Unit = push(out, f(grab(in))) override def onPull(): Unit = pull(in) setHandlers(in, out, this) } } AKKA STREAMS - GRAPHSTAGE API
34. final case class Map[In, Out](f: In ⇒ Out) extends GraphStage[FlowShape[In, Out]] { val in = Inlet[In]("Map.in") val out = Outlet[Out]("Map.out") override val shape = FlowShape(in, out) override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with InHandler with OutHandler { override def onPush(): Unit = push(out, f(grab(in))) override def onPull(): Unit = pull(in) setHandlers(in, out, this) } } AKKA STREAMS - GRAPHSTAGE API
35. final case class Map[In, Out](f: In ⇒ Out) extends GraphStage[FlowShape[In, Out]] { val in = Inlet[In]("Map.in") val out = Outlet[Out]("Map.out") override val shape = FlowShape(in, out) override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with InHandler with OutHandler { override def onPush(): Unit = push(out, f(grab(in))) override def onPull(): Unit = pull(in) setHandlers(in, out, this) } } AKKA STREAMS - GRAPHSTAGE API
36. final case class Map[In, Out](f: In ⇒ Out) extends GraphStage[FlowShape[In, Out]] { val in = Inlet[In]("Map.in") val out = Outlet[Out]("Map.out") override val shape = FlowShape(in, out) override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with InHandler with OutHandler { override def onPush(): Unit = push(out, f(grab(in))) override def onPull(): Unit = pull(in) setHandlers(in, out, this) } } AKKA STREAMS - GRAPHSTAGE API
37. final case class Map[In, Out](f: In ⇒ Out) extends GraphStage[FlowShape[In, Out]] { val in = Inlet[In]("Map.in") val out = Outlet[Out]("Map.out") override val shape = FlowShape(in, out) override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with InHandler with OutHandler { override def onPush(): Unit = push(out, f(grab(in))) override def onPull(): Unit = pull(in) setHandlers(in, out, this) } } AKKA STREAMS - GRAPHSTAGE API
38. final case class Map[In, Out](f: In ⇒ Out) extends GraphStage[FlowShape[In, Out]] { val in = Inlet[In]("Map.in") val out = Outlet[Out]("Map.out") override val shape = FlowShape(in, out) override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) with InHandler with OutHandler { override def onPush(): Unit = push(out, f(grab(in))) override def onPull(): Unit = pull(in) setHandlers(in, out, this) } } AKKA STREAMS - GRAPHSTAGE API
39. Akka Streams Graphstage onPull onPush onPull onPull onPush
40. var lastPulled: Long = System.nanoTime() var lastPushed: Long = lastPulled private val backpressure = Kamon.histogram(label + "_backpressure") override def onPush(): Unit = { push(out, grab(in)) val now = System.nanoTime() backpressure.record((lastPulled - lastPushed) * 100 / now - lastPushed) lastPushed = now } override def onPull(): Unit = { pull(in) lastPulled = System.nanoTime() } setHandlers(in, out, this) AKKA STREAMS - BACKPRESSURE
41. var lastPulled: Long = System.nanoTime() var lastPushed: Long = lastPulled private val backpressure = Kamon.histogram(label + "_backpressure") override def onPush(): Unit = { push(out, grab(in)) val now = System.nanoTime() backpressure.record((lastPulled - lastPushed) * 100 / now - lastPushed) lastPushed = now } override def onPull(): Unit = { pull(in) lastPulled = System.nanoTime() } setHandlers(in, out, this) AKKA STREAMS - BACKPRESSURE
42. var lastPulled: Long = System.nanoTime() var lastPushed: Long = lastPulled private val backpressure = Kamon.histogram(label + "_backpressure") override def onPush(): Unit = { push(out, grab(in)) val now = System.nanoTime() backpressure.record((lastPulled - lastPushed) * 100 / now - lastPushed) lastPushed = now } override def onPull(): Unit = { pull(in) lastPulled = System.nanoTime() } setHandlers(in, out, this) AKKA STREAMS - BACKPRESSURE
43. var lastPulled: Long = System.nanoTime() var lastPushed: Long = lastPulled private val backpressure = Kamon.histogram(label + "_backpressure") override def onPush(): Unit = { push(out, grab(in)) val now = System.nanoTime() backpressure.record((lastPulled - lastPushed) * 100 / now - lastPushed) lastPushed = now } override def onPull(): Unit = { pull(in) lastPulled = System.nanoTime() } setHandlers(in, out, this) AKKA STREAMS - BACKPRESSURE
44. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val evenOnly = Flow[Int].filter(_ % 2 == 0) val print = Sink.foreach(println) randomInt .via(backpressureMeter("produced")) .via(evenOnly) .via(backpressureMeter("processed")) .runWith(print) AKKA STREAMS - BACKPRESSURE
45. Backpressure dashboard (%) randomInt .via(backpressureMeter("produced")) .via(heavyEvenOnly) .via(backpressureMeter("processed")) .runWith(print)
46. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val evenOnly = Flow[Int].filter(_ % 2 == 0) val print = Sink.foreach(println) randomInt .via(backpressureMeter("produced")) .via(evenOnly) .via(backpressureMeter("processed")) .runWith(print) AKKA STREAMS - BACKPRESSURE
47. val randomInt = Source.tick(10.millis, 10.millis, NotUsed) .map(_ ⇒ Random.nextInt()) val heavyEvenOnly = Flow[Int].mapAsync(parallelism = 1) { n => after(250.millis, system.scheduler)(Future.successful(n)) }.filter(_ % 2 == 0) val print = Sink.foreach(println) randomInt .via(backpressureMeter("produced")) .via(heavyEvenOnly) .via(backpressureMeter("processed")) .runWith(print) AKKA STREAMS - BACKPRESSURE
48. Bottleneck - throughput dashboard (/s) randomInt .via(meter("produced")) .via(heavyEvenOnly) .via(meter("processed")) .runWith(print)
49. Bottleneck - backpressure dashboard (%) randomInt .via(backpressureMeter("produced")) .via(heavyEvenOnly) .via(backpressureMeter("processed")) .runWith(print)
50. Bottlenecks https://blog.colinbreck.com/maximizing-throughput-for-akka-streams/
51. Agenda Throughput Backpressure
52. akka-stream-checkpoint svezfaz/akka-stream-checkpoint throughput backpressure ratio push-pull latency liveness checks failures/completion
53. Stefano Bonetti Software Engineer @svezfaz @svez_faz