$30 off During Our Annual Pro Sale. View Details »

Reactive Streams & Akka: A great new way to handle streaming data

Reactive Streams & Akka: A great new way to handle streaming data

Slides from a talk at the London Scala Users' Group on November 25, 2014.

https://www.meetup.com/de-DE/london-scala/events/218662829/

Description:
Reactive streams - http://www.reactive-streams.org (http://www.reactive-streams.org/) - is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure on the JVM. It is a collaborative effort by Twitter, Netflix, Pivotal, Red Hat, Typesafe and others.

In this session, we want to talk about streams in general, the main concepts behind the reactive streams API, and how to handle streams with Akka ( http://akka.io/ ). After an introductory talk we are going to go through some code examples using Akka streams, an implementation of reactive streams on top of the Akka actor library. We will look into interfacing streams and actors, and handling streams with Akka's Flows.

It will we an entry level session for people interested in stream processing. Previous knowledge of Scala and/or Akka may be beneficial, but is not necessary.

Lutz Hühnken

November 25, 2014
Tweet

More Decks by Lutz Hühnken

Other Decks in Programming

Transcript

  1. View Slide

  2. Reactive Streams and Akka Streams
    Markus Jura (@markusjura)
    & Lutz Huehnken (@lutzhuehnken)

    View Slide

  3. Streams

    View Slide

  4. Reactive Streams
    Common Use of Streams
    • Bulk Data Transfer
    • Real Time Data Sources
    • Batch Processing of large data sets
    • Monitoring and Analytics
    4

    View Slide

  5. Reactive Streams
    What is a Stream?
    • Ephemeral flow of data
    • Potentially unbounded in size
    • Processed by describing transformation of data
    5

    View Slide

  6. Reactive Streams

    View Slide

  7. Reactive Streams
    Reactive Streams Projects / Companies
    • Typesafe
    • Akka Streams
    • Netflix
    • rxJava / rxScala
    • Pivotal
    • Spring Reactor
    • Redhat
    • Vert.x
    • Oracle
    7

    View Slide

  8. Reactive Streams
    Reactive
    8

    View Slide

  9. Reactive Streams
    Reactive Streams Overview
    • How do we:
    • Handle potentially infinite streams of data?
    • Handle data in a reactive manner?
    • Achieve asynchronous non-blocking data flow?
    • Avoid out of memory errors?
    9

    View Slide

  10. Reactive Streams
    Supply and Demand
    • Data Items Flow Downstream
    • Demand Flows Upstream
    • Data Items flow only when there is demand.
    10

    View Slide

  11. Reactive Streams
    Dynamic Push-Pull
    • “Push” behavior when consumer is faster
    • “Pull” behavior when producer is faster
    • Switches automatically between these
    • Batching demand allows batching data
    11

    View Slide

  12. Reactive Streams
    Reactive Streams Specification
    • Interface Specification
    • Java interfaces for implementations
    • TCK
    • Test Harness to validate implementations
    • Specification Website
    • http://www.reactive-streams.org/
    12

    View Slide

  13. Reactive Streams
    Publisher
    package org.reactivestreams;
    public interface Subscriber {...}
    public interface Publisher {
    public void subscribe(
    Subscriber subscriber);
    }
    13

    View Slide

  14. Reactive Streams
    Subscriber
    public interface Subscriber {
    public void onSubscribe(
    Subscription subscription);
    public void onNext(T element);
    public void onComplete();
    public void onError(Throwable cause);
    }
    14

    View Slide

  15. Reactive Streams
    Subscription
    public interface Subscription {
    public void cancel();
    public void request(long elements);
    }
    15

    View Slide

  16. Reactive Streams
    Backpressure
    • Downstream consumers pushing back on the
    producer to prevent flooding.
    • In reactive-streams:
    • Consumers stop requesting more elements until
    they are ready to process more.
    • Producers only fire elements if there is demand.
    16

    View Slide

  17. Reactive Streams
    Why Backpressure?
    • Explicitly design for system overload
    • Demand is propagated throughout the WHOLE flow
    • Can decide WHERE to handle overload
    • Limit the number of in-flight messages throughout the
    system
    • Bounded memory consumption
    • Bounded cpu contention
    • Recipient is in control of incoming data rate
    • Data in flight is bounded by signaled demand
    17

    View Slide

  18. Reactive Streams
    Buffering
    • We can prefetch stream elements by requesting more
    than we really need.
    • We can use this technique to ensure no "sputter" in
    the stream.
    • We can also use this technique to pull faster than
    downstream consumer.
    18

    View Slide

  19. Akka Streams & Actors

    View Slide

  20. Reactive Streams
    Basic Actor
    20
    case class Greeting(who: String)
    class GreetingActor extends Actor with
    ActorLogging {
    def receive = {
    case Greeting(who) => log.info("Hello " + who)
    }
    }
    val system = ActorSystem("MySystem")
    val greeter: ActorRef =
    system.actorOf(Props[GreetingActor])
    greeter ! Greeting("London Scala User Group")

    View Slide

  21. Reactive Streams
    Properties of Actors
    • Message Based / Event Driven
    • Isolated State
    • Sane Concurrency Model
    • Isolated Failure Handling (Supervision)
    21

    View Slide

  22. Reactive Streams
    Akka Streams – A Bridge
    • Converts Publisher/Subscriber API into Actor
    messages
    • Simplify creating Publisher/Subscribers using Actors
    • Attach Reactive streams into existing Akka
    applications
    22

    View Slide

  23. Reactive Streams
    Creating an ActorSubscriber
    import akka.stream._
    class PrintlnActor extends Actor with ActorSubscriber {
    val requestStrategy = OneByOneRequestStrategy
    def receive: Receive = {
    case ActorSubscriberMessage.OnNext(element) =>
    println(element)
    }
    }
    val printlnActor:ActorRef =
    system.actorOf(Props[PrintlnActor], "println")
    val subscriber = ActorSubscriber(printlnActor)
    23

    View Slide

  24. Reactive Streams
    Creating an ActorPublisher
    import akka.stream._
    class IntActor extends Actor with ActorPublisher[Int] {
    def receive: Receive = {
    case ActorPublisherMessage.Request(elements) =>
    while (totalDemand > 0) { onNext(1) }
    }
    }
    val intActor: ActorRef =
    system.actorOf(Props[IntActor], "intActor")
    val publisher = ActorPublisher(intActor)
    24

    View Slide

  25. Reactive Streams
    Why use Actors as Publishers?
    • Actors are smart
    • They can keep internal state to track demand and supply
    • They can buffer data to meet anticipated demand
    • Actors are powerful
    • They can spin up child actors to meet demand
    • With Akka clustering, can spread load across multiple machines
    • Actors are resilient
    • On exception, actor can be killed and restarted by supervisor
    • Actor interaction is thread-safe, and actor state is private
    25

    View Slide

  26. Actors Demo

    View Slide

  27. Flows

    View Slide

  28. Reactive Streams
    Linear Transformations
    28
    Source
    Demand
    Sink
    map …

    View Slide

  29. Reactive Streams
    Linear Transformations
    29
    Source
    Demand
    Sink
    drop
    map

    View Slide

  30. Reactive Streams
    Linear Stream Transformations
    • Deterministic (like for collections)
    • map, filter, collect, grouped, drop, take, groupBy, ...
    • Time-Based
    • takeWithin, dropWithin, groupedWithin, ...
    • Rate-Detached
    • expand, conflate, buffer, ...
    • asynchronous
    • mapAsync, mapAsyncUnordered, ...
    30

    View Slide

  31. Reactive Streams
    Non-linear Stream Transformations
    • Fan-In
    • merge, concat, zip
    • Fan-Out
    • broadcast, route, unzip
    31

    View Slide

  32. Reactive Streams
    Fan In
    32

    View Slide

  33. Reactive Streams
    Fan Out
    33

    View Slide

  34. Reactive Streams
    Materialization
    • Akka Streams separate the what from the how
    • declarative Source/Flow/Sink DSL to create blueprint
    • FlowMaterializer turns this into running Actors
    • this allows alternative materialization strategies
    • optimization
    • verification / validation
    • cluster deployment
    • only Akka Actors for now, but more to come!
    34

    View Slide

  35. FlowGraph Demo

    View Slide

  36. Play

    View Slide

  37. Reactive Streams
    Streaming Data
    • Play can stream data using Iteratees and Enumerators
    • Streamed data through chunked encoding, using Ok.stream()
    • But Iteratees and Enumerators are complicated.
    • And Reactive Streams are simple.
    37

    View Slide

  38. Reactive Streams
    Play 2.4 Experimental Features
    • Reactive Streams Integration!
    • Adapts Futures, Promises, Enumerators and Iteratees
    • Note: potential for event loss, no performance tuning
    • All access through play.api.libs.streams.Streams
    38

    View Slide

  39. Reactive Streams
    Streaming video through Play
    def stream = Action {
    val headers =
    Seq(CONTENT_TYPE -> "video/mp4",
    CACHE_CONTROL -> "no-cache")
    val framePublisher =
    video.FFMpeg.readFile(mp4, Akka.system)
    val frameEnumerator =
    Streams.publisherToEnumerator(framePublisher)
    val bytesEnumeratee = Enumeratee.map[Frame](encodeFrame)
    val chunkedVideoStream =
    frameEnumerator.through(bytesEnumeratee)
    Ok.stream(chunkedVideoStream).withHeaders(headers: _*)
    }
    39

    View Slide

  40. Play Demo

    View Slide

  41. Questions

    View Slide

  42. Thanks
    @markusjura
    @lutzhuehnken

    View Slide

  43. ©Typesafe 2014 – All Rights Reserved

    View Slide