Compositional Streaming with FS2

In recent years, a number of open source Scala libraries have appeared that support working with data streams. In this talk, we’ll look at Functional Streams for Scala (FS2), the library formerly known as Scalaz Stream, and explore its unique take on stream processing. We’ll look at working with impure data sources in a pure world, data transformations, and patterns for stream based program design.

Presented at Scalae By The Bay 2016 in San Francisco on November 13, 2016.

Transcript

1. Compositional Streaming with FS2
   Michael Pilquist // @mpilquist
   Scala By The Bay
   November 2016
   

   View Slide

2. scalaz-stream => fs2
   2
   

   View Slide

3. scalaz-stream => fs2
   • Dependencies: scalaz & scodec-bits
   • Actively maintained!
   • scalaz 7.{1,2,3}
   • scala 2.{10,11,12}
   3
   libraryDependencies +=
   "org.scalaz.stream" %%
   "scalaz-stream" % "0.8.6"
   libraryDependencies +=
   "co.fs2" %% "fs2-core" % "0.9.2"
   • No dependencies
   • Supports Scala.js
   • Bindings available for Scalaz and Cats
   • Significantly improved API
   • Smaller core interpreter
   • Correct
   • More parametricity
   https://github.com/functional-streams-for-scala/fs2
   series/0.8 series/0.9
   

   View Slide

4. Stream
   4
   class Stream[+F[_], +O]
   Output values
   Effects
   

   View Slide

5. Pure Streams
   5
   scala> val s = Stream(1, 2, 3)
   s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))
   

   View Slide

6. Pure Streams
   6
   scala> val s = Stream(1, 2, 3)
   s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))
   scala> val xs = (s  s.map(_*10)).toList
   xs: List[Int] = List(1, 2, 3, 10, 20, 30)
   

   View Slide

7. Pure Streams
   7
   scala> val s = Stream(1, 2, 3)
   s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))
   scala> val xs = (s  s.map(_*10)).toList
   xs: List[Int] = List(1, 2, 3, 10, 20, 30)
   scala> val ys = s.intersperse(0).toList
   ys: List[Int] = List(1, 0, 2, 0, 3)
   

   View Slide

8. Pure Streams
   8
   scala> val s = Stream(1, 2, 3)
   s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))
   scala> val xs = (s  s.map(_*10)).toList
   xs: List[Int] = List(1, 2, 3, 10, 20, 30)
   scala> val ys = s.intersperse(0).toList
   ys: List[Int] = List(1, 0, 2, 0, 3)
   toList is only available
   on pure streams
   

   View Slide

9. Pure Streams
   9
   scala> val fibs: Stream[Nothing, Int] =
   Stream(0, 1)  fibs.zipWith(fibs.tail)(_ + _)
   fibs: fs2.Stream[Nothing,Int] = 
   scala> fibs.take(10).toList // CAUTION: *NOT* Memoized!
   res0: List[Int] = List(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)
   

   View Slide

10. Effectful Streams
    10
    val sample: Task[Sample] =
    Task.delay { sensor.read() }
    

    View Slide

11. Effectful Streams
    11
    val sample: Task[Sample] =
    Task.delay { sensor.read() }
    val samples: Stream[Task, Sample] =
    Stream.eval(sample).repeat
    

    View Slide

12. Effectful Streams
    12
    val sample: Task[Sample] =
    Task.delay { sensor.read() }
    val samples: Stream[Task, Sample] =
    Stream.eval(sample).repeat
    val firstTen: Task[Vector[Sample]] =
    samples.take(10).runLog
    

    View Slide

13. Effectful Streams
    13
    val sample: Task[Sample] =
    Task.delay { sensor.read() }
    val samples: Stream[Task, Sample] =
    Stream.eval(sample).repeat
    val firstTen: Task[Vector[Sample]] =
    samples.take(10).runLog
    val result: Vector[Sample] =
    firstTen.unsafeRun()
    

    View Slide

14. Running Effectful Streams
    14
    implicit class StreamRunOps[F[_]: Catchable, O](
    val self: Stream[F,O]
    ) {
    def run: F[Unit]
    def runLog: F[Vector[O]]
    def runLast: F[Option[O]]
    def runFold[A](z: A)(f: (A,O)  A): F[A]
    }
    

    View Slide

15. Transforming Streams
    15
    Pipe[F[_], -I, +O]
    Output values
    Effects
    Input values
    

    View Slide

16. Pipe Examples
    16
    object text {
    def utf8Decode[F[_]]: Pipe[F, Byte, String] = ???
    def lines[F[_]]: Pipe[F, String, String] = ???
    }
    val linesOfFile: Stream[Task, String] =
    io.file.readAll[Task](myFile, 4096).
    through(text.utf8Decode).
    through(text.lines)
    

    View Slide

17. Combining Streams
    17
    Pipe2[F[_], -I, -I2, +O]
    Output values
    Effects
    Input values
    

    View Slide

18. Pipe2 Examples
    18
    fibs.zipWith(fibs.tail)(_+_)
    

    View Slide

19. Pipe2 Examples
    19
    fibs.zipWith(fibs.tail)(_+_)
    // alternatively
    fibs.pure.through2(fibs.tail)(pipe2.zipWith(_+_))
    

    View Slide

20. Pipe2 Examples
    20
    val x = Stream(1, 2, 3)
    val y = Stream(4, 5, 6)
    val z = x interleave y
    val elems = z.toList
    // elems: List[String] = List(1, 4, 2, 5, 3, 6)
    

    View Slide

21. Pipe2 Examples
    21
    val x = Stream(1, 2, 3)
    val y = Stream(4, 5, 6)
    val z = x interleave y
    // alternatively
    x.pure.through2(y)(pipe2.interleave)
    val elems = z.toList
    // elems: List[String] = List(1, 4, 2, 5, 3, 6)
    

    View Slide

22. Sinking Streams
    22
    Sink[F[_], -I]
    Effects
    Input values
    

    View Slide

23. Sink Examples
    23
    val outputFile: Sink[Task, Byte] =
    io.file.writeAll[Task](myFile)
    

    View Slide

24. Sink Examples
    24
    val outputFile: Sink[Task, Byte] =
    io.file.writeAll[Task](myFile)
    val toFile: Stream[Task, Unit] =
    lines.to(outputFile)
    

    View Slide

25. Sink Examples
    25
    val outputFile: Sink[Task, Byte] =
    io.file.writeAll[Task](myFile)
    val toFile: Stream[Task, Unit] =
    lines.to(outputFile)
    val prg: Task[Unit] = toFile.run
    

    View Slide

26. Sink Examples
    26
    val outputFile: Sink[Task, Byte] =
    io.file.writeAll[Task](myFile)
    val toFile: Stream[Task, Unit] =
    lines.to(outputFile)
    val prg: Task[Unit] = toFile.run
    prg.unsafeRun()
    

    View Slide

27. Sink Examples: Writing events to Kafka
    27
    val kafkaTopic: Sink[Task, Record] =
    Producer.sink(Topic("foo"), kafkaSettings)
    

    View Slide

28. Sink Examples: Writing events to Kafka
    28
    val kafkaTopic: Sink[Task, Record] =
    Producer.sink(Topic("foo"), kafkaSettings)
    val events: Stream[Task, Event] = ???
    

    View Slide

29. Sink Examples: Writing events to Kafka
    29
    val kafkaTopic: Sink[Task, Record] =
    Producer.sink(Topic("foo"), kafkaSettings)
    val events: Stream[Task, Event] = ???
    val program: Stream[Task, Unit] =
    events.map(eventToRecord).to(kafkaTopic)
    

    View Slide

30. Sink Examples: Writing events to Kafka
    30
    val kafkaTopic: Sink[Task, Record] =
    Producer.sink(Topic("foo"), kafkaSettings)
    val events: Stream[Task, Event] = ???
    val program: Stream[Task, Unit] =
    events.map(eventToRecord).to(kafkaTopic)
    program.run.unsafeRun()
    

    View Slide

31. Composing Stream Transformations
    31
    Do pipes compose as well as functions?
    

    View Slide

32. Composing Stream Transformations
    32
    type Pipe[F[_],-I,+O] =
    Stream[F,I]  Stream[F,O]
    Do pipes compose as well as functions?
    

    View Slide

33. Composing Stream Transformations
    33
    type Pipe[F[_],-I,+O] =
    Stream[F,I]  Stream[F,O]
    type Pipe2[F[_],-I,-I2,+O] =
    (Stream[F,I], Stream[F,I2])  Stream[F,O]
    Do pipes compose as well as functions?
    

    View Slide

34. Composing Stream Transformations
    34
    type Pipe[F[_],-I,+O] =
    Stream[F,I]  Stream[F,O]
    type Pipe2[F[_],-I,-I2,+O] =
    (Stream[F,I], Stream[F,I2])  Stream[F,O]
    type Sink[F[_],-I] = Pipe[F,I,Unit]
    Do pipes compose as well as functions?
    

    View Slide

35. Composing Stream Transformations
    35
    type Pipe[F[_],-I,+O] =
    Stream[F,I]  Stream[F,O]
    type Pipe2[F[_],-I,-I2,+O] =
    (Stream[F,I], Stream[F,I2])  Stream[F,O]
    type Sink[F[_],-I] = Pipe[F,I,Unit]
    // Function composition!
    src.through(text.utfDecode andThen text.lines)
    Do pipes compose as well as functions?
    

    View Slide

36. README
    36
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    

    View Slide

37. README
    37
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    

    View Slide

38. README
    38
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    

    View Slide

39. README
    39
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    

    View Slide

40. README
    40
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    intersperse("\n").
    

    View Slide

41. README
    41
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    intersperse("\n").
    through(text.utf8Encode).
    

    View Slide

42. README
    42
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    intersperse("\n").
    through(text.utf8Encode).
    to(io.file.writeAll(Paths.get("celsius.txt"))).
    

    View Slide

43. README
    43
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    val converter: Task[Unit] =
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    intersperse("\n").
    through(text.utf8Encode).
    to(io.file.writeAll(Paths.get("celsius.txt"))).
    run
    

    View Slide

44. README
    44
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    val converter: Task[Unit] =
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    intersperse("\n").
    through(text.utf8Encode).
    to(io.file.writeAll(Paths.get("celsius.txt"))).
    run
    converter.unsafeRun()
    

    View Slide

45. README
    45
    import fs2.{io, text, Task}, java.nio.file.Paths
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    val converter: Task[Unit] =
    io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
    through(text.utf8Decode andThen text.lines).
    filter(s  !s.trim.isEmpty && !s.startsWith("//")).
    map(line  fahrenheitToCelsius(line.toDouble).toString).
    intersperse("\n").
    through(text.utf8Encode).
    to(io.file.writeAll(Paths.get("celsius.txt"))).
    run
    converter.unsafeRun()
    • Input file is read & output file is
    written in a streamy fashion
    • Constant memory usage
    • Errors are handled
    • I/O errors
    • Exceptions thrown by
    function passed to map
    • File handles are acquired
    when necessary and are
    guaranteed to be released
    

    View Slide

46. Resource Allocation
    46
    def bracket[F[_],R,A](acquire: F[R])(
    use: R  Stream[F,A],
    release: R  F[Unit]
    ): Stream[F,A]
    • Evaluates acquire to produce a resource
    • Passes resource to use to generate a
    stream
    • Decorates the stream returned from use
    so that the resource is released when
    the inner stream terminates
    • Stream termination occurs when:
    • Natural end of stream is reached
    • Puller of stream decides to stop
    pulling (e.g., take(5))
    • Unhandled error occurs
    

    View Slide

47. Pulls and Handles
    47
    Stream[F, A] Pull[F, A, R]
    open
    close
    • Closing a pull results in a stream
    • Managed resources acquired via
    Pull.acquire
    • Pull establishes a "scope" in which
    managed resources are tracked
    • Pull forms a monad in R
    • Pull[F, A, R] can:
    • evaluate effects of type F
    • output values of type A
    • pass a resource of type R
    

    View Slide

48. Pulls and Handles
    48
    def open: Pull[F, Nothing, Handle[F,O]] = ???
    Allows us to pull elements from the stream
    

    View Slide

49. Pulls and Handles
    48
    def open: Pull[F, Nothing, Handle[F,O]] = ???
    Allows us to pull elements from the stream
    src.open.flatMap { h 
    h.receive1 { (o, _) 
    Pull.output1(o)
    }
    }.close
    head
    

    View Slide

50. Pulls and Handles
    48
    def open: Pull[F, Nothing, Handle[F,O]] = ???
    Allows us to pull elements from the stream
    src.open.flatMap { h 
    h.receive1 { (o, _) 
    Pull.output1(o)
    }
    }.close
    head
    src.open.flatMap { h 
    h.receive1 { (_, h) 
    h.echo
    }
    }.close
    tail
    

    View Slide

51. Pulls and Handles
    49
    src.open.flatMap { h 
    h.receive1 { (o, _) 
    Pull.output1(o)
    }
    }.close
    src.open.flatMap { h 
    h.receive1 { (_, h) 
    h.echo
    }
    }.close
    src.open.flatMap(f).close
    src.pull(f)
    

    View Slide

52. Recursive Pulls
    50
    def take[F[_], A](count: Int): Pipe[F, A, A] = {
    src  src.pull(???)
    }
    

    View Slide

53. Recursive Pulls
    51
    def take[F[_], A](count: Int): Pipe[F, A, A] = {
    def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  {
    }
    src  src.pull(loop(count))
    }
    

    View Slide

54. Recursive Pulls
    52
    def take[F[_], A](count: Int): Pipe[F, A, A] = {
    def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  {
    if (remaining <= 0) {
    Pull.done
    } else {
    }
    }
    src  src.pull(loop(count))
    }
    

    View Slide

55. Recursive Pulls
    53
    def take[F[_], A](count: Int): Pipe[F, A, A] = {
    def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  {
    if (remaining <= 0) {
    Pull.done
    } else {
    h.receive1 { (a, h) 
    Pull.output1(a)  loop(remaining - 1)(h)
    }
    }
    }
    src  src.pull(loop(count))
    }
    

    View Slide

56. Recursive Pulls
    54
    def take[F[_], A](count: Int): Pipe[F, A, A] = {
    def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  {
    if (remaining <= 0) {
    Pull.done
    } else {
    h.receive1 { (a, h) 
    Pull.output1(a)  loop(remaining - 1)(h)
    }
    }
    }
    src  src.pull(loop(count))
    }
    • Recursive pulls model state
    machines
    • State stored as params to
    recursive call
    • Multiple states can be
    represented as multiple co-
    recursive functions
    

    View Slide

57. Interruption
    X
    val src: Stream[Task, A] = ???
    

    View Slide

58. Interruption
    X
    val src: Stream[Task, A] = ???
    import fs2.async.mutable.Signal
    val mkCancel: Task[Signal[Task, Boolean]] =
    async.signalOf[Task, Boolean](false)
    

    View Slide

59. Interruption
    X
    val src: Stream[Task, A] = ???
    import fs2.async.mutable.Signal
    val mkCancel: Task[Signal[Task, Boolean]] =
    async.signalOf[Task, Boolean](false)
    val interruptibleSource: Stream[Task, A] =
    Stream.eval(mkCancel).flatMap { cancel 
    // Somewhere in here, call cancel.set(true)
    src.interruptWhen(cancel)
    }
    

    View Slide

60. Asynchronous Buffering
    55
    def bufferAsync[F[_], A](bufferSize: Int): Pipe[F, A, A] =
    source  ???
    

    View Slide

61. Asynchronous Buffering
    56
    def bufferAsync[F[_], A](
    bufferSize: Int)(implicit F: Async[F]
    ): Pipe[F, A, A] = source  {
    val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] =
    async.boundedQueue(bufferSize)
    }
    

    View Slide

62. Asynchronous Buffering
    57
    def bufferAsync[F[_], A](
    bufferSize: Int)(implicit F: Async[F]
    ): Pipe[F, A, A] = source  {
    val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] =
    async.boundedQueue(bufferSize)
    Stream.eval(mkQueue).flatMap { queue 
    }
    }
    

    View Slide

63. Asynchronous Buffering
    58
    def bufferAsync[F[_], A](
    bufferSize: Int)(implicit F: Async[F]
    ): Pipe[F, A, A] = source  {
    val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] =
    async.boundedQueue(bufferSize)
    Stream.eval(mkQueue).flatMap { queue 
    val fill: Stream[F, Nothing] =
    source.chunks.noneTerminate.attempt.to(queue.enqueue).drain
    }
    }
    

    View Slide

64. Asynchronous Buffering
    59
    def bufferAsync[F[_], A](
    bufferSize: Int)(implicit F: Async[F]
    ): Pipe[F, A, A] = source  {
    val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] =
    async.boundedQueue(bufferSize)
    Stream.eval(mkQueue).flatMap { queue 
    val fill: Stream[F, Nothing] =
    source.chunks.noneTerminate.attempt.to(queue.enqueue).drain
    val out: Stream[F, A] =
    queue.dequeue.through(pipe.rethrow).unNoneTerminate.
    flatMap(Stream.chunk)
    }
    }
    

    View Slide

65. Asynchronous Buffering
    60
    def bufferAsync[F[_], A](
    bufferSize: Int)(implicit F: Async[F]
    ): Pipe[F, A, A] = source  {
    val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] =
    async.boundedQueue(bufferSize)
    Stream.eval(mkQueue).flatMap { queue 
    val fill: Stream[F, Nothing] =
    source.chunks.noneTerminate.attempt.to(queue.enqueue).drain
    val out: Stream[F, A] =
    queue.dequeue.through(pipe.rethrow).unNoneTerminate.
    flatMap(Stream.chunk)
    fill merge out
    }
    }
    

    View Slide

66. Concurrency
    X
    object concurrent {
    def join[F[_]: Async, O](
    maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ???
    }
    

    View Slide

67. Concurrency
    X
    object concurrent {
    def join[F[_]: Async, O](
    maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ???
    }
    def evalMapConcurrent[F[_]: Async, A, B](
    maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] =
    src  ???
    

    View Slide

68. Concurrency
    X
    object concurrent {
    def join[F[_]: Async, O](
    maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ???
    }
    def evalMapConcurrent[F[_]: Async, A, B](
    maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] = {
    src  concurrent.join(maxOpen)(
    src.map(a  Stream.eval(f(a))))
    }
    

    View Slide

69. Type Classes from fs2.util
    61
    Functor
    Applicative
    Monad
    Traverse
    

    View Slide

70. Type Classes from fs2.util
    62
    Functor
    Applicative
    Monad
    Traverse
    Catchable
    • Tracks exceptions thrown
    during evaluation
    • Allows extraction of failure
    

    View Slide

71. Type Classes from fs2.util
    63
    Functor
    Applicative
    Monad
    Traverse
    Catchable Suspendable
    • Supports deferred
    evaluation
    • Provides suspend and
    delay methods
    

    View Slide

72. Type Classes from fs2.util
    64
    Functor
    Applicative
    Monad
    Traverse
    Catchable Suspendable
    Effect
    • Provides mechanism
    which evaluates an F[A]
    • Callback based
    unsafeRunAsync
    

    View Slide

73. Type Classes from fs2.util
    65
    Functor
    Applicative
    Monad
    Traverse
    Catchable Suspendable
    Effect
    Async
    • Effect which support
    asynchronous refs
    • Async.Ref[F, A] provides a
    "memory cell" that
    supports a variant of
    compare-and-set
    • All async primitives (e.g.,
    semaphore, signal, queue)
    built with Refs
    

    View Slide

74. So what?
    • Combinators expressed polymorphically in effect type
    • Easier to reason about what a method does when minimum type class
    constraint is made
    • Multitude of Tasks
    • fs2, scalaz, Monix
    • Multitude of type constructors
    • scodec-stream's Cursor effect
    • Task + custom behaviors (e.g., TraceTask)
    66
    

    View Slide

75. Use Case: Network Analysis Tool
    67
    Capture
    Source
    Recorder Rules
    Rules
    Rules
    UI
    Elastic
    Search
    Monitoring
    pcap
    Decoder
    Stream[Task,
    TimeStamped[Option[A]]]
    Pipe[Task,
    TimeStamped[Option[A]],
    TimeStamped[Option[A]]]
    

    View Slide

76. Use Case: Network Analysis Tool
    68
    Capture
    Source
    Recorder Rules
    Rules
    Rules
    UI
    Elastic
    Search
    Monitoring
    pcap
    Decoder
    Supports capturing via:
    • UDP datagrams sent directly to app
    • UDP or TCP monitoring via libpcap
    • Capture from coaxial RF
    • Playback of pcap file
    

    View Slide

77. Use Case: Network Analysis Tool
    69
    Capture
    Source
    Recorder Rules
    Rules
    Rules
    UI
    Elastic
    Search
    Monitoring
    pcap
    Decoder
    Supports capturing via:
    • UDP datagrams sent directly to app
    • UDP or TCP monitoring via libpcap
    • Capture from coaxial RF
    • Playback of pcap file
    udp.open[Task](addr).flatMap { socket 
    socket.reads().map { p 
    TimeStamped.now((port, p.bytes))
    }
    }
    

    View Slide

78. Use Case: Network Analysis Tool
    70
    Capture
    Source
    Recorder Rules
    Rules
    Rules
    UI
    Elastic
    Search
    Monitoring
    pcap
    Decoder
    Supports capturing via:
    • UDP datagrams sent directly to app
    • UDP or TCP monitoring via libpcap
    • Capture from coaxial RF
    • Playback of pcap file
    Stream.bracket(Pcap.open(…))(
    h  Stream.repeatEval(poll(h)),
    h  h.release
    )
    

    View Slide

79. Use Case: Network Analysis Tool
    71
    Capture
    Source
    Recorder Rules
    Rules
    Rules
    UI
    Elastic
    Search
    Monitoring
    pcap
    Decoder
    Supports capturing via:
    • UDP datagrams sent directly to app
    • UDP or TCP monitoring via libpcap
    • Capture from coaxial RF
    • Playback of pcap file
    Stream.bracket(RfDevice.open(…))(
    h  udp.open[Task](…).flatMap(…),
    h  h.release
    )
    

    View Slide

80. Use Case: Network Analysis Tool
    72
    Capture
    Source
    Recorder Rules
    Rules
    Rules
    UI
    Elastic
    Search
    Monitoring
    pcap
    Decoder
    Supports capturing via:
    • UDP datagrams sent directly to app
    • UDP or TCP monitoring via libpcap
    • Capture from coaxial RF
    • Playback of pcap file
    • Stream large files from disk
    • Filter inputs based on various criteria
    • Playback:
    • in realtime
    • multiplier of realtime
    • as fast as possible
    

    View Slide

81. PCAP File Playback
    73
    val timestamped: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] =
    timestampedDecoder.decodeMmap(new FileInputStream(path).getChannel)
    val throttlingFactor: Option[Double] = ???
    val throttled: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] =
    throttlingFactor.fold(timestamped) { factor 
    timestamped through throttled(factor)
    }
    val source: Stream[Task, TimeStamped[Option[Either[UnknownEtherType, B]]]] =
    throttled through interpolateTicks()
    

    View Slide

82. PCAP File Playback
    74
    def throttle[F[_]: Async, A](
    throttlingFactor: Double)(implicit scheduler: Scheduler
    ): Pipe[F, TimeStamped[A], TimeStamped[A]] = ???
    

    View Slide

83. PCAP File Playback
    def throttle[F[_]: Async, A](
    throttlingFactor: Double)(implicit scheduler: Scheduler
    ): Pipe[F, TimeStamped[A], TimeStamped[A]] = {
    val tickPeriod = 100.millis
    val ticks: Stream[F, Unit] =
    time.awakeEvery[F](tickPeriod).map(_  ())
    source  (source through2 ticks)(throttleWithClock(tickPeriod))
    }
    

    View Slide

84. PCAP File Playback
    76
    def throttle[F[_]: Async, A](
    throttlingFactor: Double)(implicit scheduler: Scheduler
    ): Pipe[F, TimeStamped[A], TimeStamped[A]] = {
    val tickPeriod = 100.millis
    val ticks: Stream[F, Unit] =
    time.awakeEvery[F](tickPeriod).map(_  ())
    source  (source through2 ticks)(throttleWithClock(tickPeriod))
    }
    awaitInput awaitTick
    input /
    items pending
    clock tick /
    nothing pending
    clock tick /
    output pending
    input /
    nothing pending
    

    View Slide

85. 77
    def throttleWithClock(
    tickPeriod: FiniteDuration
    ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = {
    }
    

    View Slide

86. 78
    def throttleWithClock(
    tickPeriod: FiniteDuration
    ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = {
    def awaitInput(upto: Instant): ??? =
    def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): ??? =
    }
    

    View Slide

87. 79
    def throttleWithClock(
    tickPeriod: FiniteDuration
    ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = {
    type PullFromSourceOrTicks =
    (Handle[F, TimeStamped[A]], Handle[F, Unit]) 
    Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])]
    def awaitInput(upto: Instant): PullFromSourceOrTicks =
    (src, ticks)  ???
    def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks =
    (src, ticks)  ???
    }
    

    View Slide

88. 80
    def throttleWithClock(
    tickPeriod: FiniteDuration
    ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = {
    type PullFromSourceOrTicks =
    (Handle[F, TimeStamped[A]], Handle[F, Unit]) 
    Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])]
    def awaitInput(upto: Instant): PullFromSourceOrTicks =
    (src, ticks)  src.receive { (chunk, tl)  ??? }
    def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks =
    (src, ticks)  ???
    }
    

    View Slide

89. 81
    def throttleWithClock(
    tickPeriod: FiniteDuration
    ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = {
    type PullFromSourceOrTicks =
    (Handle[F, TimeStamped[A]], Handle[F, Unit]) 
    Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])]
    def awaitInput(upto: Instant): PullFromSourceOrTicks =
    (src, ticks)  src.receive { (chunk, tl)  ??? }
    def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks =
    (src, ticks)  ticks.receive1 { (tick, tl)  ??? }
    }
    

    View Slide

90. 82
    def throttleWithClock(
    tickPeriod: FiniteDuration
    ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = {
    type PullFromSourceOrTicks =
    (Handle[F, TimeStamped[A]], Handle[F, Unit]) 
    Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])]
    def awaitInput(upto: Instant): PullFromSourceOrTicks =
    (src, ticks)  src.receive { (chunk, tl)  ??? }
    def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks =
    (src, ticks)  ticks.receive1 { (tick, tl)  ??? }
    _.pull2(_) { (src, ticks)  src.receive1 { (ta, tl) 
    Pull.output1(ta)  awaitInput(ta.time)(tl, ticks)
    }}
    }
    

    View Slide

91. Compositional Streaming with FS2
    • 0.9.2 available now for Scala 2.11 & 2.12
    • 1.0.0 in progress — source compatible with 0.9
    • Vibrant Ecosystem
    • Interop: scalaz-stream, akka-stream, cats, scalaz
    • Infrastructure: doobie, http4s, kafka
    • Contact me via Gitter.im chat, Github issue tracker, or @mpilquist
    83
    

    View Slide