Compositional Streaming with FS2

Compositional Streaming with FS2

In recent years, a number of open source Scala libraries have appeared that support working with data streams. In this talk, we’ll look at Functional Streams for Scala (FS2), the library formerly known as Scalaz Stream, and explore its unique take on stream processing. We’ll look at working with impure data sources in a pure world, data transformations, and patterns for stream based program design.

Presented at Scalae By The Bay 2016 in San Francisco on November 13, 2016.

C9ab1175a6981a2f67ce8d08aa17c15a?s=128

Michael Pilquist

November 13, 2016
Tweet

Transcript

  1. Compositional Streaming with FS2 Michael Pilquist // @mpilquist Scala By

    The Bay November 2016
  2. scalaz-stream => fs2 2

  3. scalaz-stream => fs2 • Dependencies: scalaz & scodec-bits • Actively

    maintained! • scalaz 7.{1,2,3} • scala 2.{10,11,12} 3 libraryDependencies += "org.scalaz.stream" %% "scalaz-stream" % "0.8.6" libraryDependencies += "co.fs2" %% "fs2-core" % "0.9.2" • No dependencies • Supports Scala.js • Bindings available for Scalaz and Cats • Significantly improved API • Smaller core interpreter • Correct • More parametricity https://github.com/functional-streams-for-scala/fs2 series/0.8 series/0.9
  4. Stream 4 class Stream[+F[_], +O] Output values Effects

  5. Pure Streams 5 scala> val s = Stream(1, 2, 3)

    s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))
  6. Pure Streams 6 scala> val s = Stream(1, 2, 3)

    s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30)
  7. Pure Streams 7 scala> val s = Stream(1, 2, 3)

    s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30) scala> val ys = s.intersperse(0).toList ys: List[Int] = List(1, 0, 2, 0, 3)
  8. Pure Streams 8 scala> val s = Stream(1, 2, 3)

    s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30) scala> val ys = s.intersperse(0).toList ys: List[Int] = List(1, 0, 2, 0, 3) toList is only available on pure streams
  9. Pure Streams 9 scala> val fibs: Stream[Nothing, Int] = Stream(0,

    1)  fibs.zipWith(fibs.tail)(_ + _) fibs: fs2.Stream[Nothing,Int] =  scala> fibs.take(10).toList // CAUTION: *NOT* Memoized! res0: List[Int] = List(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)
  10. Effectful Streams 10 val sample: Task[Sample] = Task.delay { sensor.read()

    }
  11. Effectful Streams 11 val sample: Task[Sample] = Task.delay { sensor.read()

    } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat
  12. Effectful Streams 12 val sample: Task[Sample] = Task.delay { sensor.read()

    } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat val firstTen: Task[Vector[Sample]] = samples.take(10).runLog
  13. Effectful Streams 13 val sample: Task[Sample] = Task.delay { sensor.read()

    } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat val firstTen: Task[Vector[Sample]] = samples.take(10).runLog val result: Vector[Sample] = firstTen.unsafeRun()
  14. Running Effectful Streams 14 implicit class StreamRunOps[F[_]: Catchable, O]( val

    self: Stream[F,O] ) { def run: F[Unit] def runLog: F[Vector[O]] def runLast: F[Option[O]] def runFold[A](z: A)(f: (A,O)  A): F[A] }
  15. Transforming Streams 15 Pipe[F[_], -I, +O] Output values Effects Input

    values
  16. Pipe Examples 16 object text { def utf8Decode[F[_]]: Pipe[F, Byte,

    String] = ??? def lines[F[_]]: Pipe[F, String, String] = ??? } val linesOfFile: Stream[Task, String] = io.file.readAll[Task](myFile, 4096). through(text.utf8Decode). through(text.lines)
  17. Combining Streams 17 Pipe2[F[_], -I, -I2, +O] Output values Effects

    Input values
  18. Pipe2 Examples 18 fibs.zipWith(fibs.tail)(_+_)

  19. Pipe2 Examples 19 fibs.zipWith(fibs.tail)(_+_) // alternatively fibs.pure.through2(fibs.tail)(pipe2.zipWith(_+_))

  20. Pipe2 Examples 20 val x = Stream(1, 2, 3) val

    y = Stream(4, 5, 6) val z = x interleave y val elems = z.toList // elems: List[String] = List(1, 4, 2, 5, 3, 6)
  21. Pipe2 Examples 21 val x = Stream(1, 2, 3) val

    y = Stream(4, 5, 6) val z = x interleave y // alternatively x.pure.through2(y)(pipe2.interleave) val elems = z.toList // elems: List[String] = List(1, 4, 2, 5, 3, 6)
  22. Sinking Streams 22 Sink[F[_], -I] Effects Input values

  23. Sink Examples 23 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile)

  24. Sink Examples 24 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val

    toFile: Stream[Task, Unit] = lines.to(outputFile)
  25. Sink Examples 25 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val

    toFile: Stream[Task, Unit] = lines.to(outputFile) val prg: Task[Unit] = toFile.run
  26. Sink Examples 26 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val

    toFile: Stream[Task, Unit] = lines.to(outputFile) val prg: Task[Unit] = toFile.run prg.unsafeRun()
  27. Sink Examples: Writing events to Kafka 27 val kafkaTopic: Sink[Task,

    Record] = Producer.sink(Topic("foo"), kafkaSettings)
  28. Sink Examples: Writing events to Kafka 28 val kafkaTopic: Sink[Task,

    Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ???
  29. Sink Examples: Writing events to Kafka 29 val kafkaTopic: Sink[Task,

    Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ??? val program: Stream[Task, Unit] = events.map(eventToRecord).to(kafkaTopic)
  30. Sink Examples: Writing events to Kafka 30 val kafkaTopic: Sink[Task,

    Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ??? val program: Stream[Task, Unit] = events.map(eventToRecord).to(kafkaTopic) program.run.unsafeRun()
  31. Composing Stream Transformations 31 Do pipes compose as well as

    functions?
  32. Composing Stream Transformations 32 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O]

    Do pipes compose as well as functions?
  33. Composing Stream Transformations 33 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O]

    type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] Do pipes compose as well as functions?
  34. Composing Stream Transformations 34 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O]

    type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] type Sink[F[_],-I] = Pipe[F,I,Unit] Do pipes compose as well as functions?
  35. Composing Stream Transformations 35 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O]

    type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] type Sink[F[_],-I] = Pipe[F,I,Unit] // Function composition! src.through(text.utfDecode andThen text.lines) Do pipes compose as well as functions?
  36. README 36 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
  37. README 37 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines).
  38. README 38 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")).
  39. README 39 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString).
  40. README 40 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n").
  41. README 41 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode).
  42. README 42 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))).
  43. README 43 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run
  44. README 44 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run converter.unsafeRun()
  45. README 45 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double):

    Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run converter.unsafeRun() • Input file is read & output file is written in a streamy fashion • Constant memory usage • Errors are handled • I/O errors • Exceptions thrown by function passed to map • File handles are acquired when necessary and are guaranteed to be released
  46. Resource Allocation 46 def bracket[F[_],R,A](acquire: F[R])( use: R  Stream[F,A],

    release: R  F[Unit] ): Stream[F,A] • Evaluates acquire to produce a resource • Passes resource to use to generate a stream • Decorates the stream returned from use so that the resource is released when the inner stream terminates • Stream termination occurs when: • Natural end of stream is reached • Puller of stream decides to stop pulling (e.g., take(5)) • Unhandled error occurs
  47. Pulls and Handles 47 Stream[F, A] Pull[F, A, R] open

    close • Closing a pull results in a stream • Managed resources acquired via Pull.acquire • Pull establishes a "scope" in which managed resources are tracked • Pull forms a monad in R • Pull[F, A, R] can: • evaluate effects of type F • output values of type A • pass a resource of type R
  48. Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] =

    ??? Allows us to pull elements from the stream
  49. Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] =

    ??? Allows us to pull elements from the stream src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close head
  50. Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] =

    ??? Allows us to pull elements from the stream src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close head src.open.flatMap { h  h.receive1 { (_, h)  h.echo } }.close tail
  51. Pulls and Handles 49 src.open.flatMap { h  h.receive1 {

    (o, _)  Pull.output1(o) } }.close src.open.flatMap { h  h.receive1 { (_, h)  h.echo } }.close src.open.flatMap(f).close src.pull(f)
  52. Recursive Pulls 50 def take[F[_], A](count: Int): Pipe[F, A, A]

    = { src  src.pull(???) }
  53. Recursive Pulls 51 def take[F[_], A](count: Int): Pipe[F, A, A]

    = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { } src  src.pull(loop(count)) }
  54. Recursive Pulls 52 def take[F[_], A](count: Int): Pipe[F, A, A]

    = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { } } src  src.pull(loop(count)) }
  55. Recursive Pulls 53 def take[F[_], A](count: Int): Pipe[F, A, A]

    = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { h.receive1 { (a, h)  Pull.output1(a)  loop(remaining - 1)(h) } } } src  src.pull(loop(count)) }
  56. Recursive Pulls 54 def take[F[_], A](count: Int): Pipe[F, A, A]

    = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { h.receive1 { (a, h)  Pull.output1(a)  loop(remaining - 1)(h) } } } src  src.pull(loop(count)) } • Recursive pulls model state machines • State stored as params to recursive call • Multiple states can be represented as multiple co- recursive functions
  57. Interruption X val src: Stream[Task, A] = ???

  58. Interruption X val src: Stream[Task, A] = ??? import fs2.async.mutable.Signal

    val mkCancel: Task[Signal[Task, Boolean]] = async.signalOf[Task, Boolean](false)
  59. Interruption X val src: Stream[Task, A] = ??? import fs2.async.mutable.Signal

    val mkCancel: Task[Signal[Task, Boolean]] = async.signalOf[Task, Boolean](false) val interruptibleSource: Stream[Task, A] = Stream.eval(mkCancel).flatMap { cancel  // Somewhere in here, call cancel.set(true) src.interruptWhen(cancel) }
  60. Asynchronous Buffering 55 def bufferAsync[F[_], A](bufferSize: Int): Pipe[F, A, A]

    = source  ???
  61. Asynchronous Buffering 56 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F]

    ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) }
  62. Asynchronous Buffering 57 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F]

    ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  } }
  63. Asynchronous Buffering 58 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F]

    ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain } }
  64. Asynchronous Buffering 59 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F]

    ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain val out: Stream[F, A] = queue.dequeue.through(pipe.rethrow).unNoneTerminate. flatMap(Stream.chunk) } }
  65. Asynchronous Buffering 60 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F]

    ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain val out: Stream[F, A] = queue.dequeue.through(pipe.rethrow).unNoneTerminate. flatMap(Stream.chunk) fill merge out } }
  66. Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen:

    Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? }
  67. Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen:

    Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? } def evalMapConcurrent[F[_]: Async, A, B]( maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] = src  ???
  68. Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen:

    Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? } def evalMapConcurrent[F[_]: Async, A, B]( maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] = { src  concurrent.join(maxOpen)( src.map(a  Stream.eval(f(a)))) }
  69. Type Classes from fs2.util 61 Functor Applicative Monad Traverse

  70. Type Classes from fs2.util 62 Functor Applicative Monad Traverse Catchable

    • Tracks exceptions thrown during evaluation • Allows extraction of failure
  71. Type Classes from fs2.util 63 Functor Applicative Monad Traverse Catchable

    Suspendable • Supports deferred evaluation • Provides suspend and delay methods
  72. Type Classes from fs2.util 64 Functor Applicative Monad Traverse Catchable

    Suspendable Effect • Provides mechanism which evaluates an F[A] • Callback based unsafeRunAsync
  73. Type Classes from fs2.util 65 Functor Applicative Monad Traverse Catchable

    Suspendable Effect Async • Effect which support asynchronous refs • Async.Ref[F, A] provides a "memory cell" that supports a variant of compare-and-set • All async primitives (e.g., semaphore, signal, queue) built with Refs
  74. So what? • Combinators expressed polymorphically in effect type •

    Easier to reason about what a method does when minimum type class constraint is made • Multitude of Tasks • fs2, scalaz, Monix • Multitude of type constructors • scodec-stream's Cursor effect • Task + custom behaviors (e.g., TraceTask) 66
  75. Use Case: Network Analysis Tool 67 Capture Source Recorder Rules

    Rules Rules UI Elastic Search Monitoring pcap Decoder Stream[Task, TimeStamped[Option[A]]] Pipe[Task, TimeStamped[Option[A]], TimeStamped[Option[A]]]
  76. Use Case: Network Analysis Tool 68 Capture Source Recorder Rules

    Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file
  77. Use Case: Network Analysis Tool 69 Capture Source Recorder Rules

    Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file udp.open[Task](addr).flatMap { socket  socket.reads().map { p  TimeStamped.now((port, p.bytes)) } }
  78. Use Case: Network Analysis Tool 70 Capture Source Recorder Rules

    Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file Stream.bracket(Pcap.open(…))( h  Stream.repeatEval(poll(h)), h  h.release )
  79. Use Case: Network Analysis Tool 71 Capture Source Recorder Rules

    Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file Stream.bracket(RfDevice.open(…))( h  udp.open[Task](…).flatMap(…), h  h.release )
  80. Use Case: Network Analysis Tool 72 Capture Source Recorder Rules

    Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file • Stream large files from disk • Filter inputs based on various criteria • Playback: • in realtime • multiplier of realtime • as fast as possible
  81. PCAP File Playback 73 val timestamped: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] =

    timestampedDecoder.decodeMmap(new FileInputStream(path).getChannel) val throttlingFactor: Option[Double] = ??? val throttled: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] = throttlingFactor.fold(timestamped) { factor  timestamped through throttled(factor) } val source: Stream[Task, TimeStamped[Option[Either[UnknownEtherType, B]]]] = throttled through interpolateTicks()
  82. PCAP File Playback 74 def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit

    scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = ???
  83. PCAP File Playback def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler:

    Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = { val tickPeriod = 100.millis val ticks: Stream[F, Unit] = time.awakeEvery[F](tickPeriod).map(_  ()) source  (source through2 ticks)(throttleWithClock(tickPeriod)) }
  84. PCAP File Playback 76 def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit

    scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = { val tickPeriod = 100.millis val ticks: Stream[F, Unit] = time.awakeEvery[F](tickPeriod).map(_  ()) source  (source through2 ticks)(throttleWithClock(tickPeriod)) } awaitInput awaitTick input / items pending clock tick / nothing pending clock tick / output pending input / nothing pending
  85. 77 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]]

    = { }
  86. 78 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]]

    = { def awaitInput(upto: Instant): ??? = def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): ??? = }
  87. 79 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]]

    = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  ??? def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ??? }
  88. 80 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]]

    = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ??? }
  89. 81 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]]

    = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ticks.receive1 { (tick, tl)  ??? } }
  90. 82 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]]

    = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ticks.receive1 { (tick, tl)  ??? } _.pull2(_) { (src, ticks)  src.receive1 { (ta, tl)  Pull.output1(ta)  awaitInput(ta.time)(tl, ticks) }} }
  91. Compositional Streaming with FS2 • 0.9.2 available now for Scala

    2.11 & 2.12 • 1.0.0 in progress — source compatible with 0.9 • Vibrant Ecosystem • Interop: scalaz-stream, akka-stream, cats, scalaz • Infrastructure: doobie, http4s, kafka • Contact me via Gitter.im chat, Github issue tracker, or @mpilquist 83