Compositional Streaming with FS2

In recent years, a number of open source Scala libraries have appeared that support working with data streams. In this talk, we’ll look at Functional Streams for Scala (FS2), the library formerly known as Scalaz Stream, and explore its unique take on stream processing. We’ll look at working with impure data sources in a pure world, data transformations, and patterns for stream based program design.

Presented at Scalae By The Bay 2016 in San Francisco on November 13, 2016.

Transcript

1. Compositional Streaming with FS2 Michael Pilquist // @mpilquist Scala By The Bay November 2016
2. scalaz-stream => fs2 2
3. scalaz-stream => fs2 • Dependencies: scalaz & scodec-bits • Actively maintained! • scalaz 7.{1,2,3} • scala 2.{10,11,12} 3 libraryDependencies += "org.scalaz.stream" %% "scalaz-stream" % "0.8.6" libraryDependencies += "co.fs2" %% "fs2-core" % "0.9.2" • No dependencies • Supports Scala.js • Bindings available for Scalaz and Cats • Significantly improved API • Smaller core interpreter • Correct • More parametricity https://github.com/functional-streams-for-scala/fs2 series/0.8 series/0.9
4. Stream 4 class Stream[+F[_], +O] Output values Effects
5. Pure Streams 5 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))
6. Pure Streams 6 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30)
7. Pure Streams 7 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30) scala> val ys = s.intersperse(0).toList ys: List[Int] = List(1, 0, 2, 0, 3)
8. Pure Streams 8 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30) scala> val ys = s.intersperse(0).toList ys: List[Int] = List(1, 0, 2, 0, 3) toList is only available on pure streams
9. Pure Streams 9 scala> val fibs: Stream[Nothing, Int] = Stream(0, 1)  fibs.zipWith(fibs.tail)(_ + _) fibs: fs2.Stream[Nothing,Int] =  scala> fibs.take(10).toList // CAUTION: *NOT* Memoized! res0: List[Int] = List(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)
10. Effectful Streams 10 val sample: Task[Sample] = Task.delay { sensor.read() }
11. Effectful Streams 11 val sample: Task[Sample] = Task.delay { sensor.read() } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat
12. Effectful Streams 12 val sample: Task[Sample] = Task.delay { sensor.read() } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat val firstTen: Task[Vector[Sample]] = samples.take(10).runLog
13. Effectful Streams 13 val sample: Task[Sample] = Task.delay { sensor.read() } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat val firstTen: Task[Vector[Sample]] = samples.take(10).runLog val result: Vector[Sample] = firstTen.unsafeRun()
14. Running Effectful Streams 14 implicit class StreamRunOps[F[_]: Catchable, O]( val self: Stream[F,O] ) { def run: F[Unit] def runLog: F[Vector[O]] def runLast: F[Option[O]] def runFold[A](z: A)(f: (A,O)  A): F[A] }
15. Transforming Streams 15 Pipe[F[_], -I, +O] Output values Effects Input values
16. Pipe Examples 16 object text { def utf8Decode[F[_]]: Pipe[F, Byte, String] = ??? def lines[F[_]]: Pipe[F, String, String] = ??? } val linesOfFile: Stream[Task, String] = io.file.readAll[Task](myFile, 4096). through(text.utf8Decode). through(text.lines)
17. Combining Streams 17 Pipe2[F[_], -I, -I2, +O] Output values Effects Input values
18. Pipe2 Examples 18 fibs.zipWith(fibs.tail)(_+_)
19. Pipe2 Examples 19 fibs.zipWith(fibs.tail)(_+_) // alternatively fibs.pure.through2(fibs.tail)(pipe2.zipWith(_+_))
20. Pipe2 Examples 20 val x = Stream(1, 2, 3) val y = Stream(4, 5, 6) val z = x interleave y val elems = z.toList // elems: List[String] = List(1, 4, 2, 5, 3, 6)
21. Pipe2 Examples 21 val x = Stream(1, 2, 3) val y = Stream(4, 5, 6) val z = x interleave y // alternatively x.pure.through2(y)(pipe2.interleave) val elems = z.toList // elems: List[String] = List(1, 4, 2, 5, 3, 6)
22. Sinking Streams 22 Sink[F[_], -I] Effects Input values
23. Sink Examples 23 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile)
24. Sink Examples 24 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val toFile: Stream[Task, Unit] = lines.to(outputFile)
25. Sink Examples 25 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val toFile: Stream[Task, Unit] = lines.to(outputFile) val prg: Task[Unit] = toFile.run
26. Sink Examples 26 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val toFile: Stream[Task, Unit] = lines.to(outputFile) val prg: Task[Unit] = toFile.run prg.unsafeRun()
27. Sink Examples: Writing events to Kafka 27 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings)
28. Sink Examples: Writing events to Kafka 28 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ???
29. Sink Examples: Writing events to Kafka 29 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ??? val program: Stream[Task, Unit] = events.map(eventToRecord).to(kafkaTopic)
30. Sink Examples: Writing events to Kafka 30 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ??? val program: Stream[Task, Unit] = events.map(eventToRecord).to(kafkaTopic) program.run.unsafeRun()
31. Composing Stream Transformations 31 Do pipes compose as well as functions?
32. Composing Stream Transformations 32 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] Do pipes compose as well as functions?
33. Composing Stream Transformations 33 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] Do pipes compose as well as functions?
34. Composing Stream Transformations 34 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] type Sink[F[_],-I] = Pipe[F,I,Unit] Do pipes compose as well as functions?
35. Composing Stream Transformations 35 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] type Sink[F[_],-I] = Pipe[F,I,Unit] // Function composition! src.through(text.utfDecode andThen text.lines) Do pipes compose as well as functions?
36. README 36 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).
37. README 37 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines).
38. README 38 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")).
39. README 39 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString).
40. README 40 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n").
41. README 41 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode).
42. README 42 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))).
43. README 43 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run
44. README 44 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run converter.unsafeRun()
45. README 45 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run converter.unsafeRun() • Input file is read & output file is written in a streamy fashion • Constant memory usage • Errors are handled • I/O errors • Exceptions thrown by function passed to map • File handles are acquired when necessary and are guaranteed to be released
46. Resource Allocation 46 def bracket[F[_],R,A](acquire: F[R])( use: R  Stream[F,A], release: R  F[Unit] ): Stream[F,A] • Evaluates acquire to produce a resource • Passes resource to use to generate a stream • Decorates the stream returned from use so that the resource is released when the inner stream terminates • Stream termination occurs when: • Natural end of stream is reached • Puller of stream decides to stop pulling (e.g., take(5)) • Unhandled error occurs
47. Pulls and Handles 47 Stream[F, A] Pull[F, A, R] open close • Closing a pull results in a stream • Managed resources acquired via Pull.acquire • Pull establishes a "scope" in which managed resources are tracked • Pull forms a monad in R • Pull[F, A, R] can: • evaluate effects of type F • output values of type A • pass a resource of type R
48. Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] = ??? Allows us to pull elements from the stream
49. Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] = ??? Allows us to pull elements from the stream src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close head
50. Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] = ??? Allows us to pull elements from the stream src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close head src.open.flatMap { h  h.receive1 { (_, h)  h.echo } }.close tail
51. Pulls and Handles 49 src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close src.open.flatMap { h  h.receive1 { (_, h)  h.echo } }.close src.open.flatMap(f).close src.pull(f)
52. Recursive Pulls 50 def take[F[_], A](count: Int): Pipe[F, A, A] = { src  src.pull(???) }
53. Recursive Pulls 51 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { } src  src.pull(loop(count)) }
54. Recursive Pulls 52 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { } } src  src.pull(loop(count)) }
55. Recursive Pulls 53 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { h.receive1 { (a, h)  Pull.output1(a)  loop(remaining - 1)(h) } } } src  src.pull(loop(count)) }
56. Recursive Pulls 54 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { h.receive1 { (a, h)  Pull.output1(a)  loop(remaining - 1)(h) } } } src  src.pull(loop(count)) } • Recursive pulls model state machines • State stored as params to recursive call • Multiple states can be represented as multiple co- recursive functions
57. Interruption X val src: Stream[Task, A] = ???
58. Interruption X val src: Stream[Task, A] = ??? import fs2.async.mutable.Signal val mkCancel: Task[Signal[Task, Boolean]] = async.signalOf[Task, Boolean](false)
59. Interruption X val src: Stream[Task, A] = ??? import fs2.async.mutable.Signal val mkCancel: Task[Signal[Task, Boolean]] = async.signalOf[Task, Boolean](false) val interruptibleSource: Stream[Task, A] = Stream.eval(mkCancel).flatMap { cancel  // Somewhere in here, call cancel.set(true) src.interruptWhen(cancel) }
60. Asynchronous Buffering 55 def bufferAsync[F[_], A](bufferSize: Int): Pipe[F, A, A] = source  ???
61. Asynchronous Buffering 56 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) }
62. Asynchronous Buffering 57 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  } }
63. Asynchronous Buffering 58 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain } }
64. Asynchronous Buffering 59 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain val out: Stream[F, A] = queue.dequeue.through(pipe.rethrow).unNoneTerminate. flatMap(Stream.chunk) } }
65. Asynchronous Buffering 60 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain val out: Stream[F, A] = queue.dequeue.through(pipe.rethrow).unNoneTerminate. flatMap(Stream.chunk) fill merge out } }
66. Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? }
67. Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? } def evalMapConcurrent[F[_]: Async, A, B]( maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] = src  ???
68. Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? } def evalMapConcurrent[F[_]: Async, A, B]( maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] = { src  concurrent.join(maxOpen)( src.map(a  Stream.eval(f(a)))) }
69. Type Classes from fs2.util 61 Functor Applicative Monad Traverse
70. Type Classes from fs2.util 62 Functor Applicative Monad Traverse Catchable • Tracks exceptions thrown during evaluation • Allows extraction of failure
71. Type Classes from fs2.util 63 Functor Applicative Monad Traverse Catchable Suspendable • Supports deferred evaluation • Provides suspend and delay methods
72. Type Classes from fs2.util 64 Functor Applicative Monad Traverse Catchable Suspendable Effect • Provides mechanism which evaluates an F[A] • Callback based unsafeRunAsync
73. Type Classes from fs2.util 65 Functor Applicative Monad Traverse Catchable Suspendable Effect Async • Effect which support asynchronous refs • Async.Ref[F, A] provides a "memory cell" that supports a variant of compare-and-set • All async primitives (e.g., semaphore, signal, queue) built with Refs
74. So what? • Combinators expressed polymorphically in effect type • Easier to reason about what a method does when minimum type class constraint is made • Multitude of Tasks • fs2, scalaz, Monix • Multitude of type constructors • scodec-stream's Cursor effect • Task + custom behaviors (e.g., TraceTask) 66
75. Use Case: Network Analysis Tool 67 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Stream[Task, TimeStamped[Option[A]]] Pipe[Task, TimeStamped[Option[A]], TimeStamped[Option[A]]]
76. Use Case: Network Analysis Tool 68 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file
77. Use Case: Network Analysis Tool 69 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file udp.open[Task](addr).flatMap { socket  socket.reads().map { p  TimeStamped.now((port, p.bytes)) } }
78. Use Case: Network Analysis Tool 70 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file Stream.bracket(Pcap.open(…))( h  Stream.repeatEval(poll(h)), h  h.release )
79. Use Case: Network Analysis Tool 71 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file Stream.bracket(RfDevice.open(…))( h  udp.open[Task](…).flatMap(…), h  h.release )
80. Use Case: Network Analysis Tool 72 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file • Stream large files from disk • Filter inputs based on various criteria • Playback: • in realtime • multiplier of realtime • as fast as possible
81. PCAP File Playback 73 val timestamped: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] = timestampedDecoder.decodeMmap(new FileInputStream(path).getChannel) val throttlingFactor: Option[Double] = ??? val throttled: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] = throttlingFactor.fold(timestamped) { factor  timestamped through throttled(factor) } val source: Stream[Task, TimeStamped[Option[Either[UnknownEtherType, B]]]] = throttled through interpolateTicks()
82. PCAP File Playback 74 def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = ???
83. PCAP File Playback def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = { val tickPeriod = 100.millis val ticks: Stream[F, Unit] = time.awakeEvery[F](tickPeriod).map(_  ()) source  (source through2 ticks)(throttleWithClock(tickPeriod)) }
84. PCAP File Playback 76 def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = { val tickPeriod = 100.millis val ticks: Stream[F, Unit] = time.awakeEvery[F](tickPeriod).map(_  ()) source  (source through2 ticks)(throttleWithClock(tickPeriod)) } awaitInput awaitTick input / items pending clock tick / nothing pending clock tick / output pending input / nothing pending
85. 77 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { }
86. 78 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { def awaitInput(upto: Instant): ??? = def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): ??? = }
87. 79 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  ??? def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ??? }
88. 80 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ??? }
89. 81 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ticks.receive1 { (tick, tl)  ??? } }
90. 82 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ticks.receive1 { (tick, tl)  ??? } _.pull2(_) { (src, ticks)  src.receive1 { (ta, tl)  Pull.output1(ta)  awaitInput(ta.time)(tl, ticks) }} }
91. Compositional Streaming with FS2 • 0.9.2 available now for Scala 2.11 & 2.12 • 1.0.0 in progress — source compatible with 0.9 • Vibrant Ecosystem • Interop: scalaz-stream, akka-stream, cats, scalaz • Infrastructure: doobie, http4s, kafka • Contact me via Gitter.im chat, Github issue tracker, or @mpilquist 83