Compositional Streaming with FS2

Slide 1

Slide 1 text

Compositional Streaming with FS2 Michael Pilquist // @mpilquist Scala By The Bay November 2016

Slide 2

Slide 2 text

scalaz-stream => fs2 2

Slide 3

Slide 3 text

scalaz-stream => fs2 • Dependencies: scalaz & scodec-bits • Actively maintained! • scalaz 7.{1,2,3} • scala 2.{10,11,12} 3 libraryDependencies += "org.scalaz.stream" %% "scalaz-stream" % "0.8.6" libraryDependencies += "co.fs2" %% "fs2-core" % "0.9.2" • No dependencies • Supports Scala.js • Bindings available for Scalaz and Cats • Significantly improved API • Smaller core interpreter • Correct • More parametricity https://github.com/functional-streams-for-scala/fs2 series/0.8 series/0.9

Slide 4

Slide 4 text

Stream 4 class Stream[+F[_], +O] Output values Effects

Slide 5

Slide 5 text

Pure Streams 5 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3)))

Slide 6

Slide 6 text

Pure Streams 6 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30)

Slide 7

Slide 7 text

Pure Streams 7 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30) scala> val ys = s.intersperse(0).toList ys: List[Int] = List(1, 0, 2, 0, 3)

Slide 8

Slide 8 text

Pure Streams 8 scala> val s = Stream(1, 2, 3) s: fs2.Stream[Nothing,Int] = Segment(Emit(Chunk(1, 2, 3))) scala> val xs = (s  s.map(_*10)).toList xs: List[Int] = List(1, 2, 3, 10, 20, 30) scala> val ys = s.intersperse(0).toList ys: List[Int] = List(1, 0, 2, 0, 3) toList is only available on pure streams

Slide 9

Slide 9 text

Pure Streams 9 scala> val fibs: Stream[Nothing, Int] = Stream(0, 1)  fibs.zipWith(fibs.tail)(_ + _) fibs: fs2.Stream[Nothing,Int] =  scala> fibs.take(10).toList // CAUTION: *NOT* Memoized! res0: List[Int] = List(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)

Slide 10

Slide 10 text

Effectful Streams 10 val sample: Task[Sample] = Task.delay { sensor.read() }

Slide 11

Slide 11 text

Effectful Streams 11 val sample: Task[Sample] = Task.delay { sensor.read() } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat

Slide 12

Slide 12 text

Effectful Streams 12 val sample: Task[Sample] = Task.delay { sensor.read() } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat val firstTen: Task[Vector[Sample]] = samples.take(10).runLog

Slide 13

Slide 13 text

Effectful Streams 13 val sample: Task[Sample] = Task.delay { sensor.read() } val samples: Stream[Task, Sample] = Stream.eval(sample).repeat val firstTen: Task[Vector[Sample]] = samples.take(10).runLog val result: Vector[Sample] = firstTen.unsafeRun()

Slide 14

Slide 14 text

Running Effectful Streams 14 implicit class StreamRunOps[F[_]: Catchable, O]( val self: Stream[F,O] ) { def run: F[Unit] def runLog: F[Vector[O]] def runLast: F[Option[O]] def runFold[A](z: A)(f: (A,O)  A): F[A] }

Slide 15

Slide 15 text

Transforming Streams 15 Pipe[F[_], -I, +O] Output values Effects Input values

Slide 16

Slide 16 text

Pipe Examples 16 object text { def utf8Decode[F[_]]: Pipe[F, Byte, String] = ??? def lines[F[_]]: Pipe[F, String, String] = ??? } val linesOfFile: Stream[Task, String] = io.file.readAll[Task](myFile, 4096). through(text.utf8Decode). through(text.lines)

Slide 17

Slide 17 text

Combining Streams 17 Pipe2[F[_], -I, -I2, +O] Output values Effects Input values

Slide 18

Slide 18 text

Pipe2 Examples 18 fibs.zipWith(fibs.tail)(_+_)

Slide 19

Slide 19 text

Pipe2 Examples 19 fibs.zipWith(fibs.tail)(_+_) // alternatively fibs.pure.through2(fibs.tail)(pipe2.zipWith(_+_))

Slide 20

Slide 20 text

Pipe2 Examples 20 val x = Stream(1, 2, 3) val y = Stream(4, 5, 6) val z = x interleave y val elems = z.toList // elems: List[String] = List(1, 4, 2, 5, 3, 6)

Slide 21

Slide 21 text

Pipe2 Examples 21 val x = Stream(1, 2, 3) val y = Stream(4, 5, 6) val z = x interleave y // alternatively x.pure.through2(y)(pipe2.interleave) val elems = z.toList // elems: List[String] = List(1, 4, 2, 5, 3, 6)

Slide 22

Slide 22 text

Sinking Streams 22 Sink[F[_], -I] Effects Input values

Slide 23

Slide 23 text

Sink Examples 23 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile)

Slide 24

Slide 24 text

Sink Examples 24 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val toFile: Stream[Task, Unit] = lines.to(outputFile)

Slide 25

Slide 25 text

Sink Examples 25 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val toFile: Stream[Task, Unit] = lines.to(outputFile) val prg: Task[Unit] = toFile.run

Slide 26

Slide 26 text

Sink Examples 26 val outputFile: Sink[Task, Byte] = io.file.writeAll[Task](myFile) val toFile: Stream[Task, Unit] = lines.to(outputFile) val prg: Task[Unit] = toFile.run prg.unsafeRun()

Slide 27

Slide 27 text

Sink Examples: Writing events to Kafka 27 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings)

Slide 28

Slide 28 text

Sink Examples: Writing events to Kafka 28 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ???

Slide 29

Slide 29 text

Sink Examples: Writing events to Kafka 29 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ??? val program: Stream[Task, Unit] = events.map(eventToRecord).to(kafkaTopic)

Slide 30

Slide 30 text

Sink Examples: Writing events to Kafka 30 val kafkaTopic: Sink[Task, Record] = Producer.sink(Topic("foo"), kafkaSettings) val events: Stream[Task, Event] = ??? val program: Stream[Task, Unit] = events.map(eventToRecord).to(kafkaTopic) program.run.unsafeRun()

Slide 31

Slide 31 text

Composing Stream Transformations 31 Do pipes compose as well as functions?

Slide 32

Slide 32 text

Composing Stream Transformations 32 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] Do pipes compose as well as functions?

Slide 33

Slide 33 text

Composing Stream Transformations 33 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] Do pipes compose as well as functions?

Slide 34

Slide 34 text

Composing Stream Transformations 34 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] type Sink[F[_],-I] = Pipe[F,I,Unit] Do pipes compose as well as functions?

Slide 35

Slide 35 text

Composing Stream Transformations 35 type Pipe[F[_],-I,+O] = Stream[F,I]  Stream[F,O] type Pipe2[F[_],-I,-I2,+O] = (Stream[F,I], Stream[F,I2])  Stream[F,O] type Sink[F[_],-I] = Pipe[F,I,Unit] // Function composition! src.through(text.utfDecode andThen text.lines) Do pipes compose as well as functions?

Slide 36

Slide 36 text

README 36 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096).

Slide 37

Slide 37 text

README 37 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines).

Slide 38

Slide 38 text

README 38 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")).

Slide 39

Slide 39 text

README 39 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString).

Slide 40

Slide 40 text

README 40 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n").

Slide 41

Slide 41 text

README 41 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode).

Slide 42

Slide 42 text

README 42 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))).

Slide 43

Slide 43 text

README 43 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run

Slide 44

Slide 44 text

README 44 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run converter.unsafeRun()

Slide 45

Slide 45 text

README 45 import fs2.{io, text, Task}, java.nio.file.Paths def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) val converter: Task[Unit] = io.file.readAll[Task](Paths.get("fahrenheit.txt"), 4096). through(text.utf8Decode andThen text.lines). filter(s  !s.trim.isEmpty && !s.startsWith("//")). map(line  fahrenheitToCelsius(line.toDouble).toString). intersperse("\n"). through(text.utf8Encode). to(io.file.writeAll(Paths.get("celsius.txt"))). run converter.unsafeRun() • Input file is read & output file is written in a streamy fashion • Constant memory usage • Errors are handled • I/O errors • Exceptions thrown by function passed to map • File handles are acquired when necessary and are guaranteed to be released

Slide 46

Slide 46 text

Resource Allocation 46 def bracket[F[_],R,A](acquire: F[R])( use: R  Stream[F,A], release: R  F[Unit] ): Stream[F,A] • Evaluates acquire to produce a resource • Passes resource to use to generate a stream • Decorates the stream returned from use so that the resource is released when the inner stream terminates • Stream termination occurs when: • Natural end of stream is reached • Puller of stream decides to stop pulling (e.g., take(5)) • Unhandled error occurs

Slide 47

Slide 47 text

Pulls and Handles 47 Stream[F, A] Pull[F, A, R] open close • Closing a pull results in a stream • Managed resources acquired via Pull.acquire • Pull establishes a "scope" in which managed resources are tracked • Pull forms a monad in R • Pull[F, A, R] can: • evaluate effects of type F • output values of type A • pass a resource of type R

Slide 48

Slide 48 text

Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] = ??? Allows us to pull elements from the stream

Slide 49

Slide 49 text

Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] = ??? Allows us to pull elements from the stream src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close head

Slide 50

Slide 50 text

Pulls and Handles 48 def open: Pull[F, Nothing, Handle[F,O]] = ??? Allows us to pull elements from the stream src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close head src.open.flatMap { h  h.receive1 { (_, h)  h.echo } }.close tail

Slide 51

Slide 51 text

Pulls and Handles 49 src.open.flatMap { h  h.receive1 { (o, _)  Pull.output1(o) } }.close src.open.flatMap { h  h.receive1 { (_, h)  h.echo } }.close src.open.flatMap(f).close src.pull(f)

Slide 52

Slide 52 text

Recursive Pulls 50 def take[F[_], A](count: Int): Pipe[F, A, A] = { src  src.pull(???) }

Slide 53

Slide 53 text

Recursive Pulls 51 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { } src  src.pull(loop(count)) }

Slide 54

Slide 54 text

Recursive Pulls 52 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { } } src  src.pull(loop(count)) }

Slide 55

Slide 55 text

Recursive Pulls 53 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { h.receive1 { (a, h)  Pull.output1(a)  loop(remaining - 1)(h) } } } src  src.pull(loop(count)) }

Slide 56

Slide 56 text

Recursive Pulls 54 def take[F[_], A](count: Int): Pipe[F, A, A] = { def loop(remaining: Int): Handle[F, A]  Pull[F, A, Unit] = h  { if (remaining <= 0) { Pull.done } else { h.receive1 { (a, h)  Pull.output1(a)  loop(remaining - 1)(h) } } } src  src.pull(loop(count)) } • Recursive pulls model state machines • State stored as params to recursive call • Multiple states can be represented as multiple co- recursive functions

Slide 57

Slide 57 text

Interruption X val src: Stream[Task, A] = ???

Slide 58

Slide 58 text

Interruption X val src: Stream[Task, A] = ??? import fs2.async.mutable.Signal val mkCancel: Task[Signal[Task, Boolean]] = async.signalOf[Task, Boolean](false)

Slide 59

Slide 59 text

Interruption X val src: Stream[Task, A] = ??? import fs2.async.mutable.Signal val mkCancel: Task[Signal[Task, Boolean]] = async.signalOf[Task, Boolean](false) val interruptibleSource: Stream[Task, A] = Stream.eval(mkCancel).flatMap { cancel  // Somewhere in here, call cancel.set(true) src.interruptWhen(cancel) }

Slide 60

Slide 60 text

Asynchronous Buffering 55 def bufferAsync[F[_], A](bufferSize: Int): Pipe[F, A, A] = source  ???

Slide 61

Slide 61 text

Asynchronous Buffering 56 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) }

Slide 62

Slide 62 text

Asynchronous Buffering 57 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  } }

Slide 63

Slide 63 text

Asynchronous Buffering 58 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain } }

Slide 64

Slide 64 text

Asynchronous Buffering 59 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain val out: Stream[F, A] = queue.dequeue.through(pipe.rethrow).unNoneTerminate. flatMap(Stream.chunk) } }

Slide 65

Slide 65 text

Asynchronous Buffering 60 def bufferAsync[F[_], A]( bufferSize: Int)(implicit F: Async[F] ): Pipe[F, A, A] = source  { val mkQueue: F[Queue[F, Attempt[Option[Chunk[A]]]]] = async.boundedQueue(bufferSize) Stream.eval(mkQueue).flatMap { queue  val fill: Stream[F, Nothing] = source.chunks.noneTerminate.attempt.to(queue.enqueue).drain val out: Stream[F, A] = queue.dequeue.through(pipe.rethrow).unNoneTerminate. flatMap(Stream.chunk) fill merge out } }

Slide 66

Slide 66 text

Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? }

Slide 67

Slide 67 text

Concurrency X object concurrent { def join[F[_]: Async, O]( maxOpen: Int)(outer: Stream[F,Stream[F,O]]): Stream[F,O] = ??? } def evalMapConcurrent[F[_]: Async, A, B]( maxOpen: Int)(f: A  F[B]): Pipe[F, A, B] = src  ???

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Type Classes from fs2.util 61 Functor Applicative Monad Traverse

Slide 70

Slide 70 text

Type Classes from fs2.util 62 Functor Applicative Monad Traverse Catchable • Tracks exceptions thrown during evaluation • Allows extraction of failure

Slide 71

Slide 71 text

Type Classes from fs2.util 63 Functor Applicative Monad Traverse Catchable Suspendable • Supports deferred evaluation • Provides suspend and delay methods

Slide 72

Slide 72 text

Type Classes from fs2.util 64 Functor Applicative Monad Traverse Catchable Suspendable Effect • Provides mechanism which evaluates an F[A] • Callback based unsafeRunAsync

Slide 73

Slide 73 text

Type Classes from fs2.util 65 Functor Applicative Monad Traverse Catchable Suspendable Effect Async • Effect which support asynchronous refs • Async.Ref[F, A] provides a "memory cell" that supports a variant of compare-and-set • All async primitives (e.g., semaphore, signal, queue) built with Refs

Slide 74

Slide 74 text

So what? • Combinators expressed polymorphically in effect type • Easier to reason about what a method does when minimum type class constraint is made • Multitude of Tasks • fs2, scalaz, Monix • Multitude of type constructors • scodec-stream's Cursor effect • Task + custom behaviors (e.g., TraceTask) 66

Slide 75

Slide 75 text

Use Case: Network Analysis Tool 67 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Stream[Task, TimeStamped[Option[A]]] Pipe[Task, TimeStamped[Option[A]], TimeStamped[Option[A]]]

Slide 76

Slide 76 text

Use Case: Network Analysis Tool 68 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file

Slide 77

Slide 77 text

Use Case: Network Analysis Tool 69 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file udp.open[Task](addr).flatMap { socket  socket.reads().map { p  TimeStamped.now((port, p.bytes)) } }

Slide 78

Slide 78 text

Use Case: Network Analysis Tool 70 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file Stream.bracket(Pcap.open(…))( h  Stream.repeatEval(poll(h)), h  h.release )

Slide 79

Slide 79 text

Use Case: Network Analysis Tool 71 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file Stream.bracket(RfDevice.open(…))( h  udp.open[Task](…).flatMap(…), h  h.release )

Slide 80

Slide 80 text

Use Case: Network Analysis Tool 72 Capture Source Recorder Rules Rules Rules UI Elastic Search Monitoring pcap Decoder Supports capturing via: • UDP datagrams sent directly to app • UDP or TCP monitoring via libpcap • Capture from coaxial RF • Playback of pcap file • Stream large files from disk • Filter inputs based on various criteria • Playback: • in realtime • multiplier of realtime • as fast as possible

Slide 81

Slide 81 text

PCAP File Playback 73 val timestamped: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] = timestampedDecoder.decodeMmap(new FileInputStream(path).getChannel) val throttlingFactor: Option[Double] = ??? val throttled: Stream[Task, TimeStamped[Either[UnknownEtherType, B]]] = throttlingFactor.fold(timestamped) { factor  timestamped through throttled(factor) } val source: Stream[Task, TimeStamped[Option[Either[UnknownEtherType, B]]]] = throttled through interpolateTicks()

Slide 82

Slide 82 text

PCAP File Playback 74 def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = ???

Slide 83

Slide 83 text

PCAP File Playback def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = { val tickPeriod = 100.millis val ticks: Stream[F, Unit] = time.awakeEvery[F](tickPeriod).map(_  ()) source  (source through2 ticks)(throttleWithClock(tickPeriod)) }

Slide 84

Slide 84 text

PCAP File Playback 76 def throttle[F[_]: Async, A]( throttlingFactor: Double)(implicit scheduler: Scheduler ): Pipe[F, TimeStamped[A], TimeStamped[A]] = { val tickPeriod = 100.millis val ticks: Stream[F, Unit] = time.awakeEvery[F](tickPeriod).map(_  ()) source  (source through2 ticks)(throttleWithClock(tickPeriod)) } awaitInput awaitTick input / items pending clock tick / nothing pending clock tick / output pending input / nothing pending

Slide 85

Slide 85 text

77 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { }

Slide 86

Slide 86 text

78 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { def awaitInput(upto: Instant): ??? = def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): ??? = }

Slide 87

Slide 87 text

79 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  ??? def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ??? }

Slide 88

Slide 88 text

80 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ??? }

Slide 89

Slide 89 text

81 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ticks.receive1 { (tick, tl)  ??? } }

Slide 90

Slide 90 text

82 def throttleWithClock( tickPeriod: FiniteDuration ): Pipe2[F, TimeStamped[A], Unit, TimeStamped[A]] = { type PullFromSourceOrTicks = (Handle[F, TimeStamped[A]], Handle[F, Unit])  Pull[F, TimeStamped[A], (Handle[F, TimeStamped[A]], Handle[F, Unit])] def awaitInput(upto: Instant): PullFromSourceOrTicks = (src, ticks)  src.receive { (chunk, tl)  ??? } def awaitTick(upto: Instant, p: Chunk[TimeStamped[A]]): PullFromSourceOrTicks = (src, ticks)  ticks.receive1 { (tick, tl)  ??? } _.pull2(_) { (src, ticks)  src.receive1 { (ta, tl)  Pull.output1(ta)  awaitInput(ta.time)(tl, ticks) }} }

Slide 91

Slide 91 text

Compositional Streaming with FS2 • 0.9.2 available now for Scala 2.11 & 2.12 • 1.0.0 in progress — source compatible with 0.9 • Vibrant Ecosystem • Interop: scalaz-stream, akka-stream, cats, scalaz • Infrastructure: doobie, http4s, kafka • Contact me via Gitter.im chat, Github issue tracker, or @mpilquist 83