A sky full of streams

A sky full of streams

Stream processing may sound intimidating, and often unnecessary. There's a reason - many of us rarely (or never) feel like they have an actual need to do it, because the size of their data is never large enough not to fit in memory. But is that the only reason to use stream processing?

As it turns out, most of us stream data on a regular basis, just without being fully aware of it. What is more, having realized that, we can benefit from streaming in all kinds of situations, by decomposing a larger problem into smaller pieces we can reuse and reason about independently.

In this talk, I'll briefly introduce to you fs2 - the functional streaming library for Scala, and its many usecases. You'll see what problems you can solve with it, as well as rough outlines of potential solutions.

We'll also learn a bit about what compositionality means and what makes fs2 a truly compositional library.

Links in the talk:

Compose your program flow with Stream, by Fabio Labella: https://youtu.be/x3GLwl1FxcA
Compositional Programming, by Runar Bjarnason: https://youtu.be/ElLxn_l7P2I
My blog: https://blog.kubukoz.com
Code for this talk: https://github.com/kubukoz/talks/tree/master/sky-full-of-streams
My YT channel: https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q

08f642741fba006656cb86fb61c160b3?s=128

Jakub Kozłowski

September 03, 2019
Tweet

Transcript

  1. A sky full of streams Embracing compositionality of functional streams

    Jakub Kozłowski
  2. Streams around you

  3. Listening to events (UI, Queue, ...) Scanning paginated results (external

    APIs) Traversing a list multiple times class Traversing( findProject: ProjectId => IO[Project], findIssues: Project => IO[List[IssueId]], findIssue: IssueId => IO[Issue], allProjectIds: IO[List[ProjectId]], me: UserId ) { def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue] } = ???
  4. trait Projects { def getPage(afterProject: Option[ProjectId]): IO[List[Project]] } trait Issues

    { def getPage( projectId: ProjectId, afterIssue: Option[IssueId] ): IO[List[Issue]] } class Github(projects: Projects, issues: Issues) { def findFirstWithMatchingIssue( predicate: Issue => Boolean ): IO[Option[Project]] } Listening to events (UI, Queue, ...) Scanning paginated results (external APIs) Traversing a list multiple times = ???
  5. trait Listener { def create[A: Decoder]( handler: A => Unit

    ): Unit } Listening to events (UI, Queue, ...) Scanning paginated results (external APIs) Traversing a list multiple times = ???
  6. Many, many others - TCP connections - HTTP calls -

    Scheduled work - Files, binary data - Websockets
  7. Common core?

  8. Common core? Potentially making control flow decisions

  9. Common core? Potential parallelism at each step Potentially making control

    flow decisions
  10. Common core? Potential effects at each step Potential parallelism at

    each step Potentially making control flow decisions
  11. Common core? Handling multiple values Potential effects at each step

    Potential parallelism at each step Potentially making control flow decisions
  12. Common core? Handling multiple values Potential effects at each step

    Potential parallelism at each step Potentially making control flow decisions Resource safety
  13. Common core? Handling multiple values Potential effects at each step

    Potential parallelism at each step Potentially making control flow decisions Resource safety
  14. Common core? Handling multiple values Potential effects at each step

    Potential parallelism at each step Potentially making control flow decisions Resource safety
  15. fs2.Stream[F[_], O]

  16. fs2.Stream[F[_], O] An immutable, lazy value

  17. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to

    ∞ values || Fails with a Throwable
  18. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to

    ∞ values || Fails with a Throwable Can have effects of F
  19. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to

    ∞ values || Fails with a Throwable Can have effects of F F[A] = usually IO[A]
  20. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to

    ∞ values || Fails with a Throwable Can have effects of F F[A] = usually IO[A] Or, less formally...
  21. fs2.Stream[F[_], O] A description of: A sequence of effects potentially

    producing values
  22. fs2.Stream[F[_], O] A description of: A series of effects producing

    values
  23. fs2.Stream[F[_], O] A description of: A series of effectsproducing values

  24. fs2.Stream[F[_], O] A description of: A series of effects producing

    values
  25. fs2.Stream[F[_], O] A description of: A series of effects producing

    values
  26. Pure streams ✨ fs2.Stream[fs2.Pure, O] Can be evaluated to a

    list without effects
  27. Pure streams ✨ fs2.Stream[fs2.Pure, O] Thanks to a clever hack:

  28. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type

    Pure[A] <: Nothing } Thanks to a clever hack:
  29. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type

    Pure[A] <: Nothing } Thanks to a clever hack: And covariance:
  30. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type

    Pure[A] <: Nothing } Thanks to a clever hack: And covariance: final class Stream[+F[_], +O]
  31. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type

    Pure[A] <: Nothing } Thanks to a clever hack: And covariance: final class Stream[+F[_], +O] val pure: Stream[Pure, Int] = Stream(1, 2, 3) val io : Stream[F, Int] = p
  32. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type

    Pure[A] <: Nothing } Thanks to a clever hack: And covariance: final class Stream[+F[_], +O] val pure: Stream[Pure, Int] = Stream(1, 2, 3) val io : Stream[F, Int] = p For any F[_]
  33. Building fs2 streams

  34. Lift a single value Building fs2 streams

  35. Lift a single value Stream.emit (_: A ) Building fs2

    streams
  36. Lift a single value Lift a sequence Stream.emit (_: A

    ) Building fs2 streams
  37. Lift a single value Lift a sequence Stream.emit (_: A

    ) Stream.emits(_: Seq[A]) Building fs2 streams
  38. Lift a single value Lift a single effect Lift a

    sequence Stream.emit (_: A ) Stream.emits(_: Seq[A]) Building fs2 streams
  39. Stream.eval (_: F[A]) Lift a single value Lift a single

    effect Lift a sequence Stream.emit (_: A ) Stream.emits(_: Seq[A]) Building fs2 streams
  40. Infinite streams ∞ Stream.constant(42) Stream.awakeEvery[IO](1.second) Stream.iterate(1.0)(_ * 2)

  41. Infinite streams ∞ Out of time/memory? Try these:

  42. .drain Infinite streams ∞ Out of time/memory? Try these:

  43. .drain .head Infinite streams ∞ Out of time/memory? Try these:

  44. .drain .head .take(10) Infinite streams ∞ Out of time/memory? Try

    these:
  45. .drain .head .take(10) .interruptAfter(10.minutes) Infinite streams ∞ Out of time/memory?

    Try these:
  46. .drain .head .take(10) .interruptAfter(10.minutes) .interruptWhen(Stream.random[IO].map(_ % 10 == 0)) Infinite

    streams ∞ Out of time/memory? Try these:
  47. .drain .head .take(10) .interruptAfter(10.minutes) .interruptWhen(Stream.random[IO].map(_ % 10 == 0)) Infinite

    streams ∞ Out of time/memory? Try these: When this produces true
  48. Consuming streams val example: Stream[IO, Int] example.compile.<tab> Effectful streams need

    to be "compiled" to the effect
  49. Consuming streams val example: Stream[IO, Int] example.compile.<tab> Effectful streams need

    to be "compiled" to the effect
  50. Transforming streams numbers .map(_ % 10 + 1) .flatMap {

    until => Stream.range(0, until) } .evalMap(showOut) val numbers: Stream[IO, Int] = Stream.random[IO] def showOut(i: Int) = IO(println(i))
  51. Transforming streams numbers .map(_ % 10 + 1) .flatMap {

    until => Stream.range(0, until) } .evalMap(showOut) Transform each element Replace element with sub-stream and flatten Transform each element efectfully def showOut(i: Int) = IO(println(i))
  52. Transforming streams numbers .map(_ % 10 + 1) .flatMap {

    until => Stream.range(0, until) } .evalMap(showOut) def showOut(i: Int) = IO(println(i)) //[-467868903, 452477122, 1143039958, ...] //[-2, 3, 9, ...] //[0, 1, 2, // 0, 1, 2, 3, 4, 5, 6, 7, 8, ...]
  53. What actually happens numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap

    { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut)
  54. What actually happens numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap

    { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut) random: -467868903 map: -2 random: 452477122 map: 3 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 random: 1143039958 map: 9 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 flatMap: 3 3 flatMap: 4 4 flatMap: 5 5 flatMap: 6 6 flatMap: 7 7 flatMap: 8 8
  55. What actually happens numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap

    { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut) random: -467868903 map: -2 random: 452477122 map: 3 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 random: 1143039958 map: 9 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 flatMap: 3 3 flatMap: 4 4 flatMap: 5 5 flatMap: 6 6 flatMap: 7 7 flatMap: 8 8
  56. What actually happens random: -467868903 map: -2 random: 452477122 map:

    3 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 random: 1143039958 map: 9 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 flatMap: 3 3 flatMap: 4 4 flatMap: 5 5 flatMap: 6 6 flatMap: 7 7 flatMap: 8 8 numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut)
  57. Control flow Stream.bracket { IO { new BufferedReader(new FileReader(new File("./build.sbt")))

    } }(f => IO(f.close())) val file: Stream[IO, BufferedReader] =
  58. Stream.bracket { IO { new BufferedReader(new FileReader(new File("./build.sbt"))) } }(f

    => IO(f.close())) val file: Stream[IO, BufferedReader] = ... val process = file.flatMap { reader => Stream .eval(IO(Option(reader.readLine()))) .repeat .unNoneTerminate }.map(_.length) Stream.sleep_[IO](200.millis) ++ Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++ Stream(1, 3, 5) ++ Control flow
  59. Stream(1, 3, 5) ++ file.flatMap { reader => Stream .eval(IO(Option(reader.readLine())))

    .repeat .unNoneTerminate }.map(_.length) Stream.sleep_[IO](200.millis) ++ Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++ 3 known elements 5 ~random elements No elements, 200ms wait Lots of elements (1 per file line) Control flow
  60. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration]

    - emit n values every 1 millisecond - emit n values every 2 milliseconds - emit n values every 4 milliseconds - ... Reset delay every time something is emitted by `resets`
  61. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration]

    = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
  62. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration]

    = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
  63. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration]

    = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently ... resets.evalMap(_ => restart.put(()) *> showResetting) } }
  64. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration]

    = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
  65. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration]

    = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
  66. val slowDown = slowDownEveryNTicks( resets = Stream.awakeEvery[IO](3.seconds).void, n = 5

    ) clientMessages.zipLeft(slowDown) Demo time!
  67. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue]]
  68. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue]] def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
  69. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue]] def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
  70. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue]] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
  71. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue]] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] = projects .filter(!excludedProject(_)) .through(issues) .filter(!excludedIssue(_)) .filter(_.creator === me) .compile .toList def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
  72. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue]] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] = projects .find(inProject) .through(issues) .head.compile.last = projects .filter(!excludedProject(_)) .through(issues) .filter(!excludedIssue(_)) .filter(_.creator === me) .compile .toList def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
  73. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean,

    excludedIssue: Issue => Boolean ): IO[List[Issue] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] = projects .find(inProject) .through(issues) .head.compile.last = projects .filter(!excludedProject(_)) .through(issues) .filter(!excludedIssue(_)) .filter(_.creator === me) .compile .toList def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue] type Pipe[F[_], -I, +O] = Stream[F, I] => Stream[F, O]
  74. Reusable transformations val filterText: Pipe[IO, Message, TextMessage] = _.evalMap {

    case msg: TextMessage => msg.some.pure[IO] case msg => logger.error(s"Not a text message: $msg").as(none) }.unNone def decode[A: Decoder]: Pipe[IO, String, A] = _.map(io.circe.parser.decode[A](_)).evalMap { case Right(v) => v.some.pure[IO] case Left(e) => logger.error(e)("Decoding error").as(none) }.unNone consumerMessages .through(filterText) .map(_.getText) .through(decode[UserEvent])
  75. Compositionality and streams

  76. Compositionality principle The meaning of an expression is the meaning

    of its parts and the way they are combined together.
  77. Compositionality and referential transparency val program: IO[Unit] = createFork.bracket(commit(a))(closeFork(_)) >>

    sendEvent(commitSuccessful(data))
  78. Compositionality and referential transparency val program: IO[Unit] = { val

    prog1 = createFork.bracket(commit(data))(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 >> prog2 } val program: IO[Unit] = createFork.bracket(commit(a))(closeFork(_)) >> sendEvent(commitSuccessful(data))
  79. Referential transparency is not always enough val program: IO[Unit] =

    { val prog1 = createFork.bracket(commit(data))(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 >> prog2 }
  80. Referential transparency is not always enough def program(use: Repository =>

    IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 }
  81. Referential transparency is not always enough def program(use: Repository =>

    IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 }
  82. Referential transparency is not always enough def program(use: Repository =>

    IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 } program(quiteLongStream(_).compile.toList):
  83. Referential transparency is not always enough def program(use: Repository =>

    IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 } program(quiteLongStream(_).compile.toList):
  84. Streams are self-contained val program: Stream[IO, Repository] = Stream.bracket(createFork)(closeFork(_)) ++

    Stream.eval_(sendEvent(commitSuccessful(data))) program.flatMap(quiteLongStream)
  85. def tupleWith42[A](fa: IO[A]): IO[(A, Int)] = fa.map(a => (a, 42))

    How can we test this function?
  86. val mostUsefulIssues = issueSource.filter(_.upvotes > 100) How can we test

    this stream?
  87. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) = issueSource.filter(_.upvotes > 100) How can

    we test this stream?
  88. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) = issueSource.filter(_.upvotes > 100) How can

    we test this stream? mostUsefulIssues( Stream(Issue("#1", "/u/root", 90), Issue("#2", "/u/root", 110)) ).toList === List(Issue("#2", "/u/root", 110))
  89. fs2 is truly compositional

  90. fs2 is truly compositional - No matter where the data

    comes from, it's always the same abstraction
  91. fs2 is truly compositional - No matter where the data

    comes from, it's always the same abstraction - All combinators work on all the streams of compatible types
  92. fs2 is truly compositional - No matter where the data

    comes from, it's always the same abstraction - All combinators work on all the streams of compatible types - Stream scales from pure streams to complex processes with lots of resources and concurrency
  93. fs2 is truly compositional - No matter where the data

    comes from, it's always the same abstraction - All combinators work on all the streams of compatible types - Stream scales from pure streams to complex processes with lots of resources and concurrency - List + IO on superpowers with far less responsibility for you
  94. val converter: Stream[IO, Unit] = Stream.resource(Blocker[IO]).flatMap { blocker => def

    fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blocker, 4096) .through(text.utf8Decode) .through(text.lines) .filter(s => !s.trim.isEmpty && !s.startsWith("//")) .map(line => fahrenheitToCelsius(line.toDouble).toString) .intersperse("\n") .through(text.utf8Encode) .through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blocker)) }
  95. def server( blocker: Blocker ): Stream[IO, Resource[IO, fs2.io.tcp.Socket[IO]]] = Stream.resource(fs2.io.tcp.SocketGroup[IO](blocker)).flatMap

    { group => group.server[IO]( new InetSocketAddress("0.0.0.0", 8080) ) } val clientMessages = Stream .resource(Blocker[IO]) .flatMap(server) .map { Stream .resource(_) .flatMap(_.reads(1024)) .through(fs2.text.utf8Decode) .through(fs2.text.lines) .map("Message: " + _) } .parJoin(maxOpen = 10)
  96. None
  97. "co.fs2" %% "fs2-core" % "2.1.0" https://fs2.io Give it a try!

  98. Acknowledgements Thanks to Fabio Labella for help with this talk!

    Thanks to Michael Pilquist, Paul Chiusano, Pavel Chlupacek, Fabio and all the contributors of fs2 and cats-effect/cats Massively inspired by: - Declarative control flow with fs2 streams by Fabio Labella: https://www.youtube.com/watch?v=x3GLwl1FxcA - Compositional Programming by Runar Bjarnason: https://www.youtube.com/watch?v=ElLxn_l7P2I
  99. Thank you blog.kubukoz.com @kubukoz Slides: https://speakerdeck.com/kubukoz Code: https://git.io/fjpEv

  100. Thank you blog.kubukoz.com @kubukoz Slides: https://speakerdeck.com/kubukoz Code: https://git.io/fjpEv Find me

    on YouTube! (https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q)