A sky full of streams

Stream processing may sound intimidating, and often unnecessary. There's a reason - many of us rarely (or never) feel like they have an actual need to do it, because the size of their data is never large enough not to fit in memory. But is that the only reason to use stream processing?

As it turns out, most of us stream data on a regular basis, just without being fully aware of it. What is more, having realized that, we can benefit from streaming in all kinds of situations, by decomposing a larger problem into smaller pieces we can reuse and reason about independently.

In this talk, I'll briefly introduce to you fs2 - the functional streaming library for Scala, and its many usecases. You'll see what problems you can solve with it, as well as rough outlines of potential solutions.

We'll also learn a bit about what compositionality means and what makes fs2 a truly compositional library.

Links in the talk:

Compose your program flow with Stream, by Fabio Labella: https://youtu.be/x3GLwl1FxcA
Compositional Programming, by Runar Bjarnason: https://youtu.be/ElLxn_l7P2I
My blog: https://blog.kubukoz.com
Code for this talk: https://github.com/kubukoz/talks/tree/master/sky-full-of-streams
My YT channel: https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q

Transcript

1. A sky full of streams Embracing compositionality of functional streams Jakub Kozłowski
2. Streams around you
3. Listening to events (UI, Queue, ...) Scanning paginated results (external APIs) Traversing a list multiple times class Traversing( findProject: ProjectId => IO[Project], findIssues: Project => IO[List[IssueId]], findIssue: IssueId => IO[Issue], allProjectIds: IO[List[ProjectId]], me: UserId ) { def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue] } = ???
4. trait Projects { def getPage(afterProject: Option[ProjectId]): IO[List[Project]] } trait Issues { def getPage( projectId: ProjectId, afterIssue: Option[IssueId] ): IO[List[Issue]] } class Github(projects: Projects, issues: Issues) { def findFirstWithMatchingIssue( predicate: Issue => Boolean ): IO[Option[Project]] } Listening to events (UI, Queue, ...) Scanning paginated results (external APIs) Traversing a list multiple times = ???
5. trait Listener { def create[A: Decoder]( handler: A => Unit ): Unit } Listening to events (UI, Queue, ...) Scanning paginated results (external APIs) Traversing a list multiple times = ???
6. Many, many others - TCP connections - HTTP calls - Scheduled work - Files, binary data - Websockets
7. Common core?
8. Common core? Potentially making control flow decisions
9. Common core? Potential parallelism at each step Potentially making control flow decisions
10. Common core? Potential effects at each step Potential parallelism at each step Potentially making control flow decisions
11. Common core? Handling multiple values Potential effects at each step Potential parallelism at each step Potentially making control flow decisions
12. Common core? Handling multiple values Potential effects at each step Potential parallelism at each step Potentially making control flow decisions Resource safety
13. Common core? Handling multiple values Potential effects at each step Potential parallelism at each step Potentially making control flow decisions Resource safety
14. Common core? Handling multiple values Potential effects at each step Potential parallelism at each step Potentially making control flow decisions Resource safety
15. fs2.Stream[F[_], O]
16. fs2.Stream[F[_], O] An immutable, lazy value
17. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to ∞ values || Fails with a Throwable
18. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to ∞ values || Fails with a Throwable Can have effects of F
19. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to ∞ values || Fails with a Throwable Can have effects of F F[A] = usually IO[A]
20. fs2.Stream[F[_], O] An immutable, lazy value Emits from 0 to ∞ values || Fails with a Throwable Can have effects of F F[A] = usually IO[A] Or, less formally...
21. fs2.Stream[F[_], O] A description of: A sequence of effects potentially producing values
22. fs2.Stream[F[_], O] A description of: A series of effects producing values
23. fs2.Stream[F[_], O] A description of: A series of effectsproducing values
24. fs2.Stream[F[_], O] A description of: A series of effects producing values
25. fs2.Stream[F[_], O] A description of: A series of effects producing values
26. Pure streams ✨ fs2.Stream[fs2.Pure, O] Can be evaluated to a list without effects
27. Pure streams ✨ fs2.Stream[fs2.Pure, O] Thanks to a clever hack:
28. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type Pure[A] <: Nothing } Thanks to a clever hack:
29. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type Pure[A] <: Nothing } Thanks to a clever hack: And covariance:
30. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type Pure[A] <: Nothing } Thanks to a clever hack: And covariance: final class Stream[+F[_], +O]
31. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type Pure[A] <: Nothing } Thanks to a clever hack: And covariance: final class Stream[+F[_], +O] val pure: Stream[Pure, Int] = Stream(1, 2, 3) val io : Stream[F, Int] = p
32. Pure streams ✨ fs2.Stream[fs2.Pure, O] package object fs2 { type Pure[A] <: Nothing } Thanks to a clever hack: And covariance: final class Stream[+F[_], +O] val pure: Stream[Pure, Int] = Stream(1, 2, 3) val io : Stream[F, Int] = p For any F[_]
33. Building fs2 streams
34. Lift a single value Building fs2 streams
35. Lift a single value Stream.emit (_: A ) Building fs2 streams
36. Lift a single value Lift a sequence Stream.emit (_: A ) Building fs2 streams
37. Lift a single value Lift a sequence Stream.emit (_: A ) Stream.emits(_: Seq[A]) Building fs2 streams
38. Lift a single value Lift a single effect Lift a sequence Stream.emit (_: A ) Stream.emits(_: Seq[A]) Building fs2 streams
39. Stream.eval (_: F[A]) Lift a single value Lift a single effect Lift a sequence Stream.emit (_: A ) Stream.emits(_: Seq[A]) Building fs2 streams
40. Infinite streams ∞ Stream.constant(42) Stream.awakeEvery[IO](1.second) Stream.iterate(1.0)(_ * 2)
41. Infinite streams ∞ Out of time/memory? Try these:
42. .drain Infinite streams ∞ Out of time/memory? Try these:
43. .drain .head Infinite streams ∞ Out of time/memory? Try these:
44. .drain .head .take(10) Infinite streams ∞ Out of time/memory? Try these:
45. .drain .head .take(10) .interruptAfter(10.minutes) Infinite streams ∞ Out of time/memory? Try these:
46. .drain .head .take(10) .interruptAfter(10.minutes) .interruptWhen(Stream.random[IO].map(_ % 10 == 0)) Infinite streams ∞ Out of time/memory? Try these:
47. .drain .head .take(10) .interruptAfter(10.minutes) .interruptWhen(Stream.random[IO].map(_ % 10 == 0)) Infinite streams ∞ Out of time/memory? Try these: When this produces true
48. Consuming streams val example: Stream[IO, Int] example.compile.<tab> Effectful streams need to be "compiled" to the effect
49. Consuming streams val example: Stream[IO, Int] example.compile.<tab> Effectful streams need to be "compiled" to the effect
50. Transforming streams numbers .map(_ % 10 + 1) .flatMap { until => Stream.range(0, until) } .evalMap(showOut) val numbers: Stream[IO, Int] = Stream.random[IO] def showOut(i: Int) = IO(println(i))
51. Transforming streams numbers .map(_ % 10 + 1) .flatMap { until => Stream.range(0, until) } .evalMap(showOut) Transform each element Replace element with sub-stream and flatten Transform each element efectfully def showOut(i: Int) = IO(println(i))
52. Transforming streams numbers .map(_ % 10 + 1) .flatMap { until => Stream.range(0, until) } .evalMap(showOut) def showOut(i: Int) = IO(println(i)) //[-467868903, 452477122, 1143039958, ...] //[-2, 3, 9, ...] //[0, 1, 2, // 0, 1, 2, 3, 4, 5, 6, 7, 8, ...]
53. What actually happens numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut)
54. What actually happens numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut) random: -467868903 map: -2 random: 452477122 map: 3 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 random: 1143039958 map: 9 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 flatMap: 3 3 flatMap: 4 4 flatMap: 5 5 flatMap: 6 6 flatMap: 7 7 flatMap: 8 8
55. What actually happens numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut) random: -467868903 map: -2 random: 452477122 map: 3 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 random: 1143039958 map: 9 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 flatMap: 3 3 flatMap: 4 4 flatMap: 5 5 flatMap: 6 6 flatMap: 7 7 flatMap: 8 8
56. What actually happens random: -467868903 map: -2 random: 452477122 map: 3 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 random: 1143039958 map: 9 flatMap: 0 0 flatMap: 1 1 flatMap: 2 2 flatMap: 3 3 flatMap: 4 4 flatMap: 5 5 flatMap: 6 6 flatMap: 7 7 flatMap: 8 8 numbers.debug("random") .map(_ % 10 + 1).debug("map") .flatMap { until => Stream.range(0, until) }.debug("flatMap") .evalMap(showOut)
57. Control flow Stream.bracket { IO { new BufferedReader(new FileReader(new File("./build.sbt"))) } }(f => IO(f.close())) val file: Stream[IO, BufferedReader] =
58. Stream.bracket { IO { new BufferedReader(new FileReader(new File("./build.sbt"))) } }(f => IO(f.close())) val file: Stream[IO, BufferedReader] = ... val process = file.flatMap { reader => Stream .eval(IO(Option(reader.readLine()))) .repeat .unNoneTerminate }.map(_.length) Stream.sleep_[IO](200.millis) ++ Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++ Stream(1, 3, 5) ++ Control flow
59. Stream(1, 3, 5) ++ file.flatMap { reader => Stream .eval(IO(Option(reader.readLine()))) .repeat .unNoneTerminate }.map(_.length) Stream.sleep_[IO](200.millis) ++ Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++ 3 known elements 5 ~random elements No elements, 200ms wait Lots of elements (1 per file line) Control flow
60. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration] - emit n values every 1 millisecond - emit n values every 2 milliseconds - emit n values every 4 milliseconds - ... Reset delay every time something is emitted by `resets`
61. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration] = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
62. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration] = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
63. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration] = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently ... resets.evalMap(_ => restart.put(()) *> showResetting) } }
64. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration] = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
65. def slowDownEveryNTicks( resets: Stream[IO, Unit], n: Int ): Stream[IO, FiniteDuration] = { val showSlowingDown = Stream.eval_(IO(println("---------- Slowing down! ----------"))) val showResetting = IO(println("---------- Resetting delays! ----------")) val delaysExponential: Stream[IO, FiniteDuration] = Stream .iterate(1.millisecond)(_ * 2) .flatMap { Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown } Stream.eval(MVar.empty[IO, Unit]).flatMap { restart => val delaysUntilReset = delaysExponential.interruptWhen(restart.take.attempt) delaysUntilReset.repeat concurrently resets.evalMap(_ => restart.put(()) *> showResetting) } }
66. val slowDown = slowDownEveryNTicks( resets = Stream.awakeEvery[IO](3.seconds).void, n = 5 ) clientMessages.zipLeft(slowDown) Demo time!
67. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue]]
68. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue]] def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
69. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue]] def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
70. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue]] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
71. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue]] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] = projects .filter(!excludedProject(_)) .through(issues) .filter(!excludedIssue(_)) .filter(_.creator === me) .compile .toList def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
72. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue]] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] = projects .find(inProject) .through(issues) .head.compile.last = projects .filter(!excludedProject(_)) .through(issues) .filter(!excludedIssue(_)) .filter(_.creator === me) .compile .toList def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue]]
73. Inversion of flow control def allMyIssuesInProjectsWithExclusions( excludedProject: Project => Boolean, excludedIssue: Issue => Boolean ): IO[List[Issue] val projects: Stream[IO, Project] val issues: Pipe[IO, Project, Issue] = projects .find(inProject) .through(issues) .head.compile.last = projects .filter(!excludedProject(_)) .through(issues) .filter(!excludedIssue(_)) .filter(_.creator === me) .compile .toList def firstIssue( inProject: Project => Boolean, ): IO[Option[Issue] type Pipe[F[_], -I, +O] = Stream[F, I] => Stream[F, O]
74. Reusable transformations val filterText: Pipe[IO, Message, TextMessage] = _.evalMap { case msg: TextMessage => msg.some.pure[IO] case msg => logger.error(s"Not a text message: $msg").as(none) }.unNone def decode[A: Decoder]: Pipe[IO, String, A] = _.map(io.circe.parser.decode[A](_)).evalMap { case Right(v) => v.some.pure[IO] case Left(e) => logger.error(e)("Decoding error").as(none) }.unNone consumerMessages .through(filterText) .map(_.getText) .through(decode[UserEvent])
75. Compositionality and streams
76. Compositionality principle The meaning of an expression is the meaning of its parts and the way they are combined together.
77. Compositionality and referential transparency val program: IO[Unit] = createFork.bracket(commit(a))(closeFork(_)) >> sendEvent(commitSuccessful(data))
78. Compositionality and referential transparency val program: IO[Unit] = { val prog1 = createFork.bracket(commit(data))(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 >> prog2 } val program: IO[Unit] = createFork.bracket(commit(a))(closeFork(_)) >> sendEvent(commitSuccessful(data))
79. Referential transparency is not always enough val program: IO[Unit] = { val prog1 = createFork.bracket(commit(data))(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 >> prog2 }
80. Referential transparency is not always enough def program(use: Repository => IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 }
81. Referential transparency is not always enough def program(use: Repository => IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 }
82. Referential transparency is not always enough def program(use: Repository => IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 } program(quiteLongStream(_).compile.toList):
83. Referential transparency is not always enough def program(use: Repository => IO[A]): IO[A] = { val prog1 = createFork.bracket(use)(closeFork(_)) val prog2 = sendEvent(commitSuccessful(data)) prog1 <* prog2 } program(quiteLongStream(_).compile.toList):
84. Streams are self-contained val program: Stream[IO, Repository] = Stream.bracket(createFork)(closeFork(_)) ++ Stream.eval_(sendEvent(commitSuccessful(data))) program.flatMap(quiteLongStream)
85. def tupleWith42[A](fa: IO[A]): IO[(A, Int)] = fa.map(a => (a, 42)) How can we test this function?
86. val mostUsefulIssues = issueSource.filter(_.upvotes > 100) How can we test this stream?
87. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) = issueSource.filter(_.upvotes > 100) How can we test this stream?
88. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) = issueSource.filter(_.upvotes > 100) How can we test this stream? mostUsefulIssues( Stream(Issue("#1", "/u/root", 90), Issue("#2", "/u/root", 110)) ).toList === List(Issue("#2", "/u/root", 110))
89. fs2 is truly compositional
90. fs2 is truly compositional - No matter where the data comes from, it's always the same abstraction
91. fs2 is truly compositional - No matter where the data comes from, it's always the same abstraction - All combinators work on all the streams of compatible types
92. fs2 is truly compositional - No matter where the data comes from, it's always the same abstraction - All combinators work on all the streams of compatible types - Stream scales from pure streams to complex processes with lots of resources and concurrency
93. fs2 is truly compositional - No matter where the data comes from, it's always the same abstraction - All combinators work on all the streams of compatible types - Stream scales from pure streams to complex processes with lots of resources and concurrency - List + IO on superpowers with far less responsibility for you
94. val converter: Stream[IO, Unit] = Stream.resource(Blocker[IO]).flatMap { blocker => def fahrenheitToCelsius(f: Double): Double = (f - 32.0) * (5.0/9.0) io.file.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blocker, 4096) .through(text.utf8Decode) .through(text.lines) .filter(s => !s.trim.isEmpty && !s.startsWith("//")) .map(line => fahrenheitToCelsius(line.toDouble).toString) .intersperse("\n") .through(text.utf8Encode) .through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blocker)) }
95. def server( blocker: Blocker ): Stream[IO, Resource[IO, fs2.io.tcp.Socket[IO]]] = Stream.resource(fs2.io.tcp.SocketGroup[IO](blocker)).flatMap { group => group.server[IO]( new InetSocketAddress("0.0.0.0", 8080) ) } val clientMessages = Stream .resource(Blocker[IO]) .flatMap(server) .map { Stream .resource(_) .flatMap(_.reads(1024)) .through(fs2.text.utf8Decode) .through(fs2.text.lines) .map("Message: " + _) } .parJoin(maxOpen = 10)
97. "co.fs2" %% "fs2-core" % "2.1.0" https://fs2.io Give it a try!
98. Acknowledgements Thanks to Fabio Labella for help with this talk! Thanks to Michael Pilquist, Paul Chiusano, Pavel Chlupacek, Fabio and all the contributors of fs2 and cats-effect/cats Massively inspired by: - Declarative control flow with fs2 streams by Fabio Labella: https://www.youtube.com/watch?v=x3GLwl1FxcA - Compositional Programming by Runar Bjarnason: https://www.youtube.com/watch?v=ElLxn_l7P2I
99. Thank you blog.kubukoz.com @kubukoz Slides: https://speakerdeck.com/kubukoz Code: https://git.io/fjpEv
100. Thank you blog.kubukoz.com @kubukoz Slides: https://speakerdeck.com/kubukoz Code: https://git.io/fjpEv Find me on YouTube! (https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q)