A sky full of streams

Stream processing may sound intimidating, and often unnecessary. There's a reason - many of us rarely (or never) feel like they have an actual need to do it, because the size of their data is never large enough not to fit in memory. But is that the only reason to use stream processing?

As it turns out, most of us stream data on a regular basis, just without being fully aware of it. What is more, having realized that, we can benefit from streaming in all kinds of situations, by decomposing a larger problem into smaller pieces we can reuse and reason about independently.

In this talk, I'll briefly introduce to you fs2 - the functional streaming library for Scala, and its many usecases. You'll see what problems you can solve with it, as well as rough outlines of potential solutions.

We'll also learn a bit about what compositionality means and what makes fs2 a truly compositional library.

Links in the talk:

Compose your program flow with Stream, by Fabio Labella: https://youtu.be/x3GLwl1FxcA
Compositional Programming, by Runar Bjarnason: https://youtu.be/ElLxn_l7P2I
My blog: https://blog.kubukoz.com
Code for this talk: https://github.com/kubukoz/talks/tree/master/sky-full-of-streams
My YT channel: https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q

Transcript

1. A sky full of streams
   Embracing compositionality of functional streams
   Jakub Kozłowski
   

   View Slide

2. Streams around you
   

   View Slide

3. Listening to events (UI, Queue, ...)
   Scanning paginated results (external APIs)
   Traversing a list multiple times
   class Traversing(
   findProject: ProjectId => IO[Project],
   findIssues: Project => IO[List[IssueId]],
   findIssue: IssueId => IO[Issue],
   allProjectIds: IO[List[ProjectId]],
   me: UserId
   ) {
   def allMyIssuesInProjectsWithExclusions(
   excludedProject: Project => Boolean,
   excludedIssue: Issue => Boolean
   ): IO[List[Issue]
   } = ???
   

   View Slide

4. trait Projects {
   def getPage(afterProject: Option[ProjectId]): IO[List[Project]]
   }
   trait Issues {
   def getPage(
   projectId: ProjectId,
   afterIssue: Option[IssueId]
   ): IO[List[Issue]]
   }
   class Github(projects: Projects, issues: Issues) {
   def findFirstWithMatchingIssue(
   predicate: Issue => Boolean
   ): IO[Option[Project]]
   }
   Listening to events (UI, Queue, ...)
   Scanning paginated results (external APIs)
   Traversing a list multiple times
   = ???
   

   View Slide

5. trait Listener {
   def create[A: Decoder](
   handler: A => Unit
   ): Unit
   }
   Listening to events (UI, Queue, ...)
   Scanning paginated results (external APIs)
   Traversing a list multiple times
   = ???
   

   View Slide

6. Many, many others
   - TCP connections
   - HTTP calls
   - Scheduled work
   - Files, binary data
   - Websockets
   

   View Slide

7. Common core?
   

   View Slide

8. Common core?
   Potentially making control flow decisions
   

   View Slide

9. Common core?
   Potential parallelism at each step
   Potentially making control flow decisions
   

   View Slide

10. Common core?
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    

    View Slide

11. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    

    View Slide

12. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    Resource safety
    

    View Slide

13. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    Resource safety
    

    View Slide

14. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    Resource safety
    

    View Slide

15. fs2.Stream[F[_], O]
    

    View Slide

16. fs2.Stream[F[_], O]
    An immutable, lazy value
    

    View Slide

17. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    

    View Slide

18. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    Can have effects of F
    

    View Slide

19. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    Can have effects of F
    F[A] = usually IO[A]
    

    View Slide

20. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    Can have effects of F
    F[A] = usually IO[A]
    Or, less formally...
    

    View Slide

21. fs2.Stream[F[_], O]
    A description of:
    A sequence of effects potentially producing values
    

    View Slide

22. fs2.Stream[F[_], O]
    A description of:
    A series of effects producing values
    

    View Slide

23. fs2.Stream[F[_], O]
    A description of:
    A series of effectsproducing values
    

    View Slide

24. fs2.Stream[F[_], O]
    A description of:
    A series of effects producing values
    

    View Slide

25. fs2.Stream[F[_], O]
    A description of:
    A series of effects producing values
    

    View Slide

26. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    Can be evaluated to a list without effects
    

    View Slide

27. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    Thanks to a clever hack:
    

    View Slide

28. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    

    View Slide

29. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance:
    

    View Slide

30. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance: final class Stream[+F[_], +O]
    

    View Slide

31. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance: final class Stream[+F[_], +O]
    val pure: Stream[Pure, Int] = Stream(1, 2, 3)
    val io : Stream[F, Int] = p
    

    View Slide

32. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance: final class Stream[+F[_], +O]
    val pure: Stream[Pure, Int] = Stream(1, 2, 3)
    val io : Stream[F, Int] = p
    For any F[_]
    

    View Slide

33. Building fs2 streams
    

    View Slide

34. Lift a single value
    Building fs2 streams
    

    View Slide

35. Lift a single value Stream.emit (_: A )
    Building fs2 streams
    

    View Slide

36. Lift a single value
    Lift a sequence
    Stream.emit (_: A )
    Building fs2 streams
    

    View Slide

37. Lift a single value
    Lift a sequence
    Stream.emit (_: A )
    Stream.emits(_: Seq[A])
    Building fs2 streams
    

    View Slide

38. Lift a single value
    Lift a single effect
    Lift a sequence
    Stream.emit (_: A )
    Stream.emits(_: Seq[A])
    Building fs2 streams
    

    View Slide

39. Stream.eval (_: F[A])
    Lift a single value
    Lift a single effect
    Lift a sequence
    Stream.emit (_: A )
    Stream.emits(_: Seq[A])
    Building fs2 streams
    

    View Slide

40. Infinite streams ∞
    Stream.constant(42)
    Stream.awakeEvery[IO](1.second)
    Stream.iterate(1.0)(_ * 2)
    

    View Slide

41. Infinite streams ∞
    Out of time/memory? Try these:
    

    View Slide

42. .drain
    Infinite streams ∞
    Out of time/memory? Try these:
    

    View Slide

43. .drain
    .head
    Infinite streams ∞
    Out of time/memory? Try these:
    

    View Slide

44. .drain
    .head
    .take(10)
    Infinite streams ∞
    Out of time/memory? Try these:
    

    View Slide

45. .drain
    .head
    .take(10)
    .interruptAfter(10.minutes)
    Infinite streams ∞
    Out of time/memory? Try these:
    

    View Slide

46. .drain
    .head
    .take(10)
    .interruptAfter(10.minutes)
    .interruptWhen(Stream.random[IO].map(_ % 10 == 0))
    Infinite streams ∞
    Out of time/memory? Try these:
    

    View Slide

47. .drain
    .head
    .take(10)
    .interruptAfter(10.minutes)
    .interruptWhen(Stream.random[IO].map(_ % 10 == 0))
    Infinite streams ∞
    Out of time/memory? Try these:
    When this produces true
    

    View Slide

48. Consuming streams
    val example: Stream[IO, Int]
    example.compile.
    Effectful streams need to be "compiled" to the effect
    

    View Slide

49. Consuming streams
    val example: Stream[IO, Int]
    example.compile.
    Effectful streams need to be "compiled" to the effect
    

    View Slide

50. Transforming streams
    numbers
    .map(_ % 10 + 1)
    .flatMap { until =>
    Stream.range(0, until)
    }
    .evalMap(showOut)
    val numbers: Stream[IO, Int] = Stream.random[IO]
    def showOut(i: Int) = IO(println(i))
    

    View Slide

51. Transforming streams
    numbers
    .map(_ % 10 + 1)
    .flatMap { until =>
    Stream.range(0, until)
    }
    .evalMap(showOut)
    Transform each element
    Replace element with sub-stream and flatten
    Transform each element efectfully
    def showOut(i: Int) = IO(println(i))
    

    View Slide

52. Transforming streams
    numbers
    .map(_ % 10 + 1)
    .flatMap { until =>
    Stream.range(0, until)
    }
    .evalMap(showOut)
    def showOut(i: Int) = IO(println(i))
    //[-467868903, 452477122,
    1143039958, ...]
    //[-2, 3, 9, ...]
    //[0, 1, 2,
    // 0, 1, 2, 3, 4, 5, 6, 7, 8, ...]
    

    View Slide

53. What actually happens
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)
    

    View Slide

54. What actually happens
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)
    random: -467868903
    map: -2
    random: 452477122
    map: 3
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    random: 1143039958
    map: 9
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    flatMap: 3
    3
    flatMap: 4
    4
    flatMap: 5
    5
    flatMap: 6
    6
    flatMap: 7
    7
    flatMap: 8
    8
    

    View Slide

55. What actually happens
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)
    random: -467868903
    map: -2
    random: 452477122
    map: 3
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    random: 1143039958
    map: 9
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    flatMap: 3
    3
    flatMap: 4
    4
    flatMap: 5
    5
    flatMap: 6
    6
    flatMap: 7
    7
    flatMap: 8
    8
    

    View Slide

56. What actually happens
    random: -467868903
    map: -2
    random: 452477122
    map: 3
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    random: 1143039958
    map: 9
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    flatMap: 3
    3
    flatMap: 4
    4
    flatMap: 5
    5
    flatMap: 6
    6
    flatMap: 7
    7
    flatMap: 8
    8
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)
    

    View Slide

57. Control flow
    Stream.bracket {
    IO {
    new BufferedReader(new FileReader(new File("./build.sbt")))
    }
    }(f => IO(f.close()))
    val file: Stream[IO, BufferedReader] =
    

    View Slide

58. Stream.bracket {
    IO {
    new BufferedReader(new FileReader(new File("./build.sbt")))
    }
    }(f => IO(f.close()))
    val file: Stream[IO, BufferedReader] = ...
    val process =
    file.flatMap { reader =>
    Stream
    .eval(IO(Option(reader.readLine())))
    .repeat
    .unNoneTerminate
    }.map(_.length)
    Stream.sleep_[IO](200.millis) ++
    Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++
    Stream(1, 3, 5) ++
    Control flow
    

    View Slide

59. Stream(1, 3, 5) ++
    file.flatMap { reader =>
    Stream
    .eval(IO(Option(reader.readLine())))
    .repeat
    .unNoneTerminate
    }.map(_.length)
    Stream.sleep_[IO](200.millis) ++
    Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++
    3 known elements
    5 ~random elements
    No elements, 200ms wait
    Lots of elements
    (1 per file line)
    Control flow
    

    View Slide

60. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration]
    - emit n values every 1 millisecond
    - emit n values every 2 milliseconds
    - emit n values every 4 milliseconds
    - ...
    Reset delay every time something is emitted by `resets`
    

    View Slide

61. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }
    

    View Slide

62. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }
    

    View Slide

63. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently ...
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }
    

    View Slide

64. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }
    

    View Slide

65. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }
    

    View Slide

66. val slowDown =
    slowDownEveryNTicks(
    resets = Stream.awakeEvery[IO](3.seconds).void,
    n = 5
    )
    clientMessages.zipLeft(slowDown)
    Demo time!
    

    View Slide

67. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    

    View Slide

68. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]
    

    View Slide

69. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]
    

    View Slide

70. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]
    

    View Slide

71. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    =
    projects
    .filter(!excludedProject(_))
    .through(issues)
    .filter(!excludedIssue(_))
    .filter(_.creator === me)
    .compile
    .toList
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]
    

    View Slide

72. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    =
    projects
    .find(inProject)
    .through(issues)
    .head.compile.last
    =
    projects
    .filter(!excludedProject(_))
    .through(issues)
    .filter(!excludedIssue(_))
    .filter(_.creator === me)
    .compile
    .toList
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]
    

    View Slide

73. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    =
    projects
    .find(inProject)
    .through(issues)
    .head.compile.last
    =
    projects
    .filter(!excludedProject(_))
    .through(issues)
    .filter(!excludedIssue(_))
    .filter(_.creator === me)
    .compile
    .toList
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]
    type Pipe[F[_], -I, +O] = Stream[F, I] => Stream[F, O]
    

    View Slide

74. Reusable transformations
    val filterText: Pipe[IO, Message, TextMessage] =
    _.evalMap {
    case msg: TextMessage => msg.some.pure[IO]
    case msg => logger.error(s"Not a text message: $msg").as(none)
    }.unNone
    def decode[A: Decoder]: Pipe[IO, String, A] =
    _.map(io.circe.parser.decode[A](_)).evalMap {
    case Right(v) => v.some.pure[IO]
    case Left(e) => logger.error(e)("Decoding error").as(none)
    }.unNone
    consumerMessages
    .through(filterText)
    .map(_.getText)
    .through(decode[UserEvent])
    

    View Slide

75. Compositionality and streams
    

    View Slide

76. Compositionality principle
    The meaning of an expression is the meaning of its parts
    and the way they are combined together.
    

    View Slide

77. Compositionality and referential transparency
    val program: IO[Unit] =
    createFork.bracket(commit(a))(closeFork(_)) >>
    sendEvent(commitSuccessful(data))
    

    View Slide

78. Compositionality and referential transparency
    val program: IO[Unit] = {
    val prog1 = createFork.bracket(commit(data))(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 >> prog2
    }
    val program: IO[Unit] =
    createFork.bracket(commit(a))(closeFork(_)) >>
    sendEvent(commitSuccessful(data))
    

    View Slide

79. Referential transparency is not always enough
    val program: IO[Unit] = {
    val prog1 = createFork.bracket(commit(data))(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 >> prog2
    }
    

    View Slide

80. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }
    

    View Slide

81. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }
    

    View Slide

82. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }
    program(quiteLongStream(_).compile.toList):
    

    View Slide

83. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }
    program(quiteLongStream(_).compile.toList):
    

    View Slide

84. Streams are self-contained
    val program: Stream[IO, Repository] =
    Stream.bracket(createFork)(closeFork(_)) ++
    Stream.eval_(sendEvent(commitSuccessful(data)))
    program.flatMap(quiteLongStream)
    

    View Slide

85. def tupleWith42[A](fa: IO[A]): IO[(A, Int)] =
    fa.map(a => (a, 42))
    How can we test this function?
    

    View Slide

86. val mostUsefulIssues =
    issueSource.filter(_.upvotes > 100)
    How can we test this stream?
    

    View Slide

87. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) =
    issueSource.filter(_.upvotes > 100)
    How can we test this stream?
    

    View Slide

88. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) =
    issueSource.filter(_.upvotes > 100)
    How can we test this stream?
    mostUsefulIssues(
    Stream(Issue("#1", "/u/root", 90), Issue("#2", "/u/root", 110))
    ).toList === List(Issue("#2", "/u/root", 110))
    

    View Slide

89. fs2 is truly compositional
    

    View Slide

90. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    

    View Slide

91. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    - All combinators work on all the streams of compatible types
    

    View Slide

92. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    - All combinators work on all the streams of compatible types
    - Stream scales from pure streams to complex processes with lots of resources and concurrency
    

    View Slide

93. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    - All combinators work on all the streams of compatible types
    - Stream scales from pure streams to complex processes with lots of resources and concurrency
    - List + IO on superpowers with far less responsibility for you
    

    View Slide

94. val converter: Stream[IO, Unit] = Stream.resource(Blocker[IO]).flatMap { blocker =>
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blocker, 4096)
    .through(text.utf8Decode)
    .through(text.lines)
    .filter(s => !s.trim.isEmpty && !s.startsWith("//"))
    .map(line => fahrenheitToCelsius(line.toDouble).toString)
    .intersperse("\n")
    .through(text.utf8Encode)
    .through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blocker))
    }
    

    View Slide

95. def server(
    blocker: Blocker
    ): Stream[IO, Resource[IO, fs2.io.tcp.Socket[IO]]] =
    Stream.resource(fs2.io.tcp.SocketGroup[IO](blocker)).flatMap {
    group =>
    group.server[IO](
    new InetSocketAddress("0.0.0.0", 8080)
    )
    }
    val clientMessages = Stream
    .resource(Blocker[IO])
    .flatMap(server)
    .map {
    Stream
    .resource(_)
    .flatMap(_.reads(1024))
    .through(fs2.text.utf8Decode)
    .through(fs2.text.lines)
    .map("Message: " + _)
    }
    .parJoin(maxOpen = 10)
    

    View Slide

96. View Slide

97. "co.fs2" %% "fs2-core" % "2.1.0"
    https://fs2.io
    Give it a try!
    

    View Slide

98. Acknowledgements
    Thanks to Fabio Labella for help with this talk!
    Thanks to Michael Pilquist, Paul Chiusano, Pavel Chlupacek, Fabio
    and all the contributors of fs2 and cats-effect/cats
    Massively inspired by:
    - Declarative control flow with fs2 streams by Fabio Labella:
    https://www.youtube.com/watch?v=x3GLwl1FxcA
    - Compositional Programming by Runar Bjarnason:
    https://www.youtube.com/watch?v=ElLxn_l7P2I
    

    View Slide

99. Thank you
    blog.kubukoz.com
    @kubukoz
    Slides: https://speakerdeck.com/kubukoz
    Code: https://git.io/fjpEv
    

    View Slide

100. Thank you
     blog.kubukoz.com
     @kubukoz
     Slides: https://speakerdeck.com/kubukoz
     Code: https://git.io/fjpEv
     Find me on YouTube! (https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q)
     

     View Slide