$30 off During Our Annual Pro Sale. View Details »

A sky full of streams

Jakub Kozłowski
September 03, 2019

A sky full of streams

Stream processing may sound intimidating, and often unnecessary. There's a reason - many of us rarely (or never) feel like they have an actual need to do it, because the size of their data is never large enough not to fit in memory. But is that the only reason to use stream processing?

As it turns out, most of us stream data on a regular basis, just without being fully aware of it. What is more, having realized that, we can benefit from streaming in all kinds of situations, by decomposing a larger problem into smaller pieces we can reuse and reason about independently.

In this talk, I'll briefly introduce to you fs2 - the functional streaming library for Scala, and its many usecases. You'll see what problems you can solve with it, as well as rough outlines of potential solutions.

We'll also learn a bit about what compositionality means and what makes fs2 a truly compositional library.

Links in the talk:

Compose your program flow with Stream, by Fabio Labella: https://youtu.be/x3GLwl1FxcA
Compositional Programming, by Runar Bjarnason: https://youtu.be/ElLxn_l7P2I
My blog: https://blog.kubukoz.com
Code for this talk: https://github.com/kubukoz/talks/tree/master/sky-full-of-streams
My YT channel: https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q

Jakub Kozłowski

September 03, 2019
Tweet

More Decks by Jakub Kozłowski

Other Decks in Technology

Transcript

  1. A sky full of streams
    Embracing compositionality of functional streams
    Jakub Kozłowski

    View Slide

  2. Streams around you

    View Slide

  3. Listening to events (UI, Queue, ...)
    Scanning paginated results (external APIs)
    Traversing a list multiple times
    class Traversing(
    findProject: ProjectId => IO[Project],
    findIssues: Project => IO[List[IssueId]],
    findIssue: IssueId => IO[Issue],
    allProjectIds: IO[List[ProjectId]],
    me: UserId
    ) {
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]
    } = ???

    View Slide

  4. trait Projects {
    def getPage(afterProject: Option[ProjectId]): IO[List[Project]]
    }
    trait Issues {
    def getPage(
    projectId: ProjectId,
    afterIssue: Option[IssueId]
    ): IO[List[Issue]]
    }
    class Github(projects: Projects, issues: Issues) {
    def findFirstWithMatchingIssue(
    predicate: Issue => Boolean
    ): IO[Option[Project]]
    }
    Listening to events (UI, Queue, ...)
    Scanning paginated results (external APIs)
    Traversing a list multiple times
    = ???

    View Slide

  5. trait Listener {
    def create[A: Decoder](
    handler: A => Unit
    ): Unit
    }
    Listening to events (UI, Queue, ...)
    Scanning paginated results (external APIs)
    Traversing a list multiple times
    = ???

    View Slide

  6. Many, many others
    - TCP connections
    - HTTP calls
    - Scheduled work
    - Files, binary data
    - Websockets

    View Slide

  7. Common core?

    View Slide

  8. Common core?
    Potentially making control flow decisions

    View Slide

  9. Common core?
    Potential parallelism at each step
    Potentially making control flow decisions

    View Slide

  10. Common core?
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions

    View Slide

  11. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions

    View Slide

  12. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    Resource safety

    View Slide

  13. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    Resource safety

    View Slide

  14. Common core?
    Handling multiple values
    Potential effects at each step
    Potential parallelism at each step
    Potentially making control flow decisions
    Resource safety

    View Slide

  15. fs2.Stream[F[_], O]

    View Slide

  16. fs2.Stream[F[_], O]
    An immutable, lazy value

    View Slide

  17. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable

    View Slide

  18. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    Can have effects of F

    View Slide

  19. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    Can have effects of F
    F[A] = usually IO[A]

    View Slide

  20. fs2.Stream[F[_], O]
    An immutable, lazy value
    Emits from 0 to ∞ values || Fails with a Throwable
    Can have effects of F
    F[A] = usually IO[A]
    Or, less formally...

    View Slide

  21. fs2.Stream[F[_], O]
    A description of:
    A sequence of effects potentially producing values

    View Slide

  22. fs2.Stream[F[_], O]
    A description of:
    A series of effects producing values

    View Slide

  23. fs2.Stream[F[_], O]
    A description of:
    A series of effectsproducing values

    View Slide

  24. fs2.Stream[F[_], O]
    A description of:
    A series of effects producing values

    View Slide

  25. fs2.Stream[F[_], O]
    A description of:
    A series of effects producing values

    View Slide

  26. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    Can be evaluated to a list without effects

    View Slide

  27. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    Thanks to a clever hack:

    View Slide

  28. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:

    View Slide

  29. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance:

    View Slide

  30. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance: final class Stream[+F[_], +O]

    View Slide

  31. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance: final class Stream[+F[_], +O]
    val pure: Stream[Pure, Int] = Stream(1, 2, 3)
    val io : Stream[F, Int] = p

    View Slide

  32. Pure streams ✨
    fs2.Stream[fs2.Pure, O]
    package object fs2 {
    type Pure[A] <: Nothing
    }
    Thanks to a clever hack:
    And covariance: final class Stream[+F[_], +O]
    val pure: Stream[Pure, Int] = Stream(1, 2, 3)
    val io : Stream[F, Int] = p
    For any F[_]

    View Slide

  33. Building fs2 streams

    View Slide

  34. Lift a single value
    Building fs2 streams

    View Slide

  35. Lift a single value Stream.emit (_: A )
    Building fs2 streams

    View Slide

  36. Lift a single value
    Lift a sequence
    Stream.emit (_: A )
    Building fs2 streams

    View Slide

  37. Lift a single value
    Lift a sequence
    Stream.emit (_: A )
    Stream.emits(_: Seq[A])
    Building fs2 streams

    View Slide

  38. Lift a single value
    Lift a single effect
    Lift a sequence
    Stream.emit (_: A )
    Stream.emits(_: Seq[A])
    Building fs2 streams

    View Slide

  39. Stream.eval (_: F[A])
    Lift a single value
    Lift a single effect
    Lift a sequence
    Stream.emit (_: A )
    Stream.emits(_: Seq[A])
    Building fs2 streams

    View Slide

  40. Infinite streams ∞
    Stream.constant(42)
    Stream.awakeEvery[IO](1.second)
    Stream.iterate(1.0)(_ * 2)

    View Slide

  41. Infinite streams ∞
    Out of time/memory? Try these:

    View Slide

  42. .drain
    Infinite streams ∞
    Out of time/memory? Try these:

    View Slide

  43. .drain
    .head
    Infinite streams ∞
    Out of time/memory? Try these:

    View Slide

  44. .drain
    .head
    .take(10)
    Infinite streams ∞
    Out of time/memory? Try these:

    View Slide

  45. .drain
    .head
    .take(10)
    .interruptAfter(10.minutes)
    Infinite streams ∞
    Out of time/memory? Try these:

    View Slide

  46. .drain
    .head
    .take(10)
    .interruptAfter(10.minutes)
    .interruptWhen(Stream.random[IO].map(_ % 10 == 0))
    Infinite streams ∞
    Out of time/memory? Try these:

    View Slide

  47. .drain
    .head
    .take(10)
    .interruptAfter(10.minutes)
    .interruptWhen(Stream.random[IO].map(_ % 10 == 0))
    Infinite streams ∞
    Out of time/memory? Try these:
    When this produces true

    View Slide

  48. Consuming streams
    val example: Stream[IO, Int]
    example.compile.
    Effectful streams need to be "compiled" to the effect

    View Slide

  49. Consuming streams
    val example: Stream[IO, Int]
    example.compile.
    Effectful streams need to be "compiled" to the effect

    View Slide

  50. Transforming streams
    numbers
    .map(_ % 10 + 1)
    .flatMap { until =>
    Stream.range(0, until)
    }
    .evalMap(showOut)
    val numbers: Stream[IO, Int] = Stream.random[IO]
    def showOut(i: Int) = IO(println(i))

    View Slide

  51. Transforming streams
    numbers
    .map(_ % 10 + 1)
    .flatMap { until =>
    Stream.range(0, until)
    }
    .evalMap(showOut)
    Transform each element
    Replace element with sub-stream and flatten
    Transform each element efectfully
    def showOut(i: Int) = IO(println(i))

    View Slide

  52. Transforming streams
    numbers
    .map(_ % 10 + 1)
    .flatMap { until =>
    Stream.range(0, until)
    }
    .evalMap(showOut)
    def showOut(i: Int) = IO(println(i))
    //[-467868903, 452477122,
    1143039958, ...]
    //[-2, 3, 9, ...]
    //[0, 1, 2,
    // 0, 1, 2, 3, 4, 5, 6, 7, 8, ...]

    View Slide

  53. What actually happens
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)

    View Slide

  54. What actually happens
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)
    random: -467868903
    map: -2
    random: 452477122
    map: 3
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    random: 1143039958
    map: 9
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    flatMap: 3
    3
    flatMap: 4
    4
    flatMap: 5
    5
    flatMap: 6
    6
    flatMap: 7
    7
    flatMap: 8
    8

    View Slide

  55. What actually happens
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)
    random: -467868903
    map: -2
    random: 452477122
    map: 3
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    random: 1143039958
    map: 9
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    flatMap: 3
    3
    flatMap: 4
    4
    flatMap: 5
    5
    flatMap: 6
    6
    flatMap: 7
    7
    flatMap: 8
    8

    View Slide

  56. What actually happens
    random: -467868903
    map: -2
    random: 452477122
    map: 3
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    random: 1143039958
    map: 9
    flatMap: 0
    0
    flatMap: 1
    1
    flatMap: 2
    2
    flatMap: 3
    3
    flatMap: 4
    4
    flatMap: 5
    5
    flatMap: 6
    6
    flatMap: 7
    7
    flatMap: 8
    8
    numbers.debug("random")
    .map(_ % 10 + 1).debug("map")
    .flatMap { until =>
    Stream.range(0, until)
    }.debug("flatMap")
    .evalMap(showOut)

    View Slide

  57. Control flow
    Stream.bracket {
    IO {
    new BufferedReader(new FileReader(new File("./build.sbt")))
    }
    }(f => IO(f.close()))
    val file: Stream[IO, BufferedReader] =

    View Slide

  58. Stream.bracket {
    IO {
    new BufferedReader(new FileReader(new File("./build.sbt")))
    }
    }(f => IO(f.close()))
    val file: Stream[IO, BufferedReader] = ...
    val process =
    file.flatMap { reader =>
    Stream
    .eval(IO(Option(reader.readLine())))
    .repeat
    .unNoneTerminate
    }.map(_.length)
    Stream.sleep_[IO](200.millis) ++
    Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++
    Stream(1, 3, 5) ++
    Control flow

    View Slide

  59. Stream(1, 3, 5) ++
    file.flatMap { reader =>
    Stream
    .eval(IO(Option(reader.readLine())))
    .repeat
    .unNoneTerminate
    }.map(_.length)
    Stream.sleep_[IO](200.millis) ++
    Stream.random[IO].flatMap(Stream.range(0, _)).take(5) ++
    3 known elements
    5 ~random elements
    No elements, 200ms wait
    Lots of elements
    (1 per file line)
    Control flow

    View Slide

  60. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration]
    - emit n values every 1 millisecond
    - emit n values every 2 milliseconds
    - emit n values every 4 milliseconds
    - ...
    Reset delay every time something is emitted by `resets`

    View Slide

  61. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }

    View Slide

  62. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }

    View Slide

  63. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently ...
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }

    View Slide

  64. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }

    View Slide

  65. def slowDownEveryNTicks(
    resets: Stream[IO, Unit],
    n: Int
    ): Stream[IO, FiniteDuration] = {
    val showSlowingDown =
    Stream.eval_(IO(println("---------- Slowing down! ----------")))
    val showResetting =
    IO(println("---------- Resetting delays! ----------"))
    val delaysExponential: Stream[IO, FiniteDuration] =
    Stream
    .iterate(1.millisecond)(_ * 2)
    .flatMap {
    Stream.awakeDelay[IO](_).take(n.toLong) ++ showSlowingDown
    }
    Stream.eval(MVar.empty[IO, Unit]).flatMap { restart =>
    val delaysUntilReset =
    delaysExponential.interruptWhen(restart.take.attempt)
    delaysUntilReset.repeat concurrently
    resets.evalMap(_ => restart.put(()) *> showResetting)
    }
    }

    View Slide

  66. val slowDown =
    slowDownEveryNTicks(
    resets = Stream.awakeEvery[IO](3.seconds).void,
    n = 5
    )
    clientMessages.zipLeft(slowDown)
    Demo time!

    View Slide

  67. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]

    View Slide

  68. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]

    View Slide

  69. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]

    View Slide

  70. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]

    View Slide

  71. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    =
    projects
    .filter(!excludedProject(_))
    .through(issues)
    .filter(!excludedIssue(_))
    .filter(_.creator === me)
    .compile
    .toList
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]

    View Slide

  72. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    =
    projects
    .find(inProject)
    .through(issues)
    .head.compile.last
    =
    projects
    .filter(!excludedProject(_))
    .through(issues)
    .filter(!excludedIssue(_))
    .filter(_.creator === me)
    .compile
    .toList
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]]

    View Slide

  73. Inversion of flow control
    def allMyIssuesInProjectsWithExclusions(
    excludedProject: Project => Boolean,
    excludedIssue: Issue => Boolean
    ): IO[List[Issue]
    val projects: Stream[IO, Project]
    val issues: Pipe[IO, Project, Issue]
    =
    projects
    .find(inProject)
    .through(issues)
    .head.compile.last
    =
    projects
    .filter(!excludedProject(_))
    .through(issues)
    .filter(!excludedIssue(_))
    .filter(_.creator === me)
    .compile
    .toList
    def firstIssue(
    inProject: Project => Boolean,
    ): IO[Option[Issue]
    type Pipe[F[_], -I, +O] = Stream[F, I] => Stream[F, O]

    View Slide

  74. Reusable transformations
    val filterText: Pipe[IO, Message, TextMessage] =
    _.evalMap {
    case msg: TextMessage => msg.some.pure[IO]
    case msg => logger.error(s"Not a text message: $msg").as(none)
    }.unNone
    def decode[A: Decoder]: Pipe[IO, String, A] =
    _.map(io.circe.parser.decode[A](_)).evalMap {
    case Right(v) => v.some.pure[IO]
    case Left(e) => logger.error(e)("Decoding error").as(none)
    }.unNone
    consumerMessages
    .through(filterText)
    .map(_.getText)
    .through(decode[UserEvent])

    View Slide

  75. Compositionality and streams

    View Slide

  76. Compositionality principle
    The meaning of an expression is the meaning of its parts
    and the way they are combined together.

    View Slide

  77. Compositionality and referential transparency
    val program: IO[Unit] =
    createFork.bracket(commit(a))(closeFork(_)) >>
    sendEvent(commitSuccessful(data))

    View Slide

  78. Compositionality and referential transparency
    val program: IO[Unit] = {
    val prog1 = createFork.bracket(commit(data))(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 >> prog2
    }
    val program: IO[Unit] =
    createFork.bracket(commit(a))(closeFork(_)) >>
    sendEvent(commitSuccessful(data))

    View Slide

  79. Referential transparency is not always enough
    val program: IO[Unit] = {
    val prog1 = createFork.bracket(commit(data))(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 >> prog2
    }

    View Slide

  80. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }

    View Slide

  81. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }

    View Slide

  82. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }
    program(quiteLongStream(_).compile.toList):

    View Slide

  83. Referential transparency is not always enough
    def program(use: Repository => IO[A]): IO[A] = {
    val prog1 = createFork.bracket(use)(closeFork(_))
    val prog2 = sendEvent(commitSuccessful(data))
    prog1 <* prog2
    }
    program(quiteLongStream(_).compile.toList):

    View Slide

  84. Streams are self-contained
    val program: Stream[IO, Repository] =
    Stream.bracket(createFork)(closeFork(_)) ++
    Stream.eval_(sendEvent(commitSuccessful(data)))
    program.flatMap(quiteLongStream)

    View Slide

  85. def tupleWith42[A](fa: IO[A]): IO[(A, Int)] =
    fa.map(a => (a, 42))
    How can we test this function?

    View Slide

  86. val mostUsefulIssues =
    issueSource.filter(_.upvotes > 100)
    How can we test this stream?

    View Slide

  87. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) =
    issueSource.filter(_.upvotes > 100)
    How can we test this stream?

    View Slide

  88. def mostUsefulIssues[F[_]](issueSource: Stream[F, Issue]) =
    issueSource.filter(_.upvotes > 100)
    How can we test this stream?
    mostUsefulIssues(
    Stream(Issue("#1", "/u/root", 90), Issue("#2", "/u/root", 110))
    ).toList === List(Issue("#2", "/u/root", 110))

    View Slide

  89. fs2 is truly compositional

    View Slide

  90. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction

    View Slide

  91. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    - All combinators work on all the streams of compatible types

    View Slide

  92. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    - All combinators work on all the streams of compatible types
    - Stream scales from pure streams to complex processes with lots of resources and concurrency

    View Slide

  93. fs2 is truly compositional
    - No matter where the data comes from, it's always the same abstraction
    - All combinators work on all the streams of compatible types
    - Stream scales from pure streams to complex processes with lots of resources and concurrency
    - List + IO on superpowers with far less responsibility for you

    View Slide

  94. val converter: Stream[IO, Unit] = Stream.resource(Blocker[IO]).flatMap { blocker =>
    def fahrenheitToCelsius(f: Double): Double =
    (f - 32.0) * (5.0/9.0)
    io.file.readAll[IO](Paths.get("testdata/fahrenheit.txt"), blocker, 4096)
    .through(text.utf8Decode)
    .through(text.lines)
    .filter(s => !s.trim.isEmpty && !s.startsWith("//"))
    .map(line => fahrenheitToCelsius(line.toDouble).toString)
    .intersperse("\n")
    .through(text.utf8Encode)
    .through(io.file.writeAll(Paths.get("testdata/celsius.txt"), blocker))
    }

    View Slide

  95. def server(
    blocker: Blocker
    ): Stream[IO, Resource[IO, fs2.io.tcp.Socket[IO]]] =
    Stream.resource(fs2.io.tcp.SocketGroup[IO](blocker)).flatMap {
    group =>
    group.server[IO](
    new InetSocketAddress("0.0.0.0", 8080)
    )
    }
    val clientMessages = Stream
    .resource(Blocker[IO])
    .flatMap(server)
    .map {
    Stream
    .resource(_)
    .flatMap(_.reads(1024))
    .through(fs2.text.utf8Decode)
    .through(fs2.text.lines)
    .map("Message: " + _)
    }
    .parJoin(maxOpen = 10)

    View Slide

  96. View Slide

  97. "co.fs2" %% "fs2-core" % "2.1.0"
    https://fs2.io
    Give it a try!

    View Slide

  98. Acknowledgements
    Thanks to Fabio Labella for help with this talk!
    Thanks to Michael Pilquist, Paul Chiusano, Pavel Chlupacek, Fabio
    and all the contributors of fs2 and cats-effect/cats
    Massively inspired by:
    - Declarative control flow with fs2 streams by Fabio Labella:
    https://www.youtube.com/watch?v=x3GLwl1FxcA
    - Compositional Programming by Runar Bjarnason:
    https://www.youtube.com/watch?v=ElLxn_l7P2I

    View Slide

  99. Thank you
    blog.kubukoz.com
    @kubukoz
    Slides: https://speakerdeck.com/kubukoz
    Code: https://git.io/fjpEv

    View Slide

  100. Thank you
    blog.kubukoz.com
    @kubukoz
    Slides: https://speakerdeck.com/kubukoz
    Code: https://git.io/fjpEv
    Find me on YouTube! (https://www.youtube.com/channel/UCBSRCuGz9laxVv0rAnn2O9Q)

    View Slide