Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reactive in Reverse, a guest talk by Daniel Spiewak

Reactive in Reverse, a guest talk by Daniel Spiewak

The New York Times Developers invited Daniel Spiewak to give a guest talk at on reactive programming at TimesOpen, our public developer event series. Daniel is a software developer based out of Boulder, CO. Over the years, he has worked with Java, Scala, Ruby, C/C++, ML, Clojure and several experimental languages. He currently spends most of his free time researching parser theory and methodologies, particularly areas where the field intersects with functional language design, domain-specific languages and type theory.

The New York Times Developers

September 10, 2014
Tweet

More Decks by The New York Times Developers

Other Decks in Programming

Transcript

  1. scalaz-stream
    Reactive in Reverse

    View full-size slide

  2. Pull vs Push
    • "Reactive" streams
    • Java 8 streams
    • Akka streams

    View full-size slide

  3. Pull vs Push
    • "Reactive" streams
    • Java 8 streams
    • Akka streams
    • "Coreactive" streams
    • Haskell's Kmett's Machines
    • scalaz-stream

    View full-size slide

  4. Pull vs Push
    • "Reactive" streams
    • Java 8 streams
    • Akka streams
    • "Coreactive" streams
    • Haskell's Kmett's Machines
    • scalaz-stream

    View full-size slide

  5. • Push streams
    • Data assertively pushed into your flow
    • Multi-output trivial; multi-input hard
    Pull vs Push

    View full-size slide

  6. • Push streams
    • Data assertively pushed into your flow
    • Multi-output trivial; multi-input hard
    • Pull streams
    • "Turn the crank" from the end and request data
    • Multi-output hard; multi-input trivial
    Pull vs Push

    View full-size slide

  7. • Push streams
    • Backpressure is something you need to design
    • More intuitive control flow (imperatively)
    Pull vs Push

    View full-size slide

  8. • Push streams
    • Backpressure is something you need to design
    • More intuitive control flow (imperatively)
    • Pull streams
    • Backpressure is trivial (it "just works")
    • More declarative control, which can be weird
    Pull vs Push

    View full-size slide

  9. Concepts
    • Task[A]
    • Like Future, but more controlled
    • Process[Task, A]
    • A strict sequence of actions

    View full-size slide

  10. Concepts: Task
    • Fully lazy

    View full-size slide

  11. Concepts: Task
    • Fully lazy
    • Creating a Future executes immediately

    View full-size slide

  12. Concepts: Task
    • Fully lazy
    • Creating a Future executes immediately
    • No more memory leaks!

    View full-size slide

  13. Concepts: Task
    • Fully lazy
    • Creating a Future executes immediately
    • No more memory leaks!
    • Easy to move tasks between thread pools

    View full-size slide

  14. Concepts: Task
    • Fully lazy
    • Creating a Future executes immediately
    • No more memory leaks!
    • Easy to move tasks between thread pools
    • Better thread utilization

    View full-size slide

  15. Concepts: Task
    • Fully lazy
    • Creating a Future executes immediately
    • No more memory leaks!
    • Easy to move tasks between thread pools
    • Better thread utilization
    • Explicit parallelism

    View full-size slide

  16. def fib(n: Int): Task[Int] = n match {
    case 0 | 1 => Task now 1
    case n => {
    for {
    x <- fib(n - 1)
    y <- fib(n - 2)
    } yield x + y
    }
    }
    fib(42).run

    View full-size slide

  17. def fib(n: Int): Task[Int] = n match {
    case 0 | 1 => Task now 1
    case n => {
    val ND = Nondeterminism[Task]
    for {
    pair <- ND.both(fib(n - 1), fib(n - 2))
    (x, y) = pair
    } yield x + y
    }
    }
    fib(42).run

    View full-size slide

  18. def shiftPool[A](task: Task[A]): Task[A] =
    Task({ task })(MyThreadPool).join

    View full-size slide

  19. def shiftPool[A](task: Task[A]): Task[A] =
    Task.fork(task)(MyThreadPool)

    View full-size slide

  20. def futureToTask[A](f: Future[A]): Task[A] = {
    Task async { cb =>
    f onComplete {
    case Success(v) => cb(\/.right(v))
    case Failure(e) => cb(\/.left(v))
    }
    }
    }

    View full-size slide

  21. def futureToTask[A](f: Future[A]): Task[A] = {
    Task async { cb =>
    f onComplete {
    case Success(v) => cb(\/.right(v))
    case Failure(e) => cb(\/.left(v))
    }
    }
    }

    View full-size slide

  22. Concepts: Process
    • An ordered sequence of actions
    • Ask for an action…then the next…then the next
    • If you can't keep up, you ask less frequently
    • Easy to merge (just ask for data from either "side")
    • Explicit parallelism

    View full-size slide

  23. def fetchUrl(num: Int): Task[String] = {
    val fetch: Task[Task[String]] = Task delay {
    val svc = url(s"http://api.stuff.com/record/$num")
    Task fork futureToTask(Http(svc OK as.String))
    }
    fetch.join
    }

    View full-size slide

  24. val nums: Process[Task, Int] = Process.range(0, 10)
    val adjusted = nums map { _ * 2 } filter { _ < 10 }
    val pages = adjusted flatMap { num =>
    Process.eval(fetchUrl(num))
    }
    val found = pages find { _ contains "Waldo!" }
    val stuff: Task[Unit] = found to io.stdOutLines run
    stuff.run

    View full-size slide

  25. val nums1: Process[Task, Int] = Process.range(0, 10)
    val nums2: Process[Task, Int] = Process.range(11, 20)
    val nums: Process[Task, Int] = nums1 interleave nums2
    ...

    View full-size slide

  26. val i = new AtomicInteger
    val read = Task delay {
    i.getAndIncrement()
    }
    val src = Process.eval(read).repeat
    val left = src map { i => s"left: $i" }
    val right = src map { i => s"right: $i" }
    left interleave right to io.stdOutLines

    View full-size slide

  27. left: 0
    right: 1
    left: 2
    right: 3
    left: 4
    right: 5
    left: 6
    right: 7
    left: 8
    right: 9
    left: 10
    right: 11
    left: 12
    right: 13
    ...

    View full-size slide

  28. // bounded queues are for wimps...

    View full-size slide

  29. val queue = new ArrayBlockingQueue[Message](10)
    // looks like I'm a wimp
    val read: Task[Message] = Task delay { queue.take() }
    val src: Process[Task, Message] =
    Process.eval(read).repeat
    ...
    // bounded queues are for wimps...

    View full-size slide

  30. val queue = async.blockingQueue[Message](10)
    val src: Process[Task, Message] = queue.dequeue
    ...

    View full-size slide

  31. Sinks
    • Data has to go somewhere

    View full-size slide

  32. Sinks
    • Data has to go somewhere
    • Writing out to a channel

    View full-size slide

  33. Sinks
    • Data has to go somewhere
    • Writing out to a channel
    • Writing to disk

    View full-size slide

  34. Sinks
    • Data has to go somewhere
    • Writing out to a channel
    • Writing to disk
    • …or all of the above

    View full-size slide

  35. Sinks
    • Data has to go somewhere
    • Writing out to a channel
    • Writing to disk
    • …or all of the above
    • What is a sink anyway?

    View full-size slide

  36. Sinks
    • Data has to go somewhere
    • Writing out to a channel
    • Writing to disk
    • …or all of the above
    • What is a sink anyway?
    • A stream of functions!

    View full-size slide

  37. type Sink[F[_], A] = Process[F, A => F[Unit]]

    View full-size slide

  38. def write(str: String): Task[Unit] =
    Task delay { println(str) }
    val sink: Sink[Task, String] = Process.constant(write _)
    val src = Process.range(0, 10) map { _.toString }
    val results = src zip sink flatMap {
    case (str, f) => Process eval f(str)
    }
    val universe: Task[Unit] = results.run

    View full-size slide

  39. val stdOut: Sink[Task, String] = ...
    val channel: Sink[Task, String] = ...
    val src = Process.range(0, 10) map { _.toString }
    val results = src zip stdOut zip channel flatMap {
    case ((str, f1), f2) => {
    for {
    _ <- Process eval f1(str)
    _ <- Process eval f2(str)
    } yield ()
    }
    }
    val universe: Task[Unit] = results.run

    View full-size slide

  40. val stdOut: Sink[Task, String] = ...
    val channel: Sink[Task, String] = ...
    val src = Process.range(0, 10) map { _.toString }
    val results = src observe stdOut to channel
    val universe: Task[Unit] = results.run

    View full-size slide

  41. def debug[A](stream: Process[Task, A]): Process[Task, A] =
    stream map { a => s"debug: $a" } observe io.stdOutLines

    View full-size slide

  42. Concurrency
    • Always explicit!

    View full-size slide

  43. Concurrency
    • Always explicit!
    • Two forms of parallelism
    • Racing two streams into one
    • Turning a stream "sideways"

    View full-size slide

  44. Concurrency
    • Always explicit!
    • Two forms of parallelism
    • Racing two streams into one
    • Turning a stream "sideways"
    • Almost everything implemented on top of wye

    View full-size slide

  45. val left: Process[Task, Message] = ...
    val right: Process[Task, Message] = ...
    val merged: Process[Task, Message] =
    left.wye(right)(wye.merge)

    View full-size slide

  46. val left: Process[Task, Message] = ...
    val right: Process[Task, Message] = ...
    val merged: Process[Task, Message] =
    left merge right // should be "race"

    View full-size slide

  47. val left: Process[Task, Message] = ...
    val right: Process[Task, Line] = ...
    // oh NOES! teh symbols cometh!
    val merged: Process[Task, Message \/ Line] =
    left either right

    View full-size slide

  48. Useful wyes
    • wye.merge

    View full-size slide

  49. Useful wyes
    • wye.merge
    • wye.either

    View full-size slide

  50. Useful wyes
    • wye.merge
    • wye.either
    • wye.yip

    View full-size slide

  51. Useful wyes
    • wye.merge
    • wye.either
    • wye.yip
    • wye.drainL / wye.drainR

    View full-size slide

  52. Useful wyes
    • wye.merge
    • wye.either
    • wye.yip
    • wye.drainL / wye.drainR
    • Doesn't work!

    View full-size slide

  53. val nums: Process[Task, Int] = Process.range(0, 10)
    val adjusted = nums map { _ * 2 } filter { _ < 10 }
    val pages = adjusted flatMap { num =>
    Process.eval(fetchUrl(num))
    }

    View full-size slide

  54. val nums: Process[Task, Int] = Process.range(0, 10)
    val adjusted = nums map { _ * 2 } filter { _ < 10 }
    val pages: Process[Task, Task[String]] =
    adjusted map { num =>
    fetchUrl(num)
    }
    val parallel: Process[Task, String] =
    pages.gather(4)

    View full-size slide

  55. gather(n)
    • Grabs chunks of n and parallelizes

    View full-size slide

  56. gather(n)
    • Grabs chunks of n and parallelizes
    • Last chunk of stream may be truncated

    View full-size slide

  57. gather(n)
    • Grabs chunks of n and parallelizes
    • Last chunk of stream may be truncated
    • Great for finite streams!

    View full-size slide

  58. gather(n)
    • Grabs chunks of n and parallelizes
    • Last chunk of stream may be truncated
    • Great for finite streams!
    • Causes DEADLOCK on infinite streams

    View full-size slide

  59. gather(n)
    • Grabs chunks of n and parallelizes
    • Last chunk of stream may be truncated
    • Great for finite streams!
    • Causes DEADLOCK on infinite streams
    • Don't use if you source from a queue!

    View full-size slide

  60. val nums: Process[Task, Int] = Process.range(0, 10)
    val adjusted = nums map { _ * 2 } filter { _ < 10 }
    val pages: Process[Task, Process[Task, String]] =
    adjusted map { num =>
    Process.eval(fetchUrl(num))
    }
    val parallel: Process[Task, String] =
    merge.mergeN(pages)

    View full-size slide

  61. merge.mergeN
    • A little weirder to use…

    View full-size slide

  62. merge.mergeN
    • A little weirder to use…
    • Process of Process

    View full-size slide

  63. merge.mergeN
    • A little weirder to use…
    • Process of Process
    • Uses a variable bounded queue

    View full-size slide

  64. merge.mergeN
    • A little weirder to use…
    • Process of Process
    • Uses a variable bounded queue
    • Races all input streams

    View full-size slide

  65. merge.mergeN
    • A little weirder to use…
    • Process of Process
    • Uses a variable bounded queue
    • Races all input streams
    • Up to n at a time

    View full-size slide

  66. merge.mergeN
    • A little weirder to use…
    • Process of Process
    • Uses a variable bounded queue
    • Races all input streams
    • Up to n at a time
    • Almost always what you really want

    View full-size slide

  67. Chat Server
    • Uses scalaz-netty project

    View full-size slide

  68. Chat Server
    • Uses scalaz-netty project
    • Currently closed-source, but OSS soon™!

    View full-size slide

  69. Chat Server
    • Uses scalaz-netty project
    • Currently closed-source, but OSS soon™!
    • Would also work with scalaz-nio

    View full-size slide

  70. Chat Server
    • Uses scalaz-netty project
    • Currently closed-source, but OSS soon™!
    • Would also work with scalaz-nio
    • Uses scodec

    View full-size slide

  71. Chat Server
    • Uses scalaz-netty project
    • Currently closed-source, but OSS soon™!
    • Would also work with scalaz-nio
    • Uses scodec
    • Use this. Use it. It's amazing.

    View full-size slide

  72. Chat Server
    • Uses scalaz-netty project
    • Currently closed-source, but OSS soon™!
    • Would also work with scalaz-nio
    • Uses scodec
    • Use this. Use it. It's amazing.
    • Demonstrates the power of Process abstraction

    View full-size slide

  73. Server
    • Accept connections asynchronously
    • …and in parallel!

    View full-size slide

  74. Server
    • Accept connections asynchronously
    • …and in parallel!
    • Pipe inbound data to a relay queue

    View full-size slide

  75. Server
    • Accept connections asynchronously
    • …and in parallel!
    • Pipe inbound data to a relay queue
    • Pipe relay queue into the outbound channel
    • …including all history!

    View full-size slide

  76. Server
    • Accept connections asynchronously
    • …and in parallel!
    • Pipe inbound data to a relay queue
    • Pipe relay queue into the outbound channel
    • …including all history!
    • Continue until client closes connection

    View full-size slide

  77. val address: InetSocketAddress = ???
    val relay = async.topic[BitVector]
    val handlers = Netty server address map { client =>
    for {
    Exchange(src, sink) <- client
    in = src to relay.publish
    out = relay.subscribe to sink
    _ <- in merge out
    } yield ()
    }
    val server: Task[Unit] = merge.mergeN(handlers).run

    View full-size slide

  78. Client
    • Establish connection

    View full-size slide

  79. Client
    • Establish connection
    • Pipe standard input to the server (as UTF-8)

    View full-size slide

  80. Client
    • Establish connection
    • Pipe standard input to the server (as UTF-8)
    • Pipe server response to standard output

    View full-size slide

  81. Client
    • Establish connection
    • Pipe standard input to the server (as UTF-8)
    • Pipe server response to standard output
    • Continue until user fail-sauce Ctrl-C kills us

    View full-size slide

  82. implicit val codec: Codec[String] = utf8
    def transcode(ex: Exchange[BitVector, BitVector]) = {
    val decoder = decode.many[String]
    val encoder = encode.many[String]
    val Exchange(src, sink) = ex
    val src2 = src flatMap decoder.decode
    val sink2 = sink pipeIn encoder.encoder
    Exchange(src2, sink2)
    }

    View full-size slide

  83. val clientP = for {
    rawData <- Netty connect address
    Exchange(src, sink) = transcode(rawData)
    in = src to io.stdOutLines
    out = io.stdInLines to sink
    _ <- in merge out
    } yield ()
    val client: Task[Unit] = clientP.run

    View full-size slide

  84. Notes
    • Resources are managed and cannot leak

    View full-size slide

  85. Notes
    • Resources are managed and cannot leak
    • Logic is pure and encapsulated from networking

    View full-size slide

  86. Notes
    • Resources are managed and cannot leak
    • Logic is pure and encapsulated from networking
    • Backpressure "just works" (sort of)

    View full-size slide

  87. Notes
    • Resources are managed and cannot leak
    • Logic is pure and encapsulated from networking
    • Backpressure "just works" (sort of)
    • Our Topic is unbounded, because I'm lazy

    View full-size slide

  88. Notes
    • Resources are managed and cannot leak
    • Logic is pure and encapsulated from networking
    • Backpressure "just works" (sort of)
    • Our Topic is unbounded, because I'm lazy
    • Handshaking would be almost trivial

    View full-size slide

  89. Notes
    • Resources are managed and cannot leak
    • Logic is pure and encapsulated from networking
    • Backpressure "just works" (sort of)
    • Our Topic is unbounded, because I'm lazy
    • Handshaking would be almost trivial
    • Client and server logic looks almost the same!

    View full-size slide

  90. • A different take on "reactive"

    View full-size slide

  91. • A different take on "reactive"
    • Purity helps us understand complex logic!

    View full-size slide

  92. • A different take on "reactive"
    • Purity helps us understand complex logic!
    • No more puzzling about state or resource leaks

    View full-size slide

  93. • A different take on "reactive"
    • Purity helps us understand complex logic!
    • No more puzzling about state or resource leaks
    • Simple and easy combinators scale well

    View full-size slide

  94. • A different take on "reactive"
    • Purity helps us understand complex logic!
    • No more puzzling about state or resource leaks
    • Simple and easy combinators scale well
    • You know almost everything you need

    View full-size slide