Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Applicative Application

Bucharest FP
October 01, 2018

An Applicative Application

Bucharest FP

October 01, 2018
Tweet

More Decks by Bucharest FP

Other Decks in Programming

Transcript

  1. An
    Applicative
    Application

    View Slide

  2. An
    Applicative
    Application
    How to obtain OpenTSDB query batching by
    exploiting the applicative structure of a computation.

    View Slide

  3. A Word of Warning

    View Slide

  4. A Word of Warning
    I had trouble explaining this stuff

    View Slide

  5. View Slide

  6. Today's Roadmap

    View Slide

  7. Today's Roadmap
    • The Problem

    View Slide

  8. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage

    View Slide

  9. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage
    • Business Use Case — Time Series Data Aggregation

    View Slide

  10. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage
    • Business Use Case — Time Series Data Aggregation
    • The Solution

    View Slide

  11. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage
    • Business Use Case — Time Series Data Aggregation
    • The Solution
    • Inspiration

    View Slide

  12. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage
    • Business Use Case — Time Series Data Aggregation
    • The Solution
    • Inspiration
    • Intuitions — Functions, Monads & Applicatives

    View Slide

  13. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage
    • Business Use Case — Time Series Data Aggregation
    • The Solution
    • Inspiration
    • Intuitions — Functions, Monads & Applicatives
    • Code

    View Slide

  14. Today's Roadmap
    • The Problem
    • OpenTSDB Overview — Time Series Data Storage
    • Business Use Case — Time Series Data Aggregation
    • The Solution
    • Inspiration
    • Intuitions — Functions, Monads & Applicatives
    • Code
    • Future Work

    View Slide

  15. OpenTSDB Overview
    Time Series Data Storage

    View Slide

  16. http POST 'localhost:4242/api/put' << EOF
    [
    {
    "metric": "metric-a",
    "timestamp": 1538092800,
    "value": 10,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    },
    {
    "metric": "metric-b",
    "timestamp": 1538092801,
    "value": 20,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    }
    ]
    EOF

    View Slide

  17. http POST 'localhost:4242/api/put' << EOF
    [
    {
    "metric": "metric-a",
    "timestamp": 1538092800,
    "value": 10,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    },
    {
    "metric": "metric-b",
    "timestamp": 1538092801,
    "value": 20,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    }
    ]
    EOF

    View Slide

  18. http POST 'localhost:4242/api/put' << EOF
    [
    {
    "metric": "metric-a",
    "timestamp": 1538092800,
    "value": 10,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    },
    {
    "metric": "metric-b",
    "timestamp": 1538092801,
    "value": 20,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    }
    ]
    EOF

    View Slide

  19. http POST 'localhost:4242/api/put' << EOF
    [
    {
    "metric": "metric-a",
    "timestamp": 1538092800,
    "value": 10,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    },
    {
    "metric": "metric-b",
    "timestamp": 1538092801,
    "value": 20,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    }
    ]
    EOF

    View Slide

  20. http POST 'localhost:4242/api/put' << EOF
    [
    {
    "metric": "metric-a",
    "timestamp": 1538092800,
    "value": 10,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    },
    {
    "metric": "metric-b",
    "timestamp": 1538092801,
    "value": 20,
    "tags": {
    "t1": "v1",
    "t2": "v2"
    }
    }
    ]
    EOF

    View Slide

  21. http POST 'localhost:4242/api/query' << EOF
    {
    "start": "2018/05/01 00:10:00",
    "end": "2018/05/01 00:10:05",
    "timezone": "UTC",
    "queries": [
    {
    "metric": "metric-a",
    "aggregator": "none",
    "tags": {}
    },
    {
    "metric": "metric-b",
    "aggregator": "none",
    "tags": {}
    }
    ]
    }
    EOF

    View Slide

  22. View Slide

  23. Business Use Case
    Time Series Data Aggregation

    View Slide

  24. Business Use Case

    View Slide

  25. Business Use Case
    • We store historical data in OpenTSDB

    View Slide

  26. Business Use Case
    • We store historical data in OpenTSDB
    • Basis for end-of-month reports

    View Slide

  27. Business Use Case
    • We store historical data in OpenTSDB
    • Basis for end-of-month reports
    • Past: everything computed at end of month from raw OpenTSDB data

    View Slide

  28. Business Use Case
    • We store historical data in OpenTSDB
    • Basis for end-of-month reports
    • Past: everything computed at end of month from raw OpenTSDB data
    • Now: pre-compute aggregations continuously at various
    frequencies, e.g., 5 minutes, and use these for reporting

    View Slide

  29. Business Use Case
    • We store historical data in OpenTSDB
    • Basis for end-of-month reports
    • Past: everything computed at end of month from raw OpenTSDB data
    • Now: pre-compute aggregations continuously at various
    frequencies, e.g., 5 minutes, and use these for reporting
    • We call these aggregations roll-ups

    View Slide

  30. Business Use Case
    • We store historical data in OpenTSDB
    • Basis for end-of-month reports
    • Past: everything computed at end of month from raw OpenTSDB data
    • Now: pre-compute aggregations continuously at various
    frequencies, e.g., 5 minutes, and use these for reporting
    • We call these aggregations roll-ups
    • Decreases time to generate reports

    View Slide

  31. Roll-Ups as Functions

    View Slide

  32. def rollup(metric: List[Datapoint]): Double

    View Slide

  33. case class Datapoint(
    timestamp: Instant,
    value: Double,
    )
    def rollup(metric: List[Datapoint]): Double

    View Slide

  34. case class Datapoint(
    timestamp: Instant,
    value: Double,
    )
    def rollup(
    metric0: List[Datapoint],
    metric1: List[Datapoint],
    ...
    metricN: List[Datapoint],
    ): Double

    View Slide

  35. Goal: Optimize Network
    Access

    View Slide

  36. Inspiration

    View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. View Slide

  44. View Slide

  45. Similar Ideas: Haxl

    View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. Applicative lets us perform global optimizations
    on the Abstract Syntax Tree of an Embedded DSL.

    View Slide

  50. Intuitions
    Functions, Monads & Applicatives

    View Slide

  51. f(a)

    View Slide

  52. def example1[A, B](f: A => B, a: A): B =
    f(a)

    View Slide

  53. def example1[A, B](f: A => B, a: A): B =
    f(a)
    def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = {
    val ff = Applicative[F].pure(f)
    val fa = Applicative[F].pure(a)
    ff.ap(fa)
    }

    View Slide

  54. def example1[A, B](f: A => B, a: A): B =
    f(a)
    def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = {
    val ff = Applicative[F].pure(f)
    val fa = Applicative[F].pure(a)
    ff.ap(fa)
    }
    def example3[F[_]: Applicative, A, B](f: F[A => B], a: F[A]): F[B] =
    f.ap(a)

    View Slide

  55. def example1[A, B](f: A => B, a: A): B =
    f.apply(a)
    def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = {
    val ff = Applicative[F].pure(f)
    val fa = Applicative[F].pure(a)
    ff.ap(fa)
    }
    def example3[F[_]: Applicative, A, B](f: F[A => B], a: F[A]): F[B] =
    f.ap(a)

    View Slide

  56. Applicative
    Lets us embed function-like DSLs into our programs.

    View Slide

  57. Data and Effect
    Dependencies

    View Slide

  58. View Slide

  59. Functions: 1
    Possible Side-Effects
    Without Data-Dependencies

    View Slide

  60. // Potential side-effects, but we don't know.
    def foo(): String = ???
    def bar(): String = ???

    View Slide

  61. // Potential side-effects, but we don't know.
    def foo(): String = ???
    def bar(): String = ???
    // No data dependency between function calls.
    val a = foo()
    val b = bar()

    View Slide

  62. Functions: 2
    Possible Side-Effects
    With Data-Dependencies

    View Slide

  63. // Potential side-effects, but we don't know.
    def foo(): String = ???
    def bar(a: String): String = ???

    View Slide

  64. // Potential side-effects, but we don't know.
    def foo(): String = ???
    def bar(a: String): String = ???
    // Data-dependency between functions calls.
    val a = foo()
    val b = bar(a)

    View Slide

  65. Functions: 3
    Without Side-Effects
    Without Data-Dependencies

    View Slide

  66. // Effectful functions; encoded in signature.
    def foo(): Task[String] = ???
    def bar(): Task[String] = ???

    View Slide

  67. // Effectful functions; encoded in signature.
    def foo(): Task[String] = ???
    def bar(): Task[String] = ???
    // No data dependency between function calls.
    val a = foo()
    val b = bar()

    View Slide

  68. // Effectful functions; encoded in signature.
    def foo(): Task[String] = ???
    def bar(): Task[String] = ???
    // No data dependency between function calls.
    val a = foo()
    val b = bar()
    val c = a.flatMap(_ => b) // sequential

    View Slide

  69. // Effectful functions; encoded in signature.
    def foo(): Task[String] = ???
    def bar(): Task[String] = ???
    // No data dependency between function calls.
    val a = foo()
    val b = bar()
    val c = a.flatMap(_ => b) // sequential
    val d = (a, b).mapN((a, b) => ...)
    c.unsafeRunSync()

    View Slide

  70. Functions: 4
    Without Side-Effects
    With Data-Dependencies

    View Slide

  71. // Effectful functions; encoded in signature.
    def foo(): IO[String] = ???
    def bar(a: String): IO[String] = ???

    View Slide

  72. // Effectful functions; encoded in signature.
    def foo(): IO[String] = ???
    def bar(a: String): IO[String] = ???
    // Data-dependency between functions calls.
    val c = foo().flatMap(a => bar(a))
    c.unsafeRunSync()

    View Slide

  73. Flashback
    Applicative lets us embed function-like DSLs into
    our programs.
    Applicative lets us perform global optimizations
    on the Abstract Syntax Tree of an EDSL.

    View Slide

  74. Roll-Ups are Functions
    Idea: come up with an EDSL to model time series
    data fetching and constrain its usage to an Applicative
    interface.
    Then, statically analyze the EDSL to batch and deduplicate
    issued queries before making the actual network calls.

    View Slide

  75. // Independent computations
    val resultAB = rollupAB(metricA(), metricB())
    val resultAC = rollupAC(metricA(), metricC())

    View Slide

  76. // Independent computations
    val mA =
    val resultAB = rollupAB(metricA(), metricB())
    val resultAC = rollupAC(metricA(), metricC())

    View Slide

  77. // Optimize: common subexpression elimination
    val mA = metricA()
    val resultAB = rollupAB(mA, metricB())
    val resultAC = rollupAC(mA, metricC())

    View Slide

  78. View Slide

  79. Let's Write an
    Optimizing
    EDSL Compiler

    View Slide

  80. Observations
    Semantically-parallel Applicative instances don't bode
    well with Monad instances. If a type is a monadic, a lawful
    Applicative instance has to be sequential.
    This is why Cats exposes a Parallel type-class, inspired by
    PureScript, which allows client code to choose between
    semantically-sequential and semantically-parallel Applicative
    instances.

    View Slide

  81. Future Work

    View Slide

  82. Future Work
    Static Analysis on Arrow Computations

    View Slide

  83. Call for Presentations
    [email protected]

    View Slide

  84. Questions!

    View Slide

  85. Thanks!

    View Slide