An Applicative Application

0286822f506fc4621bd3ea0bcbfef238?s=47 Bucharest FP
October 01, 2018

An Applicative Application

0286822f506fc4621bd3ea0bcbfef238?s=128

Bucharest FP

October 01, 2018
Tweet

Transcript

  1. An Applicative Application

  2. An Applicative Application How to obtain OpenTSDB query batching by

    exploiting the applicative structure of a computation.
  3. A Word of Warning

  4. A Word of Warning I had trouble explaining this stuff

  5. None
  6. Today's Roadmap

  7. Today's Roadmap • The Problem

  8. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage
  9. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage • Business Use Case — Time Series Data Aggregation
  10. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution
  11. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution • Inspiration
  12. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution • Inspiration • Intuitions — Functions, Monads & Applicatives
  13. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution • Inspiration • Intuitions — Functions, Monads & Applicatives • Code
  14. Today's Roadmap • The Problem • OpenTSDB Overview — Time

    Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution • Inspiration • Intuitions — Functions, Monads & Applicatives • Code • Future Work
  15. OpenTSDB Overview Time Series Data Storage

  16. http POST 'localhost:4242/api/put' << EOF [ { "metric": "metric-a", "timestamp":

    1538092800, "value": 10, "tags": { "t1": "v1", "t2": "v2" } }, { "metric": "metric-b", "timestamp": 1538092801, "value": 20, "tags": { "t1": "v1", "t2": "v2" } } ] EOF
  17. http POST 'localhost:4242/api/put' << EOF [ { "metric": "metric-a", "timestamp":

    1538092800, "value": 10, "tags": { "t1": "v1", "t2": "v2" } }, { "metric": "metric-b", "timestamp": 1538092801, "value": 20, "tags": { "t1": "v1", "t2": "v2" } } ] EOF
  18. http POST 'localhost:4242/api/put' << EOF [ { "metric": "metric-a", "timestamp":

    1538092800, "value": 10, "tags": { "t1": "v1", "t2": "v2" } }, { "metric": "metric-b", "timestamp": 1538092801, "value": 20, "tags": { "t1": "v1", "t2": "v2" } } ] EOF
  19. http POST 'localhost:4242/api/put' << EOF [ { "metric": "metric-a", "timestamp":

    1538092800, "value": 10, "tags": { "t1": "v1", "t2": "v2" } }, { "metric": "metric-b", "timestamp": 1538092801, "value": 20, "tags": { "t1": "v1", "t2": "v2" } } ] EOF
  20. http POST 'localhost:4242/api/put' << EOF [ { "metric": "metric-a", "timestamp":

    1538092800, "value": 10, "tags": { "t1": "v1", "t2": "v2" } }, { "metric": "metric-b", "timestamp": 1538092801, "value": 20, "tags": { "t1": "v1", "t2": "v2" } } ] EOF
  21. http POST 'localhost:4242/api/query' << EOF { "start": "2018/05/01 00:10:00", "end":

    "2018/05/01 00:10:05", "timezone": "UTC", "queries": [ { "metric": "metric-a", "aggregator": "none", "tags": {} }, { "metric": "metric-b", "aggregator": "none", "tags": {} } ] } EOF
  22. None
  23. Business Use Case Time Series Data Aggregation

  24. Business Use Case

  25. Business Use Case • We store historical data in OpenTSDB

  26. Business Use Case • We store historical data in OpenTSDB

    • Basis for end-of-month reports
  27. Business Use Case • We store historical data in OpenTSDB

    • Basis for end-of-month reports • Past: everything computed at end of month from raw OpenTSDB data
  28. Business Use Case • We store historical data in OpenTSDB

    • Basis for end-of-month reports • Past: everything computed at end of month from raw OpenTSDB data • Now: pre-compute aggregations continuously at various frequencies, e.g., 5 minutes, and use these for reporting
  29. Business Use Case • We store historical data in OpenTSDB

    • Basis for end-of-month reports • Past: everything computed at end of month from raw OpenTSDB data • Now: pre-compute aggregations continuously at various frequencies, e.g., 5 minutes, and use these for reporting • We call these aggregations roll-ups
  30. Business Use Case • We store historical data in OpenTSDB

    • Basis for end-of-month reports • Past: everything computed at end of month from raw OpenTSDB data • Now: pre-compute aggregations continuously at various frequencies, e.g., 5 minutes, and use these for reporting • We call these aggregations roll-ups • Decreases time to generate reports
  31. Roll-Ups as Functions

  32. def rollup(metric: List[Datapoint]): Double

  33. case class Datapoint( timestamp: Instant, value: Double, ) def rollup(metric:

    List[Datapoint]): Double
  34. case class Datapoint( timestamp: Instant, value: Double, ) def rollup(

    metric0: List[Datapoint], metric1: List[Datapoint], ... metricN: List[Datapoint], ): Double
  35. Goal: Optimize Network Access

  36. Inspiration

  37. None
  38. None
  39. None
  40. None
  41. None
  42. None
  43. None
  44. None
  45. Similar Ideas: Haxl

  46. None
  47. None
  48. None
  49. Applicative lets us perform global optimizations on the Abstract Syntax

    Tree of an Embedded DSL.
  50. Intuitions Functions, Monads & Applicatives

  51. f(a)

  52. def example1[A, B](f: A => B, a: A): B =

    f(a)
  53. def example1[A, B](f: A => B, a: A): B =

    f(a) def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = { val ff = Applicative[F].pure(f) val fa = Applicative[F].pure(a) ff.ap(fa) }
  54. def example1[A, B](f: A => B, a: A): B =

    f(a) def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = { val ff = Applicative[F].pure(f) val fa = Applicative[F].pure(a) ff.ap(fa) } def example3[F[_]: Applicative, A, B](f: F[A => B], a: F[A]): F[B] = f.ap(a)
  55. def example1[A, B](f: A => B, a: A): B =

    f.apply(a) def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = { val ff = Applicative[F].pure(f) val fa = Applicative[F].pure(a) ff.ap(fa) } def example3[F[_]: Applicative, A, B](f: F[A => B], a: F[A]): F[B] = f.ap(a)
  56. Applicative Lets us embed function-like DSLs into our programs.

  57. Data and Effect Dependencies

  58. None
  59. Functions: 1 Possible Side-Effects Without Data-Dependencies

  60. // Potential side-effects, but we don't know. def foo(): String

    = ??? def bar(): String = ???
  61. // Potential side-effects, but we don't know. def foo(): String

    = ??? def bar(): String = ??? // No data dependency between function calls. val a = foo() val b = bar()
  62. Functions: 2 Possible Side-Effects With Data-Dependencies

  63. // Potential side-effects, but we don't know. def foo(): String

    = ??? def bar(a: String): String = ???
  64. // Potential side-effects, but we don't know. def foo(): String

    = ??? def bar(a: String): String = ??? // Data-dependency between functions calls. val a = foo() val b = bar(a)
  65. Functions: 3 Without Side-Effects Without Data-Dependencies

  66. // Effectful functions; encoded in signature. def foo(): Task[String] =

    ??? def bar(): Task[String] = ???
  67. // Effectful functions; encoded in signature. def foo(): Task[String] =

    ??? def bar(): Task[String] = ??? // No data dependency between function calls. val a = foo() val b = bar()
  68. // Effectful functions; encoded in signature. def foo(): Task[String] =

    ??? def bar(): Task[String] = ??? // No data dependency between function calls. val a = foo() val b = bar() val c = a.flatMap(_ => b) // sequential
  69. // Effectful functions; encoded in signature. def foo(): Task[String] =

    ??? def bar(): Task[String] = ??? // No data dependency between function calls. val a = foo() val b = bar() val c = a.flatMap(_ => b) // sequential val d = (a, b).mapN((a, b) => ...) c.unsafeRunSync()
  70. Functions: 4 Without Side-Effects With Data-Dependencies

  71. // Effectful functions; encoded in signature. def foo(): IO[String] =

    ??? def bar(a: String): IO[String] = ???
  72. // Effectful functions; encoded in signature. def foo(): IO[String] =

    ??? def bar(a: String): IO[String] = ??? // Data-dependency between functions calls. val c = foo().flatMap(a => bar(a)) c.unsafeRunSync()
  73. Flashback Applicative lets us embed function-like DSLs into our programs.

    Applicative lets us perform global optimizations on the Abstract Syntax Tree of an EDSL.
  74. Roll-Ups are Functions Idea: come up with an EDSL to

    model time series data fetching and constrain its usage to an Applicative interface. Then, statically analyze the EDSL to batch and deduplicate issued queries before making the actual network calls.
  75. // Independent computations val resultAB = rollupAB(metricA(), metricB()) val resultAC

    = rollupAC(metricA(), metricC())
  76. // Independent computations val mA = val resultAB = rollupAB(metricA(),

    metricB()) val resultAC = rollupAC(metricA(), metricC())
  77. // Optimize: common subexpression elimination val mA = metricA() val

    resultAB = rollupAB(mA, metricB()) val resultAC = rollupAC(mA, metricC())
  78. None
  79. Let's Write an Optimizing EDSL Compiler

  80. Observations Semantically-parallel Applicative instances don't bode well with Monad instances.

    If a type is a monadic, a lawful Applicative instance has to be sequential. This is why Cats exposes a Parallel type-class, inspired by PureScript, which allows client code to choose between semantically-sequential and semantically-parallel Applicative instances.
  81. Future Work

  82. Future Work Static Analysis on Arrow Computations

  83. Call for Presentations ionut.g.stan@gmail.com

  84. Questions!

  85. Thanks!