An Applicative Application

Slide 1

Slide 1 text

Slide 2

Slide 2 text

An Applicative Application How to obtain OpenTSDB query batching by exploiting the applicative structure of a computation.

Slide 3

Slide 3 text

A Word of Warning

Slide 4

Slide 4 text

A Word of Warning I had trouble explaining this stuff

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Today's Roadmap

Slide 7

Slide 7 text

Today's Roadmap • The Problem

Slide 8

Slide 8 text

Today's Roadmap • The Problem • OpenTSDB Overview — Time Series Data Storage

Slide 9

Slide 9 text

Today's Roadmap • The Problem • OpenTSDB Overview — Time Series Data Storage • Business Use Case — Time Series Data Aggregation

Slide 10

Slide 10 text

Today's Roadmap • The Problem • OpenTSDB Overview — Time Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution

Slide 11

Slide 11 text

Today's Roadmap • The Problem • OpenTSDB Overview — Time Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution • Inspiration

Slide 12

Slide 12 text

Today's Roadmap • The Problem • OpenTSDB Overview — Time Series Data Storage • Business Use Case — Time Series Data Aggregation • The Solution • Inspiration • Intuitions — Functions, Monads & Applicatives

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

OpenTSDB Overview Time Series Data Storage

Slide 16

Slide 16 text

http POST 'localhost:4242/api/put' << EOF [ { "metric": "metric-a", "timestamp": 1538092800, "value": 10, "tags": { "t1": "v1", "t2": "v2" } }, { "metric": "metric-b", "timestamp": 1538092801, "value": 20, "tags": { "t1": "v1", "t2": "v2" } } ] EOF

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

http POST 'localhost:4242/api/query' << EOF { "start": "2018/05/01 00:10:00", "end": "2018/05/01 00:10:05", "timezone": "UTC", "queries": [ { "metric": "metric-a", "aggregator": "none", "tags": {} }, { "metric": "metric-b", "aggregator": "none", "tags": {} } ] } EOF

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Business Use Case Time Series Data Aggregation

Slide 24

Slide 24 text

Business Use Case

Slide 25

Slide 25 text

Business Use Case • We store historical data in OpenTSDB

Slide 26

Slide 26 text

Business Use Case • We store historical data in OpenTSDB • Basis for end-of-month reports

Slide 27

Slide 27 text

Business Use Case • We store historical data in OpenTSDB • Basis for end-of-month reports • Past: everything computed at end of month from raw OpenTSDB data

Slide 28

Slide 28 text

Business Use Case • We store historical data in OpenTSDB • Basis for end-of-month reports • Past: everything computed at end of month from raw OpenTSDB data • Now: pre-compute aggregations continuously at various frequencies, e.g., 5 minutes, and use these for reporting

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Roll-Ups as Functions

Slide 32

Slide 32 text

def rollup(metric: List[Datapoint]): Double

Slide 33

Slide 33 text

case class Datapoint( timestamp: Instant, value: Double, ) def rollup(metric: List[Datapoint]): Double

Slide 34

Slide 34 text

case class Datapoint( timestamp: Instant, value: Double, ) def rollup( metric0: List[Datapoint], metric1: List[Datapoint], ... metricN: List[Datapoint], ): Double

Slide 35

Slide 35 text

Goal: Optimize Network Access

Slide 36

Slide 36 text

Inspiration

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

Similar Ideas: Haxl

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Applicative lets us perform global optimizations on the Abstract Syntax Tree of an Embedded DSL.

Slide 50

Slide 50 text

Intuitions Functions, Monads & Applicatives

Slide 51

Slide 51 text

f(a)

Slide 52

Slide 52 text

def example1[A, B](f: A => B, a: A): B = f(a)

Slide 53

Slide 53 text

def example1[A, B](f: A => B, a: A): B = f(a) def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = { val ff = Applicative[F].pure(f) val fa = Applicative[F].pure(a) ff.ap(fa) }

Slide 54

Slide 54 text

def example1[A, B](f: A => B, a: A): B = f(a) def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = { val ff = Applicative[F].pure(f) val fa = Applicative[F].pure(a) ff.ap(fa) } def example3[F[_]: Applicative, A, B](f: F[A => B], a: F[A]): F[B] = f.ap(a)

Slide 55

Slide 55 text

def example1[A, B](f: A => B, a: A): B = f.apply(a) def example2[F[_]: Applicative, A, B](f: A => B, a: A): F[B] = { val ff = Applicative[F].pure(f) val fa = Applicative[F].pure(a) ff.ap(fa) } def example3[F[_]: Applicative, A, B](f: F[A => B], a: F[A]): F[B] = f.ap(a)

Slide 56

Slide 56 text

Applicative Lets us embed function-like DSLs into our programs.

Slide 57

Slide 57 text

Data and Effect Dependencies

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

Functions: 1 Possible Side-Effects Without Data-Dependencies

Slide 60

Slide 60 text

// Potential side-effects, but we don't know. def foo(): String = ??? def bar(): String = ???

Slide 61

Slide 61 text

// Potential side-effects, but we don't know. def foo(): String = ??? def bar(): String = ??? // No data dependency between function calls. val a = foo() val b = bar()

Slide 62

Slide 62 text

Functions: 2 Possible Side-Effects With Data-Dependencies

Slide 63

Slide 63 text

// Potential side-effects, but we don't know. def foo(): String = ??? def bar(a: String): String = ???

Slide 64

Slide 64 text

// Potential side-effects, but we don't know. def foo(): String = ??? def bar(a: String): String = ??? // Data-dependency between functions calls. val a = foo() val b = bar(a)

Slide 65

Slide 65 text

Functions: 3 Without Side-Effects Without Data-Dependencies

Slide 66

Slide 66 text

// Effectful functions; encoded in signature. def foo(): Task[String] = ??? def bar(): Task[String] = ???

Slide 67

Slide 67 text

// Effectful functions; encoded in signature. def foo(): Task[String] = ??? def bar(): Task[String] = ??? // No data dependency between function calls. val a = foo() val b = bar()

Slide 68

Slide 68 text

// Effectful functions; encoded in signature. def foo(): Task[String] = ??? def bar(): Task[String] = ??? // No data dependency between function calls. val a = foo() val b = bar() val c = a.flatMap(_ => b) // sequential

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Functions: 4 Without Side-Effects With Data-Dependencies

Slide 71

Slide 71 text

// Effectful functions; encoded in signature. def foo(): IO[String] = ??? def bar(a: String): IO[String] = ???

Slide 72

Slide 72 text

// Effectful functions; encoded in signature. def foo(): IO[String] = ??? def bar(a: String): IO[String] = ??? // Data-dependency between functions calls. val c = foo().flatMap(a => bar(a)) c.unsafeRunSync()

Slide 73

Slide 73 text

Flashback Applicative lets us embed function-like DSLs into our programs. Applicative lets us perform global optimizations on the Abstract Syntax Tree of an EDSL.

Slide 74

Slide 74 text

Roll-Ups are Functions Idea: come up with an EDSL to model time series data fetching and constrain its usage to an Applicative interface. Then, statically analyze the EDSL to batch and deduplicate issued queries before making the actual network calls.

Slide 75

Slide 75 text

// Independent computations val resultAB = rollupAB(metricA(), metricB()) val resultAC = rollupAC(metricA(), metricC())

Slide 76

Slide 76 text

// Independent computations val mA = val resultAB = rollupAB(metricA(), metricB()) val resultAC = rollupAC(metricA(), metricC())

Slide 77

Slide 77 text

// Optimize: common subexpression elimination val mA = metricA() val resultAB = rollupAB(mA, metricB()) val resultAC = rollupAC(mA, metricC())

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

Let's Write an Optimizing EDSL Compiler

Slide 80

Slide 80 text

Observations Semantically-parallel Applicative instances don't bode well with Monad instances. If a type is a monadic, a lawful Applicative instance has to be sequential. This is why Cats exposes a Parallel type-class, inspired by PureScript, which allows client code to choose between semantically-sequential and semantically-parallel Applicative instances.