InfluxDB + Prometheus - Speaker Deck

Slide 1

Slide 1 text

InfluxDB + Prometheus Paul Dix CTO @ InfluxData @pauldix paul@influxdb.com

Slide 2

Slide 2 text

Open Source Time Series Database

Slide 3

Slide 3 text

• OSS - MIT License • Written in Go • SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Commercial Enterprise - HA + Scale Out Clustering

Slide 4

Slide 4 text

Data Model Measurement cpu,host=serverA,num=1,region=west idle=1.667,system=2342.2 1492214400000000000 Tags Fields nanosecond epoch

Slide 5

Slide 5 text

ﬂoat64, int64, bool, string uint64 in next release

Slide 6

Slide 6 text

Example Query select percentile(90, value) from cpu where time > now() - 12h and “region” = ‘west’ group by time(10m), host

Slide 7

Slide 7 text

Process: • OSS - MIT License • Written in Go • Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InﬂuxDB • User Deﬁned Functions • Service Discovery & Pull

Slide 8

Slide 8 text

Collect: • OSS - MIT License • Written in Go • Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InﬂuxDB, Graphite, Kafka, etc.

Slide 9

Slide 9 text

Why is a competitor at Promcon?

Slide 10

Slide 10 text

What is Prometheus?

Slide 11

Slide 11 text

Prometheus Server

Slide 12

Slide 12 text

Alert Manager

Slide 13

Slide 13 text

Exposition Format

Slide 14

Slide 14 text

Client Libraries • Supported • Go • Java • Python • Ruby • Community • C++ • C# • Node.js • PHP • etc…

Slide 15

Slide 15 text

PromQL

Slide 16

Slide 16 text

API /api/v1/series /api/v1/label//values

Slide 17

Slide 17 text

Remote Read/Write API

Slide 18

Slide 18 text

Prometheus is more than just the server

Slide 19

Slide 19 text

Inﬂux + Prometheus today

Slide 20

Slide 20 text

We’re embracing pull & push

Slide 21

Slide 21 text

Prometheus Scrape Targets Kapacitor

Slide 22

Slide 22 text

https://github.com/prometheus/prometheus/issues/2911 https://github.com/inﬂuxdata/prometheus_metric_normalizer

Slide 23

Slide 23 text

Remote Read/Write

Slide 24

Slide 24 text

Prometheus Config # Remote write configuration (for Graphite, OpenTSDB, or InfluxDB). remote_write: - url: "http://localhost:9201/write" # Remote read configuration (for InfluxDB only at the moment). remote_read: - url: "http://localhost:9201/read"

Slide 25

Slide 25 text

metric -> measurement labels -> tags value -> only ﬁeld (ﬂoat64)

Slide 26

Slide 26 text

Remote Read Protobuf over HTTP

Slide 27

Slide 27 text

Reads message Query { int64 start_timestamp_ms = 1; int64 end_timestamp_ms = 2; repeated LabelMatcher matchers = 3; } enum MatchType { EQUAL = 0; NOT_EQUAL = 1; REGEX_MATCH = 2; REGEX_NO_MATCH = 3; } message LabelMatcher { MatchType type = 1; string name = 2; string value = 3; }

Slide 28

Slide 28 text

Remote Read JSON over HTTP

Slide 29

Slide 29 text

Unnecessary Overhead

Slide 30

Slide 30 text

Can we push down processing?

Slide 31

Slide 31 text

Inﬂux + Prometheus future

Slide 32

Slide 32 text

First, pull remote gateway into InﬂuxDB!

Slide 33

Slide 33 text

Inﬂux Data Model tags + value + time

Slide 34

Slide 34 text

Like Prometheus message Sample { double value = 1; int64 timestamp_ms = 2; } message LabelPair { string name = 1; string value = 2; } message TimeSeries { repeated LabelPair labels = 1; // Sorted by time, oldest sample first. repeated Sample samples = 2; }

Slide 35

Slide 35 text

metric is __name__ label

Slide 36

Slide 36 text

But we’ll continue to support int64, bool, string, and uint64

Slide 37

Slide 37 text

IFQL - (Inﬂux Functional Query Language) select(database:"foo",where:{'metric'="requests"}) .range(start:-4h) .window(period:10m) .sum() .interpolate(start:-4h,every:10m,value:{mean($)})

Slide 38

Slide 38 text

Decouple QL from Processing from Storage

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

DAG { "operations": [ { "id": "select", "kind": "select", "spec": { "database": "mydb" } }, { "id": "range", "kind": "range", "spec": { "start": "-4h", "stop": "now" } }, { "id": "sum", "kind": "sum" } ], "edges": [ { "parent": "select", "child": "range" }, { "parent": "range", "child": "sum" } ] } }