InfluxDB + Prometheus

Slides from my Promcon2017 talk on integrating InfluxDB and Prometheus.

Transcript

1. InfluxDB + Prometheus Paul Dix CTO @ InfluxData @pauldix paul@influxdb.com
2. Open Source Time Series Database
3. • OSS - MIT License • Written in Go • SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Commercial Enterprise - HA + Scale Out Clustering
4. Data Model Measurement cpu,host=serverA,num=1,region=west idle=1.667,system=2342.2 1492214400000000000 Tags Fields nanosecond epoch
5. float64, int64, bool, string uint64 in next release
6. Example Query select percentile(90, value) from cpu where time > now() - 12h and “region” = ‘west’ group by time(10m), host
7. Process: • OSS - MIT License • Written in Go • Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InfluxDB • User Defined Functions • Service Discovery & Pull
8. Collect: • OSS - MIT License • Written in Go • Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InfluxDB, Graphite, Kafka, etc.
9. Why is a competitor at Promcon?
10. What is Prometheus?
11. Prometheus Server
12. Alert Manager
13. Exposition Format
14. Client Libraries • Supported • Go • Java • Python • Ruby • Community • C++ • C# • Node.js • PHP • etc…
15. PromQL
16. API /api/v1/series /api/v1/label/<name>/values
17. Remote Read/Write API
18. Prometheus is more than just the server
19. Influx + Prometheus today
20. We’re embracing pull & push
21. Prometheus Scrape Targets Kapacitor
22. https://github.com/prometheus/prometheus/issues/2911 https://github.com/influxdata/prometheus_metric_normalizer
23. Remote Read/Write
24. Prometheus Config # Remote write configuration (for Graphite, OpenTSDB, or InfluxDB). remote_write: - url: "http://localhost:9201/write" # Remote read configuration (for InfluxDB only at the moment). remote_read: - url: "http://localhost:9201/read"
25. metric -> measurement labels -> tags value -> only field (float64)
26. Remote Read Protobuf over HTTP
27. Reads message Query { int64 start_timestamp_ms = 1; int64 end_timestamp_ms = 2; repeated LabelMatcher matchers = 3; } enum MatchType { EQUAL = 0; NOT_EQUAL = 1; REGEX_MATCH = 2; REGEX_NO_MATCH = 3; } message LabelMatcher { MatchType type = 1; string name = 2; string value = 3; }
28. Remote Read JSON over HTTP
29. Unnecessary Overhead
30. Can we push down processing?
31. Influx + Prometheus future
32. First, pull remote gateway into InfluxDB!
33. Influx Data Model tags + value + time
34. Like Prometheus message Sample { double value = 1; int64 timestamp_ms = 2; } message LabelPair { string name = 1; string value = 2; } message TimeSeries { repeated LabelPair labels = 1; // Sorted by time, oldest sample first. repeated Sample samples = 2; }
35. metric is __name__ label
36. But we’ll continue to support int64, bool, string, and uint64
37. IFQL - (Influx Functional Query Language) select(database:"foo",where:{'metric'="requests"}) .range(start:-4h) .window(period:10m) .sum() .interpolate(start:-4h,every:10m,value:{mean($)})
38. Decouple QL from Processing from Storage
40. DAG { "operations": [ { "id": "select", "kind": "select", "spec": { "database": "mydb" } }, { "id": "range", "kind": "range", "spec": { "start": "-4h", "stop": "now" } }, { "id": "sum", "kind": "sum" } ], "edges": [ { "parent": "select", "child": "range" }, { "parent": "range", "child": "sum" } ] } }
41. Pushdown Predicates
42. Opportunities to improve storage/remote?
43. Bulk Write
44. Long term storage and query for Prometheus SaaS & On-Premise
45. Thank you @pauldix paul@influxdb.com