InfluxDB + Prometheus

Transcript

1. InfluxDB + Prometheus Paul Dix CTO @ InfluxData @pauldix paul@influxdb.com

2. Open Source Time Series Database

3. • OSS - MIT License • Written in Go •

   SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Commercial Enterprise - HA + Scale Out Clustering

4. Data Model Measurement cpu,host=serverA,num=1,region=west idle=1.667,system=2342.2 1492214400000000000 Tags Fields nanosecond epoch

5. float64, int64, bool, string uint64 in next release

6. Example Query select percentile(90, value) from cpu where time >

   now() - 12h and “region” = ‘west’ group by time(10m), host

7. Process: • OSS - MIT License • Written in Go

   • Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InfluxDB • User Defined Functions • Service Discovery & Pull

8. Collect: • OSS - MIT License • Written in Go

   • Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InfluxDB, Graphite, Kafka, etc.

9. Why is a competitor at Promcon?

10. What is Prometheus?

11. Prometheus Server

12. Alert Manager

13. Exposition Format

14. Client Libraries • Supported • Go • Java • Python

    • Ruby • Community • C++ • C# • Node.js • PHP • etc…

15. PromQL

16. API /api/v1/series /api/v1/label/<name>/values

17. Remote Read/Write API

18. Prometheus is more than just the server

19. Influx + Prometheus today

20. We’re embracing pull & push

21. Prometheus Scrape Targets Kapacitor

22. https://github.com/prometheus/prometheus/issues/2911 https://github.com/influxdata/prometheus_metric_normalizer

23. Remote Read/Write

24. Prometheus Config # Remote write configuration (for Graphite, OpenTSDB, or

    InfluxDB). remote_write: - url: "http://localhost:9201/write" # Remote read configuration (for InfluxDB only at the moment). remote_read: - url: "http://localhost:9201/read"

25. metric -> measurement labels -> tags value -> only field

    (float64)

26. Remote Read Protobuf over HTTP

27. Reads message Query { int64 start_timestamp_ms = 1; int64 end_timestamp_ms

    = 2; repeated LabelMatcher matchers = 3; } enum MatchType { EQUAL = 0; NOT_EQUAL = 1; REGEX_MATCH = 2; REGEX_NO_MATCH = 3; } message LabelMatcher { MatchType type = 1; string name = 2; string value = 3; }

28. Remote Read JSON over HTTP

29. Unnecessary Overhead

30. Can we push down processing?

31. Influx + Prometheus future

32. First, pull remote gateway into InfluxDB!

33. Influx Data Model tags + value + time

34. Like Prometheus message Sample { double value = 1; int64

    timestamp_ms = 2; } message LabelPair { string name = 1; string value = 2; } message TimeSeries { repeated LabelPair labels = 1; // Sorted by time, oldest sample first. repeated Sample samples = 2; }

35. metric is __name__ label

36. But we’ll continue to support int64, bool, string, and uint64

37. IFQL - (Influx Functional Query Language) select(database:"foo",where:{'metric'="requests"}) .range(start:-4h) .window(period:10m) .sum()

    .interpolate(start:-4h,every:10m,value:{mean($)})

38. Decouple QL from Processing from Storage

39. None

40. DAG { "operations": [ { "id": "select", "kind": "select", "spec":

    { "database": "mydb" } }, { "id": "range", "kind": "range", "spec": { "start": "-4h", "stop": "now" } }, { "id": "sum", "kind": "sum" } ], "edges": [ { "parent": "select", "child": "range" }, { "parent": "range", "child": "sum" } ] } }

41. Pushdown Predicates

42. Opportunities to improve storage/remote?

43. Bulk Write

44. Long term storage and query for Prometheus SaaS & On-Premise

45. Thank you @pauldix paul@influxdb.com