Querying Prometheus with Flux
(#fluxlang)
Paul Dix
@pauldix
paul@influxdata.com
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
• Data-scripting language
• Functional
• MIT Licensed
• Language & Runtime/Engine
Slide 4
Slide 4 text
Prometheus users: so what?
Slide 5
Slide 5 text
High availability?
Slide 6
Slide 6 text
Sharded Data?
Slide 7
Slide 7 text
Federation?
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
subqueries
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
subqueries
recording rules
Slide 15
Slide 15 text
Ad hoc exporation
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
Focus is Strength
Slide 18
Slide 18 text
Saying No is an Asset
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
Liberate the silo!
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
Language Elements
Slide 23
Slide 23 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Slide 24
Slide 24 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Comments
Slide 25
Slide 25 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Functions
Slide 26
Slide 26 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system")
Pipe forward operator
Slide 27
Slide 27 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Named Arguments
Slide 28
Slide 28 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
String Literal
Slide 29
Slide 29 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Duration Literal (relative time)
Slide 30
Slide 30 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:”2018-08-09T14:00:00Z“)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Time Literal
Slide 31
Slide 31 text
// get all data from the telegraf db
from(db:"telegraf")
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Anonymous Function
Types
• int
• uint
• float64
• string
• duration
• time
• regex
• array
• object
• function
• namespace
• table
• table stream
Slide 34
Slide 34 text
Ways to run Flux - (interpreter,
fluxd api server, InfluxDB 1.7 & 2.0)
Slide 35
Slide 35 text
Flux builder in Chronograf
Slide 36
Slide 36 text
Flux builder in Grafana
Slide 37
Slide 37 text
Flux is about:
Slide 38
Slide 38 text
Time series in Prometheus
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
// get data from Prometheus on http://localhost:9090
fromProm(query:`node_cpu_seconds_total{cpu=“0”,mode=“idle”}`)
// filter that by the last minute
|> range(start:-1m)
Nested range vectors
fromProm(host:”http://localhost:9090")
|> filter(fn: (r) => r.__name__ == "node_disk_written_bytes_total")
|> range(start:-1h)
// transform into non-negative derivative values
|> derivative()
// break those out into tables for each 10 minute block of time
|> window(every:10m)
// get the max rate of change in each 10 minute window
|> max()
// and put everything back into a single table
|> window(every:inf)
// and now let’s convert to KB
|> map(fn: (r) => r._value / 1024.0)
Project Status
• Everything in this talk is prototype (as of 2018-08-09)
• Proposed Final Language Spec
• Release flux, fluxd, InfluxDB 1.7, InfluxDB 2.0 alpha
• Iterate with community to finalize spec
• Optimizations!
• https://github.com/influxdata/flux
Slide 100
Slide 100 text
Future work
Slide 101
Slide 101 text
More complex Flux
compilations to PromQL?
Slide 102
Slide 102 text
PromQL parser for Flux
engine?
Slide 103
Slide 103 text
Add Flux into Prometheus?
Slide 104
Slide 104 text
Arrow API for Prometheus
Slide 105
Slide 105 text
Apache Arrow
Slide 106
Slide 106 text
Stream from Prometheus
Slide 107
Slide 107 text
Pushdown matcher and range
Slide 108
Slide 108 text
Later pushdown more?
Slide 109
Slide 109 text
Standardized Remote Read
API?
Slide 110
Slide 110 text
Arrow is becoming the lingua
franca in data science and big data