Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flux (#fluxlang): a new (time series) data scripting language

Paul Dix
June 14, 2018

Flux (#fluxlang): a new (time series) data scripting language

Talk from InfluxDays London on Flux, the new scripting language being built by InfluxData.

Paul Dix

June 14, 2018
Tweet

More Decks by Paul Dix

Other Decks in Technology

Transcript

  1. QUEL & POSTGRESQUEL range of E is EMPLOYEE retrieve into

    W (COMP = E.Salary / (E.Age - 18)) where E.Name = "Jones" select (e.salary / (e.age - 18)) as comp from employee as e where e.name = "Jones" SQL
  2. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system")
  3. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system") Comments
  4. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system") Duration Literal
  5. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system") Pipe forward operator
  6. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system") Anonymous Function
  7. Operators + == != ( ) - < !~ [

    ] * > =~ { } / <= = , : % >= <- . |>
  8. Types • int • uint • float64 • string •

    duration • time • regex • array • object • function • namespace
  9. Functions • count • covariance • cumulativeSum • derivative •

    difference • distinct • filter • first • from • group • integral • mean • min • percentile • range • sample • set • shift • skew • sort • spread • stateTracking • limit • map • max • window • yield • cov • highestMax • highestAverage • highestCurrent • lowestMin • join • last • stddev • sum • lowestAverage • lowestCurrent • pearsonR • stateCount • stateDuration • top • bottom
  10. Table _measurement host region _field _time _value mem A west

    free 2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10
  11. _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Column
  12. _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Record
  13. _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Group Key _measurement=mem,host=A,region=west,_field=free
  14. _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Every record has the same value! _measurement=mem,host=A,region=west,_field=free
  15. Table Per Series _measurement host region _field _time _value mem

    A west free 2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 11 _measurement host region _field _time _value mem B west free 2018-06-14T09:15:00 20 mem B west free 2018-06-14T09:14:50 22 _measurement host region _field _time _value cpu A west usage_user 2018-06-14T09:15:00 45 cpu A west usage_user 2018-06-14T09:14:50 49 _measurement host region _field _time _value cpu A west usage_system 2018-06-14T09:15:00 35 cpu A west usage_system 2018-06-14T09:14:50 38
  16. input tables -> function -> output tables // example query

    from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  17. input tables -> function -> output tables DateTime Literal //

    example query from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  18. input tables -> function -> output tables What to sum

    on? // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  19. input tables -> function -> output tables Default columns argument

    // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum(columns: [“_value”])
  20. input tables -> function -> output tables _meas ureme host

    region _field _time _valu e mem A west free 2018-06- 14T09:1 10 mem A west free 2018-06- 14T09:1 11 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 20 mem B west free 2018-06- 14T09:14 22 Input in table form // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  21. input tables -> function -> output tables _meas ureme host

    region _field _time _valu e mem A west free 2018-06- 14T09:1 10 mem A west free 2018-06- 14T09:1 11 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 20 mem B west free 2018-06- 14T09:14 22 sum() // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  22. input tables -> function -> output tables // example query

    from(db:"telegraf") |> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum() _meas ureme host region _field _time _valu e mem A west free 2018-06- 14T09:1 10 mem A west free 2018-06- 14T09:1 11 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 20 mem B west free 2018-06- 14T09:14 22 sum() _meas ureme host region _field _time _valu e mem A west free 2018-06- 14T09:1 21 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 42
  23. window // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) 30s of data (4 samples)
  24. window // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) split into 20s windows
  25. window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) Input
  26. window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 window( every:20s) // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s)
  27. window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 window( every:20s) // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) _meas ureme host region _field _time _valu e mem A west free …14:30 10 mem A west free …14:40 11 _meas ureme host region _field _time _valu e mem B west free …14:50 23 mem B west free …15:00 24 _meas ureme host region _field _time _valu e mem B west free …14:30 20 mem B west free …14:40 22 _meas ureme host region _field _time _valu e mem A west free …14:50 12 mem A west free …15:00 13
  28. window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 window( every:20s) // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) _meas ureme host region _field _time _valu e mem A west free …14:30 10 mem A west free …14:40 11 _meas ureme host region _field _time _valu e mem B west free …14:50 23 mem B west free …15:00 24 _meas ureme host region _field _time _valu e mem B west free …14:30 20 mem B west free …14:40 22 _meas ureme host region _field _time _valu e mem A west free …14:50 12 mem A west free …15:00 13 N to M tables
  29. group // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"])
  30. group // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"]) new partition key
  31. group _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"])
  32. group _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 group( keys: [“region”]) // example query from(db:"telegraf") |> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"]) _meas ureme host region _field _time _valu e mem A west free …14:30 10 mem B west free …14:30 20 mem A west free …14:40 11 mem B west free …14:40 21 mem A west free …14:50 12 mem B west free …14:50 22 mem B west free …15:00 13 mem B west free …15:00 23 N to M tables M == cardinality(group keys)
  33. showTagValues = (db, tag, start=-1h, stop=now(), predicate=(r) => true) =>

    from(db:db) |> range(start:start, stop:stop) |> filter(fn: predicate) |> group(by:[tag]) // get the distinct values for the tag |> distinct(column:tag) // collapse all tables into one |> group(none:true) // drop all columns except _value |> keep(columns: ["_value"])
  34. showTagValues = (db, tag, start=-1h, stop=now(), predicate=(r) => true) =>

    from(db:db) |> range(start:start, stop:stop) |> filter(fn: predicate) |> group(by:[tag]) // get the distinct values for the tag |> distinct(column:tag) // collapse all tables into one |> group(none:true) // drop all columns except _value |> keep(columns: ["_value"]) Assign function to variable
  35. showTagValues = (db, tag, start=-1h, stop=now(), predicate=(r) => true) =>

    from(db:db) |> range(start:start, stop:stop) |> filter(fn: predicate) |> group(by:[tag]) // get the distinct values for the tag |> distinct(column:tag) // collapse all tables into one |> group(none:true) // drop all columns except _value |> keep(columns: ["_value"]) Specify default argument value to make optional
  36. showTagValues = (db, tag, start=-1h, stop=now(), predicate=(r) => true) =>

    from(db:db) |> range(start:start, stop:stop) |> filter(fn: predicate) |> group(by:[tag]) // get the distinct values for the tag |> distinct(column:tag) // collapse all tables into one |> group(none:true) // drop all columns except _value |> keep(columns: ["_value"]) now function
  37. showTagValues = (db, tag, start=-1h, stop=now(), predicate=(r) => true) =>

    from(db:db) |> range(start:start, stop:stop) |> filter(fn: predicate) |> group(by:[tag]) // get the distinct values for the tag |> distinct(column:tag) // collapse all tables into one |> group(none:true) // drop all columns except _value |> keep(columns: ["_value"]) pass function as argument
  38. showTagValues = (db, tag, start=-1h, stop=now(), predicate=(r) => true) =>

    from(db:db) |> range(start:start, stop:stop) |> filter(fn: predicate) |> group(by:[tag]) // get the distinct values for the tag |> distinct(column:tag) // collapse all tables into one |> group(none:true) // drop all columns except _value |> keep(columns: ["_value"])
  39. Defining functions that take inputs // convert all values into

    floats from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> map(fn: (r) => float(v:r._value))
  40. Defining functions that take inputs // convert all values into

    floats from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> map(fn: (r) => float(v:r._value)) map function
  41. Defining functions that take inputs // convert all values into

    floats from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> map(fn: (r) => float(v:r._value)) float function
  42. Defining functions that take inputs // convert all values into

    floats from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> map(fn: (r) => float(v:r._value)) only named arguments!
  43. Defining functions that take inputs // convert all values into

    floats from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> map(fn: (r) => float(v:r._value)) make this a function?
  44. Defining functions that take inputs castToFloat = (table=<-) { return

    table |> map(fn: (r) => float(v:r._value)) } user defined pipe forwardable function
  45. Defining functions that take inputs // calling it from(db:"telegraf") |>

    range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> castToFloat()
  46. Defining functions that take inputs // convert all values into

    floats from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> map(fn: (r) => float(v:r._value)) castToFloat = (table=<-) { return table |> map(fn: (r) => float(v:r._value)) } from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) => r._measurement == "foo") |> castToFloat()
  47. Any pipe forward function can use arguments min(table: range(start: -1h,

    table: filter(fn: (r) => r.host == "server0", table: from(db: "mydb"))))
  48. Math across measurements foo = from(db: "mydb") |> filter(fn: (r)

    => r._measurement == "foo") |> range(start: -1h) bar = from(db: "mydb") |> filter(fn: (r) => r._measurement == "bar") |> range(start: -1h) join( tables: {foo:foo, bar:bar}, on: [“foobar”, “_time”], fn: (t) => t.foo._value + t.bar._value, ) |> yield(name: "foobar")
  49. Having Query from(db:"mydb") |> filter(fn: (r) => r._measurement == "cpu"

    and r._field == “usage_system”) |> range(start:-1h) |> window(every:10m) |> mean() // this is the having part |> filter(fn: (r) => r._value > 90)
  50. Imports and Namespaces import "math" from(db:"mydb") |> filter(fn: (r) =>

    r.host == "server0") |> range(start: -1h) |> math.square()
  51. Imports and Namespaces import "math" from(db:"mydb") |> filter(fn: (r) =>

    r.host == "server0") |> range(start: -1h) |> math.square() namespace
  52. Imports and Namespaces import “pauldix/math" from(db:"mydb") |> filter(fn: (r) =>

    r.host == "server0") |> range(start: -1h) |> math.square() Username like RubyGems
  53. Imports and Namespaces import “github.com/pauldix/math” from(db:"mydb") |> filter(fn: (r) =>

    r.host == "server0") |> range(start: -1h) |> math.square() Or from Github
  54. Exponential Moving Average from(db:"telegraf") |> range(start:-1h) |> filter(fn: (r) =>

    r._measurement == "foo") |> exponentialMovingAverage(size:-10s)
  55. SQL rolling average select id, temp, avg(temp) over (partition by

    group_nr order by time_read) as rolling_avg from ( select id, temp, time_read, interval_group, id - row_number() over (partition by interval_group order by time_read) as group_nr from ( select id, time_read, 'epoch'::timestamp + '900 seconds'::interval * (extract(epoch from time_read)::int4 / 900) as interval_group, temp from readings ) t1 ) t2 order by time_read;
  56. Alerting import “alert” from(db:"telegraf") |> range(start: -1m) |> filter(fn: (r)

    => r._measurement == "work_queue" and r._field == "depth") |> mean() |> alert.track( warn: (r) => r._value > 200, crit: (r) => r._value > 500) |> alert.limit(duration:1m) |> toSlack(config: loadConfig(key: "slack"))