InfluxDB 2.0 and Flux

39b7a68b6cbc43ec7683ad0bcc4c9570?s=47 Paul Dix
March 28, 2019

InfluxDB 2.0 and Flux

Presented at CloudConf 2019, this talk is an introduction to InfluxDB 2.0 and the new programming and query language, Flux (#fluxlang).

39b7a68b6cbc43ec7683ad0bcc4c9570?s=128

Paul Dix

March 28, 2019
Tweet

Transcript

  1. 5.
  2. 7.
  3. 10.

    Regular time series t0 t1 t2 t3 t4 t6 t7

    Samples at regular intervals
  4. 11.

    Irregular time series t0 t1 t2 t3 t4 t6 t7

    Events whenever they come in
  5. 29.

    2.0

  6. 30.

    • MIT Licensed • TSDB (write, query) • UI &

    Visualizations, Dashboards • Pull Metrics (Prometheus & OpenMetrics) • Tasks (background processing, ETL, monitoring/alerting)
  7. 31.
  8. 32.
  9. 33.
  10. 34.
  11. 37.

    Data Model • Organization • Dashboards • Tasks • Buckets

    • Scrapers & Telegraf configs • Labels • Users
  12. 38.
  13. 39.

    • Query planner • Query optimizer • Turing complete language,

    VM, and query engine • Multi-language support in Engine • Multi-data source support • InfluxDB, CLI, REPL, Go library
  14. 41.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
  15. 42.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Comments
  16. 43.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Named Arguments
  17. 44.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") String Literals
  18. 45.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Buckets, not DBs
  19. 46.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Duration Literal
  20. 47.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:2018-11-07T00:00:00Z) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Time Literal
  21. 48.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Pipe forward operator
  22. 49.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Anonymous Function
  23. 50.

    // get all data from the telegraf db from(bucket:”telegraf/autogen”) //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => (r._measurement == "cpu" or r._measurement == “cpu") and r.host == “serverA") Predicate Function
  24. 55.

    // variables some_int = 23 some_float = 23.2 some_string =

    “cpu" some_duration = 1h some_time = 2018-10-10T19:00:00
  25. 56.

    // variables some_int = 23 some_float = 23.2 some_string =

    “cpu" some_duration = 1h some_time = 2018-10-10T19:00:00 some_array = [1, 6, 20, 22]
  26. 57.

    // variables some_int = 23 some_float = 23.2 some_string =

    “cpu" some_duration = 1h some_time = 2018-10-10T19:00:00 some_array = [1, 6, 20, 22] some_object = {foo: "hello" bar: 22}
  27. 58.

    // defining a pipe forwardable function square = (tables=<-) =>

    tables |> map(fn: (r) => {r with _value: r._value * r._value})
  28. 59.

    // defining a pipe forwardable function square = (tables=<-) =>

    tables |> map(fn: (r) => {r with _value: r._value * r._value}) Accepts a pipe forward assigns to tables variable
  29. 60.

    // defining a pipe forwardable function square = (tables=<-) =>

    tables |> map(fn: (r) => {r with _value: r._value * r._value}) from(bucket:"foo") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "samples") |> square() |> filter(fn: (r) => r._value > 23.2)
  30. 61.

    // defining a pipe forwardable function square = (tables=<-) =>

    tables |> map(fn: (r) => {r with _value: r._value * r._value}) from(bucket:"foo") |> range(start: -1h) |> filter(fn: (r) => r._measurement == "samples") |> square() |> filter(fn: (r) => r._value > 23.2) Calling the function
  31. 66.

    Table _measurement host region _field _time _value mem A west

    free 2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10
  32. 67.

    _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Column
  33. 68.

    _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Record
  34. 69.

    _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Group Key _measurement=mem,host=A,region=west,_field=free
  35. 70.

    _measurement host region _field _time _value mem A west free

    2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 10 Every record has the same value! _measurement=mem,host=A,region=west,_field=free
  36. 71.

    Table Per Series _measurement host region _field _time _value mem

    A west free 2018-06-14T09:15:00 10 mem A west free 2018-06-14T09:14:50 11 _measurement host region _field _time _value mem B west free 2018-06-14T09:15:00 20 mem B west free 2018-06-14T09:14:50 22 _measurement host region _field _time _value cpu A west usage_user 2018-06-14T09:15:00 45 cpu A west usage_user 2018-06-14T09:14:50 49 _measurement host region _field _time _value cpu A west usage_system 2018-06-14T09:15:00 35 cpu A west usage_system 2018-06-14T09:14:50 38
  37. 73.

    input tables -> function -> output tables // example query

    from(bucket:"telegraf") |> range(start:2018-06-14T09:14:50, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  38. 74.

    input tables -> function -> output tables What to sum

    on? // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:50, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  39. 75.

    input tables -> function -> output tables Default columns argument

    // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:50, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum(columns: [“_value”])
  40. 76.

    input tables -> function -> output tables _meas ureme host

    region _field _time _valu e mem A west free 2018-06- 14T09:1 10 mem A west free 2018-06- 14T09:1 11 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 20 mem B west free 2018-06- 14T09:14 22 Input in table form // example query from(bucket:”telegraf") |> range(start:2018-06-14T09:14:50, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  41. 77.

    input tables -> function -> output tables _meas ureme host

    region _field _time _valu e mem A west free 2018-06- 14T09:1 10 mem A west free 2018-06- 14T09:1 11 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 20 mem B west free 2018-06- 14T09:14 22 sum() // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:50, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum()
  42. 78.

    input tables -> function -> output tables // example query

    from(bucket:"telegraf") |> range(start:2018-06-14T09:14:50, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> sum() _meas ureme host region _field _time _valu e mem A west free 2018-06- 14T09:1 10 mem A west free 2018-06- 14T09:1 11 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 20 mem B west free 2018-06- 14T09:14 22 sum() _meas ureme host region _field _time _valu e mem A west free 2018-06- 14T09:1 21 _meas ureme host region _field _time _valu e mem B west free 2018-06- 14T09:15 42
  43. 81.

    window // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) 30s of data (4 samples)
  44. 82.

    window // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) split into 20s windows
  45. 83.

    window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) Input
  46. 84.

    window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 window( every:20s) // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s)
  47. 85.

    window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 window( every:20s) // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) _meas ureme host region _field _time _valu e mem A west free …14:30 10 mem A west free …14:40 11 _meas ureme host region _field _time _valu e mem B west free …14:50 23 mem B west free …15:00 24 _meas ureme host region _field _time _valu e mem B west free …14:30 20 mem B west free …14:40 22 _meas ureme host region _field _time _valu e mem A west free …14:50 12 mem A west free …15:00 13
  48. 86.

    window _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 window( every:20s) // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> window(every:20s) _meas ureme host region _field _time _valu e mem A west free …14:30 10 mem A west free …14:40 11 _meas ureme host region _field _time _valu e mem B west free …14:50 23 mem B west free …15:00 24 _meas ureme host region _field _time _valu e mem B west free …14:30 20 mem B west free …14:40 22 _meas ureme host region _field _time _valu e mem A west free …14:50 12 mem A west free …15:00 13 N to M tables
  49. 88.

    group // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"])
  50. 89.

    group // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn:

    r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"]) new group key
  51. 90.

    group _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"])
  52. 91.

    group _meas host region _field _time _valu mem A west

    free …14:30 10 mem A west free …14:40 11 mem A west free …14:50 12 mem A west free …15:00 13 _meas host region _field _time _valu mem B west free …14:30 20 mem B west free …14:40 22 mem B west free …14:50 23 mem B west free …15:00 24 group( keys: [“region”]) // example query from(bucket:"telegraf") |> range(start:2018-06-14T09:14:30, stop:2018-06-14T09:15:01) |> filter(fn: r => r._measurement == “mem" and r._field == “free”) |> group(keys:[“region"]) _meas ureme host region _field _time _valu e mem A west free …14:30 10 mem B west free …14:30 20 mem A west free …14:40 11 mem B west free …14:40 21 mem A west free …14:50 12 mem B west free …14:50 22 mem B west free …15:00 13 mem B west free …15:00 23 N to M tables M == cardinality(group keys)
  53. 94.

    4GL

  54. 97.

    GUI

  55. 101.
  56. 102.
  57. 104.
  58. 106.

    from(bucket: "foo") |> range(start: -10m) |> filter(fn: (r) => r._measurement

    == "cpu") |> group(columns: ["_measurement"]) |> sort(columns: ["_value"]) Sorting by value!
  59. 110.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message)
  60. 111.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) tasks
  61. 112.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) cron scheduling
  62. 113.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) packages & imports
  63. 114.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) String interpolation
  64. 115.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) Ship data elsewhere
  65. 116.

    option task = { name: "email alert digest", cron: "0

    5 * * 0" } import "smtp" body = "" from(bucket: "alerts") |> range(start: -24h) |> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message") |> group(columns: ["alert"]) |> count() |> group() |> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} times\n") smtp.to( config: loadSecret(name: "smtp_digest"), to: "alerts@influxdata.com", title: "Alert digest for {now()}", body: message) Store secrets in a store like Vault
  66. 118.
  67. 119.

    • Finalizing Spec • Error Handling • Test Runner &

    CLI • User Packages • Flow Control (if/else) Status
  68. 120.

    Status • Alpha 7 this week • API, Tasks, Dashboards

    • Client Libraries (soon) • Monitoring & Alerting (soon)