Querying Prometheus with Flux

39b7a68b6cbc43ec7683ad0bcc4c9570?s=47 Paul Dix
August 09, 2018

Querying Prometheus with Flux

Talk given at PromCon 2018 where I introduce Flux (#fluxlang) and show how it can be used to query Prometheus servers.

39b7a68b6cbc43ec7683ad0bcc4c9570?s=128

Paul Dix

August 09, 2018
Tweet

Transcript

  1. 2.
  2. 8.
  3. 9.
  4. 10.
  5. 11.
  6. 13.
  7. 16.
  8. 19.
  9. 21.
  10. 23.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
  11. 24.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Comments
  12. 25.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Functions
  13. 26.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system") Pipe forward operator
  14. 27.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Named Arguments
  15. 28.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") String Literal
  16. 29.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Duration Literal (relative time)
  17. 30.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:”2018-08-09T14:00:00Z“) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Time Literal
  18. 31.

    // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Anonymous Function
  19. 32.

    Operators + == != ( ) - < !~ [

    ] * > =~ { } / <= = , : % >= <- . |>
  20. 33.

    Types • int • uint • float64 • string •

    duration • time • regex • array • object • function • namespace • table • table stream
  21. 39.
  22. 40.
  23. 42.
  24. 43.
  25. 44.
  26. 45.
  27. 46.
  28. 47.
  29. 48.
  30. 51.
  31. 52.
  32. 53.
  33. 54.
  34. 55.
  35. 59.

    fromProm() |> range(start:-1m) |> filter(fn: (r) => r.__name__ == “node_cpu_seconds_total”

    and r.mode == “idle” and r.cpu == “0”) |> keep(columns: [“name”, “cpu”, “host”, “mode”, “_value”, “_time”])
  36. 60.
  37. 61.
  38. 62.
  39. 63.
  40. 64.

    fromProm() |> range(start:-1m) |> filter(fn: (r) => r.__name__ == “node_cpu_seconds_total”

    and r.mode in [“idle”, “user”] and r.cpu == “0”) |> keep(columns: [“name”, “cpu”, “host”, “mode”, “_value”, “_time”])
  41. 65.
  42. 66.
  43. 67.
  44. 69.

    fromProm() |> range(start:-30s) |> filter(fn: (r) => r.__name__ == “node_cpu_seconds_total”

    and r.mode == “idle” and r.cpu =~ /0|1/) |> count() |> keep(columns: [“name”, “cpu”, “host”, “mode”, “_value”, “_time”])
  45. 70.
  46. 71.
  47. 72.
  48. 73.
  49. 74.
  50. 75.
  51. 78.
  52. 80.
  53. 82.
  54. 84.
  55. 88.
  56. 90.
  57. 91.
  58. 92.
  59. 93.

    Nested range vectors fromProm(host:”http://localhost:9090") |> filter(fn: (r) => r.__name__ ==

    "node_disk_written_bytes_total") |> range(start:-1h) // transform into non-negative derivative values |> derivative() // break those out into tables for each 10 minute block of time |> window(every:10m) // get the max rate of change in each 10 minute window |> max() // and put everything back into a single table |> window(every:inf) // and now let’s convert to KB |> map(fn: (r) => r._value / 1024.0)
  60. 94.

    Multiple Servers dc1 = fromProm(host:”http://prom.dc1.local:9090") |> filter(fn: (r) => r.__name__

    == “node_network_receive_bytes_total”) |> range(start:-1h) |> insertGroupKey(key: “dc”, value: “1”) dc2 = fromProm(host:”http://prom.dc2.local:9090") |> filter(fn: (r) => r.__name__ == “node_network_receive_bytes_total”) |> range(start:-1h) |> insertGroupKey(key: “dc”, value: “2”) dc1 |> union(streams: [dc2]) |> limit(n: 2) |> derivative() |> group(columns: [“dc”]) |> sum()
  61. 95.

    Work with data from many sources • from() // influx

    • fromProm() • fromMySQL() • fromCSV() • fromS3() • …
  62. 97.

    Defining Functions windowAgg = (every, fn, <-stream) => { return

    stream |> window(every: every) |> fn() |> window(every:inf) } fromProm(query: `node_cpu_seconds_total{cpu=“0”,mode=“idle”}` |> range(start: -1m) |> windowAgg(every:20s, fn: min)
  63. 98.

    Packages & Namespaces package “flux-helpers” windowAgg = (every, fn, <-stream)

    => { return stream |> window(every: every) |> fn() |> window(every:inf) } // in a new script import helpers “github.com/pauldix/flux-helpers" fromProm(query: `node_cpu_seconds_total{cpu=“0”,mode=“idle”}` |> range(start: -1m) |> helpers.windowAgg(every:20s, fn: min)
  64. 99.

    Project Status • Everything in this talk is prototype (as

    of 2018-08-09) • Proposed Final Language Spec • Release flux, fluxd, InfluxDB 1.7, InfluxDB 2.0 alpha • Iterate with community to finalize spec • Optimizations! • https://github.com/influxdata/flux