Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mixing metrics and logs with Grafana + Influx

343deb2fbfa0aff9fc98d9b439eb036c?s=47 David
June 13, 2019

Mixing metrics and logs with Grafana + Influx

Grafana’s new Explore area is adding support for both metric and logs display for the Influx datasource. This allows you to quickly access your metrics, and as part of troubleshooting, bring up related logs. We’ll also look at the latest support for Flux inside Grafana.

343deb2fbfa0aff9fc98d9b439eb036c?s=128

David

June 13, 2019
Tweet

Transcript

  1. Mixing metrics and logs with Grafana + Influx David Kaltschmidt

    @davkals InfluxDays June 2019
  2. I’m David Working on Explore, Prometheus, and Loki at Grafana

    Labs Previously: Unifying Metrics/Logs/Traces at Kausal, Weavescope david@grafana.com Twitter: @davkals
  3. Grafana Dashboarding solution Observability platform

  4. Unified way to look at data from different sources Logos

    of datasources
  5. Feb 2019: Major release Grafana 6 with Explore and better

    panel editing
  6. Latest release: Grafana 6.2 with Bar Gauge, and lazy loading

  7. Grafana + Influx

  8. Dashboard with Influx <= 1.7 - Graphical query builder -

    InfluxQL - Database authentication via username/password
  9. New: Explore with Influx <= 1.7 - New section for

    ad-hoc queries - Same query builder
  10. New: Explore with Influx <= 1.7 - Table view to

    look at raw values
  11. Log aggregation with Influx

  12. Troubleshooting journey

  13. Log aggregation with Influx - Central place to store your

    application logs - Revisit logs after apps/services/hosts have crashed - Basic analytics support with query language - Influx is not designed for logs, but may be “good enough” - Running a second system for logs adds operational complexity
  14. host1 InfluxDB host2 - Measurement: “logs” - Fields: - “message”

    - “details” - Tags: - “datacenter” - “host” - “app” Modelling log streams app1 app2 db1 db2 app1 dc1
  15. Getting logs into Influx (1/3): HTTP API and client libraries

  16. Getting logs into Influx (2/3): Telegraf logs input plugin

  17. Getting logs into Influx (3/3): Fluentd https://docs.fluentd.org/how-to-guides/syslog-influxdb

  18. New: Log viewer - New view in Explore (master, and

    soon 6.3) - Works with Influx 1.7 - Tag-based filtering for log streams - Will work with v2 shortly
  19. Log viewer: Ad-hoc value distributions, log levels, deduplication

  20. Explore: Split view to look at metrics and logs side-by-side

  21. Demo

  22. Grafana + Influx 2.0 and Flux

  23. Flux datasource for Influx >= 2.0 - External datasource (for

    now), latest version: 5.3.1 - Flux expression editor - Supports token authentication (Influx v2 only) - Install via Grafana CLI: grafana-cli plugins install grafana-influxdb-flux-datasource Repo for filing issues: https://github.com/grafana/influxdb-flux-datasource
  24. Try it out (1/3) $> docker run --name influxdb -p

    9999:9999 quay.io/influxdb/influxdb:2.0.0-alpha
  25. Try it out (2/3) $> grafana-cli plugins install grafana-influxdb-flux-datasource $>

    export INFLUX_TOKEN=<TOKEN> $> telegraf --config http://localhost:9999/api/v2/telegrafs/<CONFIG_ID>
  26. Try it out (3/3)

  27. Outlook - Unifying Flux and Influx datasources - Adding version

    drop-down to datasource config page - Explore and panels render editors based on version - Datasource uses appropriate API paths and authentication mechanism - Logs in Flux datasource - Improve tag filtering UX
  28. Thank you. UX feedback to david@grafana.com @davkals