Mixing metrics and logs with Grafana + Influx

Slide 1

Slide 1 text

Mixing metrics and logs with Grafana + Inﬂux David Kaltschmidt @davkals InﬂuxDays June 2019

Slide 2

Slide 2 text

I’m David Working on Explore, Prometheus, and Loki at Grafana Labs Previously: Unifying Metrics/Logs/Traces at Kausal, Weavescope [email protected] Twitter: @davkals

Slide 3

Slide 3 text

Grafana Dashboarding solution Observability platform

Slide 4

Slide 4 text

Uniﬁed way to look at data from diﬀerent sources Logos of datasources

Slide 5

Slide 5 text

Feb 2019: Major release Grafana 6 with Explore and better panel editing

Slide 6

Slide 6 text

Latest release: Grafana 6.2 with Bar Gauge, and lazy loading

Slide 7

Slide 7 text

Grafana + Inﬂux

Slide 8

Slide 8 text

Dashboard with Inﬂux <= 1.7 - Graphical query builder - InﬂuxQL - Database authentication via username/password

Slide 9

Slide 9 text

New: Explore with Inﬂux <= 1.7 - New section for ad-hoc queries - Same query builder

Slide 10

Slide 10 text

New: Explore with Inﬂux <= 1.7 - Table view to look at raw values

Slide 11

Slide 11 text

Log aggregation with Inﬂux

Slide 12

Slide 12 text

Troubleshooting journey

Slide 13

Slide 13 text

Log aggregation with Inﬂux - Central place to store your application logs - Revisit logs after apps/services/hosts have crashed - Basic analytics support with query language - Inﬂux is not designed for logs, but may be “good enough” - Running a second system for logs adds operational complexity

Slide 14

Slide 14 text

host1 InﬂuxDB host2 - Measurement: “logs” - Fields: - “message” - “details” - Tags: - “datacenter” - “host” - “app” Modelling log streams app1 app2 db1 db2 app1 dc1

Slide 15

Slide 15 text

Getting logs into Inﬂux (1/3): HTTP API and client libraries

Slide 16

Slide 16 text

Getting logs into Inﬂux (2/3): Telegraf logs input plugin

Slide 17

Slide 17 text

Getting logs into Inﬂux (3/3): Fluentd https://docs.fluentd.org/how-to-guides/syslog-influxdb

Slide 18

Slide 18 text

New: Log viewer - New view in Explore (master, and soon 6.3) - Works with Inﬂux 1.7 - Tag-based ﬁltering for log streams - Will work with v2 shortly

Slide 19

Slide 19 text

Log viewer: Ad-hoc value distributions, log levels, deduplication

Slide 20

Slide 20 text

Explore: Split view to look at metrics and logs side-by-side

Slide 21

Slide 21 text

Demo

Slide 22

Slide 22 text

Grafana + Inﬂux 2.0 and Flux

Slide 23

Slide 23 text

Flux datasource for Influx >= 2.0 - External datasource (for now), latest version: 5.3.1 - Flux expression editor - Supports token authentication (Influx v2 only) - Install via Grafana CLI: grafana-cli plugins install grafana-influxdb-flux-datasource Repo for filing issues: https://github.com/grafana/influxdb-flux-datasource

Slide 24

Slide 24 text

Try it out (1/3) $> docker run --name influxdb -p 9999:9999 quay.io/influxdb/influxdb:2.0.0-alpha

Slide 25

Slide 25 text

Try it out (2/3) $> grafana-cli plugins install grafana-influxdb-flux-datasource $> export INFLUX_TOKEN= $> telegraf --config http://localhost:9999/api/v2/telegrafs/

Slide 26

Slide 26 text

Try it out (3/3)

Slide 27

Slide 27 text

Outlook - Unifying Flux and Influx datasources - Adding version drop-down to datasource config page - Explore and panels render editors based on version - Datasource uses appropriate API paths and authentication mechanism - Logs in Flux datasource - Improve tag filtering UX

Slide 28

Slide 28 text

Thank you. UX feedback to [email protected] @davkals