Time series & monitoring with InfluxDB and the TICK stack

Slide 1

Slide 1 text

Time series & monitoring with InﬂuxDB & the TICK stack Paul Dix paul@inﬂuxdb.com @pauldix

Slide 2

Slide 2 text

Modern engine for metrics & events (time series data)

Slide 3

Slide 3 text

What is time series data?

Slide 4

Slide 4 text

Stock trades and quotes

Slide 5

Slide 5 text

Metrics

Slide 6

Slide 6 text

Analytics

Slide 7

Slide 7 text

Events

Slide 8

Slide 8 text

Sensor data

Slide 9

Slide 9 text

Two kinds of time series data…

Slide 10

Slide 10 text

Regular time series t0 t1 t2 t3 t4 t6 t7 Samples at regular intervals

Slide 11

Slide 11 text

Irregular time series t0 t1 t2 t3 t4 t6 t7 Events whenever they come in

Slide 12

Slide 12 text

Working with Time Series collect, store, visualize, process

Slide 13

Slide 13 text

Products Built on the Open-Source “TICK stack”

Slide 14

Slide 14 text

Store: • OSS - MIT License • Written in Go • SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Retention Policies • Continuous Queries

Slide 15

Slide 15 text

API - 2 endpoints

Slide 16

Slide 16 text

API - 2 endpoints /write?db=mydb&rp=foo Write: HTTP POST

Slide 17

Slide 17 text

API - 2 endpoints /write?db=mydb&rp=foo /query?db=mydb&rp=foo Write: HTTP POST Read: HTTP GET

Slide 18

Slide 18 text

Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY:

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 some_measurement value=23422.0,context=“asdf jkl” 1422568543702900257 BODY: Unlimited Fields

Slide 25

Slide 25 text

Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 some_measurement value=23422 1422568543702900257 BODY: Fields (types: int64, ﬂoat64, string, bool)

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Inﬂux CLI

Slide 30

Slide 30 text

Create a database CREATE DATABASE foo

Slide 31

Slide 31 text

Create a retention policy CREATE RETENTION POLICY ON DURATION REPLICATION [DEFAULT]

Slide 32

Slide 32 text

Create a retention policy CREATE RETENTION POLICY ON DURATION REPLICATION [DEFAULT] CREATE RETENTION POLICY high_precision ON mydb DURATION 7d REPLICATION 3 DEFAULT

Slide 33

Slide 33 text

Create a retention policy CREATE RETENTION POLICY ON DURATION REPLICATION [DEFAULT] CREATE RETENTION POLICY high_precision ON mydb DURATION 7d REPLICATION 3 DEFAULT Writes will go into this RP unless otherwise speciﬁed

Slide 34

Slide 34 text

Discovery

Slide 35

Slide 35 text

Inverted index of measurements and tags

Slide 36

Slide 36 text

Discovery SHOW MEASUREMENTs

Slide 37

Slide 37 text

Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA'

Slide 38

Slide 38 text

Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS

Slide 39

Slide 39 text

Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU

Slide 40

Slide 40 text

Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU SHOW TAG VALUES from CPU WITH KEY = 'region'

Slide 41

Slide 41 text

Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU SHOW TAG VALUES from CPU WITH KEY = 'region' SHOW SERIES

Slide 42

Slide 42 text

Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU SHOW TAG VALUES from CPU WITH KEY = 'region' SHOW SERIES SHOW SERIES where service = 'redis'

Slide 43

Slide 43 text

Queries

Slide 44

Slide 44 text

SQL-ish select * from some_series where time > now() - 1h

Slide 45

Slide 45 text

Aggregates select percentile(90, value) from cpu where time > now() - 1d group by time(10m)

Slide 46

Slide 46 text

Aggregates select percentile(90, value) from cpu where time > now() - 1d group by time(10m), region Group by a tag

Slide 47

Slide 47 text

Where against Regex (ﬁeld) select value from some_log_series where value =~ /.*ERROR.*/ and time > "2014-03-01" and time < "2014-03-03"

Slide 48

Slide 48 text

Where against Regex (tag) select value from some_log_series where host =~ /.*asdf.*/ and time > "2014-03-01" and time < “2014-03-03" group by host

Slide 49

Slide 49 text

Functions min max percentile ﬁrst last stddev mean count sum median distinct count(distinct) difference moving_average

Slide 50

Slide 50 text

Continuous queries CREATE CONTINUOUS QUERY "10m_event_count" ON mydb BEGIN SELECT count(value) INTO "6_months".events FROM events GROUP BY time(10m) END;

Slide 51

Slide 51 text

Collect: • OSS - MIT License • Written in Go • Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InﬂuxDB, Graphite, Kafka, etc.

Slide 52

Slide 52 text

Visualize: • OSS - AGPL License • Written in Go, React, dygraph • UI for administering TICK stack • Ad-hoc exploration and visualization • Create monitoring and alerting rules in Kapacitor • Query builder, TICK script editor, and more

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Process: • OSS - MIT License • Written in Go • Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InﬂuxDB • User Deﬁned Functions • Service Discovery & Pull

Slide 58

Slide 58 text

Thank you. Paul Dix paul@inﬂuxdb.com @pauldix