A TSDB queries tour (DevOps Italia - Bologne)

@IncontroDevOps @AurrelH95 A TSDB queries tour with a monitoring zoom

@IncontroDevOps @AurrelH95 Aurélien Hébert @AurrelH95 Software Engineer and data lover
2

@IncontroDevOps @AurrelH95 A connected world 3

@IncontroDevOps @AurrelH95 Producing more daily data 4

@IncontroDevOps @AurrelH95 Human data classiﬁcation ❖ Relational ID Name Country
Job 1 Peter Ireland Bookkeeper 2 Paolo Italy Sales 5

@IncontroDevOps @AurrelH95 And ❖ Key/Value ❖ Document ❖ Graphs ❖
... Key Value edge Node Node 6

@IncontroDevOps @AurrelH95 Server and application data 7

@IncontroDevOps @AurrelH95 Metrics A series of data point indexed by
time 8

@IncontroDevOps @AurrelH95 Time series are well known Stock market Analytics
Economic Forecasting 9

@IncontroDevOps @AurrelH95 Time series Database 10

@IncontroDevOps @AurrelH95 Many open source out there ❖ Steven Acreman
(Outlier) ❖ Top 10 Time Series Databases 11

@IncontroDevOps @AurrelH95 A monitoring use case 12

@IncontroDevOps @AurrelH95 Server data CPU disk disk I/O load kernel
memory network I/O temperature system swap ... 13

@IncontroDevOps @AurrelH95 What can we do ? 14

@IncontroDevOps @AurrelH95 1. Human data viz ❖ Raw ❖ Sampling
❖ Grouping 15

@IncontroDevOps @AurrelH95 2. Data analysis ❖ Metrics functions ❖ Operation
across metrics 16

@IncontroDevOps @AurrelH95 3. More complex analytics 17

@IncontroDevOps @AurrelH95 An example is worth 1000 words 18

@IncontroDevOps @AurrelH95 Using a server data subset Memory available CPU
usage Disks I/O 19

@IncontroDevOps @AurrelH95 Data collection Agent 20

@IncontroDevOps @AurrelH95 Memory raw data Name Meta List <time, value>
21

@IncontroDevOps @AurrelH95 Focus on a TSDB subset grafana 22

@IncontroDevOps @AurrelH95 1. Human data viz 23

@IncontroDevOps @AurrelH95 OpenTSDB api/query { "start":1535752800000, "end":1535839199999, "queries": [ {
"metric":"mem.available", "aggregator":"none" } ] } 24

@IncontroDevOps @AurrelH95 PromQL api/v1/query_range? query=mem.available& start=1535797890& end=1535818770& step=30 25

@IncontroDevOps @AurrelH95 Graphite /render? target=mem.available& from=1535797842& until=1535818822& 26

@IncontroDevOps @AurrelH95 CPU’s monitoring 27

@IncontroDevOps @AurrelH95 Reduce data point per series ❖ Keep only
one point every 2 minutes 28

@IncontroDevOps @AurrelH95 Sampling 29

@IncontroDevOps @AurrelH95 OpenTSDB "queries": [{ "metric":"cpu.usage_system", "aggregator":"sum", "downsample":"2m-avg", "tags": {
"cpu":"*" } }] 32

@IncontroDevOps @AurrelH95 OpenTSDB ➢ Main down-samplers are: avg, count, dev,
ﬁrst, last, percentiles, min, max and sum 33

@IncontroDevOps @AurrelH95 PromQL api/v1/query_range? query=cpu.usage_system{ cpu=~"cpu[0-7]*"}& start=1535797890& end=1535818770& step=2m 34

@IncontroDevOps @AurrelH95 Only last down-sampler PromQL 35

@IncontroDevOps @AurrelH95 Graphite At conﬁguration, using aggregation-rules cpu.usage_system (120) =
avg cpu.usage_system Main down-samplers are: sum, avg, min, max, percentiles and count 36

@IncontroDevOps @AurrelH95 Reduce CPU series 37

@IncontroDevOps @AurrelH95 Group CPU data ❖ Sampling synchronised timestamps ➢
Compute max aggregation 38

@IncontroDevOps @AurrelH95 OpenTSDB "queries": [{ "metric":"cpu.usage_system", "aggregator":"max", "downsample":"2m-avg", "filters":[{ "type":"regexp",
"tagk":"cpu", "filter":"cpu[0-9]+", "groupBy":false }] }] 39

@IncontroDevOps @AurrelH95 OpenTSDB ➢ Main aggregators are: avg, count, dev,
percentiles, min, max, mimmin, mimmax, sum, none (raw data) and zimsum (Difference between mimmin and min are missing values interpolation, same for mimmax and max and zimsum and sum) 40

@IncontroDevOps @AurrelH95 PromQL api/v1/query_range? query=max(cpu. usage_system{ cpu=~"cpu[0-7]*"}) start=1535797890& end=1535818770& step=2m
41

@IncontroDevOps @AurrelH95 PromQL ➢ Grouping operator can be one of:
sum, avg, min, max, stddev, stdvar, count, topk, bottomk and quantile 42

@IncontroDevOps @AurrelH95 Graphite /render? target=aggregate(cpu. usage_system,'max')& from=1535797842& until=1535818822& 43

@IncontroDevOps @AurrelH95 Graphite ➢ Main aggregators are avg, median, sum,
min, max, diff, stddev, count, range, multiply and last 44

@IncontroDevOps @AurrelH95 Be able to see data 45

@IncontroDevOps @AurrelH95 Disk I/O’s monitoring 46

@IncontroDevOps @AurrelH95 2. Data analysis 47

@IncontroDevOps @AurrelH95 Compute a rate From bytes to bytes per
seconds 48

@IncontroDevOps @AurrelH95 OpenTSDB "queries": [{ "metric":"diskio.writes", "aggregator":"sum", "downsample":"2m-avg", "rateOptions": {
"counter":true, "dropResets":true }, "tags": { "name":"*" } }] 49

@IncontroDevOps @AurrelH95 OpenTSDB functions Only rate operation 50

@IncontroDevOps @AurrelH95 PromQL api/v1/query_range? query=rate(diskio.wri tes[2m])& start=1535797890& end=1535818770& step=2m 51

@IncontroDevOps @AurrelH95 PromQL functions Around 50 functions Mean_over_time, max_over_time Delta,
rate, sqrt Topk, sort 52

@IncontroDevOps @AurrelH95 Graphite /render? target=divideSeries( derivative(diskio.writes), 60)& from=1535797842& until=1535818822& 53

@IncontroDevOps @AurrelH95 Graphite functions TimeSlice, TimeShift Integral, Interpolate, derivative More
than 100 functions Unique, sort LinearRegression, exponential smoothing PieAverage, legendValue MovingMean, MovingMax 54

@IncontroDevOps @AurrelH95 Disk I/O writes times 55

@IncontroDevOps @AurrelH95 Series operators Disk I/O time Disk I/O writes
56

@IncontroDevOps @AurrelH95 Series operators Prometheus: rate(diskio.write_time[2m]) / on(name) rate(diskio.writes[2m]) Graphite:
divideSeriesList(derivative( diskio.write_time), derivative(diskio.writes)) 57

@IncontroDevOps @AurrelH95 Graphite and promQL review 58

@IncontroDevOps @AurrelH95 Common usage ❖ Succinct time series queries ❖
Same functionality ❖ Analytics 59

@IncontroDevOps @AurrelH95 Data model structure ❖ Labels with Key/Value map
attached to each metrics with Prometheus ❖ Name with dot separated component for Graphite 60

@IncontroDevOps @AurrelH95 Languages review PromQL: ❖ Structured ❖ Easier to
compute operation on multiple series Graphite: ❖ More Time series functions ➢ stats ➢ maths ➢ graphs 61

@IncontroDevOps @AurrelH95 3. Complex analytics 62

@IncontroDevOps @AurrelH95 Warp 10 api/v0/exec: [ "token" "cpu.average" { "cpu"
"~cpu[0-9]+" } 1535818770 10 h ] FETCH [ SWAP bucketizer.mean 0 2 m 0 ] BUCKETIZE [ SWAP [ "host" ] reducer.max ] REDUCE 63

@IncontroDevOps @AurrelH95 Hello Exo World use case 64

@IncontroDevOps @AurrelH95 Warp10 - hands on 65

@IncontroDevOps @AurrelH95 Hello Exo World (HEW) result 66

@IncontroDevOps @AurrelH95 Warp 10 review ❖ Dedicated language ❖ A
time series workﬂow ❖ Queries complexity ❖ Abstraction needed to end user 67

@IncontroDevOps @AurrelH95 And the Elastic Stack? 68

@IncontroDevOps @AurrelH95 The Elastic Time Series stack .es(index=test*, metric=min:mem.available).mvavg(10) 69

@IncontroDevOps @AurrelH95 Does the job ❖ Mix of visualization ❖
Multiple series ❖ Lof of functions (functions) ❖ Less control on data ❖ Need a graphical tool (Timelion on Kibana) ❖ Lower query performance 70

@IncontroDevOps @AurrelH95 M3 TSDB POST /query { "namespace": "test", "query":
{ "regexp": { "field": "city", "regexp": ".*" } }, "rangeStart": 0, "rangeEnd":'"$(date +"%s")"' } 71

@IncontroDevOps @AurrelH95 Nobody’s is perfect 72

@IncontroDevOps @AurrelH95 Different use cases, differents TSDB... 73

@IncontroDevOps @AurrelH95 Different use cases, differents TSDB... Wait we are
missing one, aren't we? 74

@IncontroDevOps @AurrelH95 Different use cases, differents TSDB... Wait we are
missing one, aren't we? 75

@IncontroDevOps @AurrelH95 From InﬂuxQL SELECT max("usage_system") FROM "telegraf".."cpu" WHERE "host"
= 'ahe-XPS-13-9360' AND time > now() - 12h GROUP BY time(10m) ❖ First iteration ❖ Database queries ❖ Familiar SQL user 76

@IncontroDevOps @AurrelH95 InﬂuxQL drawback Time series data are NOT relational
InﬂuxQL had limitations for advanced use cases 77

@IncontroDevOps @AurrelH95 To IFQL select(db:"telegraf") .where(exp:{"_measurement"=="cpu" AND "_field"=="usage_system") .range(start:-12h) .window(every:10m)
.max() ❖ Time series API ❖ Functional paradigm ❖ Consistent semantics 78

@IncontroDevOps @AurrelH95 And ﬂux POST query= from(bucket:"telegraf") |> filter(fn: (r)
=> r._measurement == "cpu" AND r._field == "usage_system") |> range(start:-12h) |> group(by: ["host"]) |> window(every: 2m) |> max() ❖ Data language ❖ Lot of native functions ❖ User deﬁned function ❖ A usable language 79

@IncontroDevOps @AurrelH95 Flux as a query language 80 Working on
data locally is more powerful

@IncontroDevOps @AurrelH95 And Flux as a proxy Run the same
ﬂux operation on data loaded in memory Any LTS

@IncontroDevOps @AurrelH95 Flux as proxy - from Flux FromProm Parameters
Query, Matcher, Step, Basic auth FromRange Rewrite prom query with start and end FilterSpec Update prom query with flux filters Connect Connect flux to a Prometheus database Fetch Execute a Prom query or a Remote read and parse result as Influx data

@IncontroDevOps @AurrelH95 Flux - From Prometheus fromProm(query: "max(my.series{dc=\"test\"})", step: 120s)
|> range(start: -5m) fromProm(name:"my.series") |> filter(fn: (r) => r.host == "my.host" and r.dc == "test") |> range(start: -5m) |> max() ❖ Prometheus queries ❖ Prometheus Remote read

@IncontroDevOps @AurrelH95 A time series language? ❖ Quick access to
the data ❖ Times series native features ❖ Back-end agnostic ❖ Simplify user experience 84

@IncontroDevOps @AurrelH95 OVH State 85

@IncontroDevOps @AurrelH95 TSL aims 86 ❖ A clear analytics syntax
❖ Making it developer friendly ❖ Support data ﬂow and complex queries ❖ Keep it simple

@IncontroDevOps @AurrelH95 A simple TSL query 87 select("cpu.usage_system") .where("cpu~cpu[0-7]*") .last(12h)
.sampleBy(5m,max) .groupBy(mean) .rate() Select Where* Last? From? Sampling? GroupBy? Operations*

@IncontroDevOps @AurrelH95 More complex query 88 ❖ Declare and use
variables ❖ Times series sets operators ❖ Custom operators: meta, transform, arithmetic, windows, ﬁlters...

@IncontroDevOps @AurrelH95 HEW use case with TSL sample = select('sap.flux')
.where('KEPLERID=6541920') .from("2009-05-02T00:56:10.000000Z", to="2013-05-11T12:02:06.000000Z") .timesplit(6h,100,"record") .filterByLabels('record~[2-5]') .sampleBy(2h, min, false, "none") trend = sample.window(mean, 5, 5) sub(sample,trend) .on('KEPLERID','record') .lessThan(-20.0) ❖ Support complex use cases 89

@IncontroDevOps @AurrelH95 TSL is 90 ovh/tsl ovh/ovh-tsl-datasource

@IncontroDevOps @AurrelH95 Thanks!

A TSDB queries tour (DevOps Italia - Bologne)

A TSDB queries tour (DevOps Italia - Bologne)

More Decks by Aurélien HÉBERT

Other Decks in Programming

Featured

Transcript