Introduction to Time Series (Cloud Native Wales, April 2019)

Slide 1

Slide 1 text

Noswaith dda David McKay @rawkode Developer Relations Manager @InfluxDB | #InfluxDB Cloud Native Wales

Slide 2

Slide 2 text

Introduction to Time Series

Slide 3

Slide 3 text

Before we begin …

Slide 4

Slide 4 text

Pop Quiz “Invented” When?

Slide 5

Slide 5 text

Encoding First Used … ? First Used 410 BC

Slide 6

Slide 6 text

Encoding “Documented” in The Lives of the Noble Grecians and Romans, by Roman historian Plutarch.

Slide 7

Slide 7 text

Alcibiades suddenly raised the Athenian ensign in the admiral shop, and fell upon those galleys of the Peloponnesians …

Slide 8

Slide 8 text

In the 14th century, things hadn’t actually advanced much more. The Black Book of Admiralty listed 2 signals: 1 flag or 2 flags Encoding

Slide 9

Slide 9 text

By the 15th century there were 15 flags, each with a single meaning. Encoding

Slide 10

Slide 10 text

Finally, in the late 17th century; a French system existed (Mahé de la Bourdonnais) with 10 coloured flags, representing 0-9 Encoding

Slide 11

Slide 11 text

Sharding First Used … ? First Used 150 BC

Slide 12

Slide 12 text

Sharding First “documented” example was in ~150 AD, invented and described by Polybius.

Slide 13

Slide 13 text

We take the alphabet and divide it into five parts, each consisting of five letters.

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

History of Time Series

Slide 17

Slide 17 text

The earliest form of a company which issued public shares was the case of the publicani during the Roman Republic. The Romans Did It

Slide 18

Slide 18 text

Like modern joint-stock companies, the publicani were legal bodies independent of their members whose ownership was divided into shares, or partes. There is evidence that these shares were sold to public investors and traded in a type of over-the-counter market in the Forum, near the Temple of Castor and Pollux. The shares fluctuated in value, encouraging the activity of speculators, or quaestors.

Slide 19

Slide 19 text

In 1602 … First IPO: Dutch East India Company

Slide 20

Slide 20 text

In 1873 … First US IPO: Bank of North America

Slide 21

Slide 21 text

In 1884 … What was the price of wheat?

Slide 22

Slide 22 text

A Comparison of the Fluctuations in the Price of Wheat and in the Cotton and Silk Imports into Great Britain First Documented Time Series J. H. Poynting Journal of the Statistical Society of London Vol. 47, No. 1 (Mar., 1884), pp. 34-74

Slide 23

Slide 23 text

What is all this? This is the first (or one of) paper that added the dimension of time to statistical mathematics

Slide 24

Slide 24 text

All data becomes an order of magnitude more interesting on the time dimension @rawkode

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Most data is best understood in the dimension of time @pauldix, CTO

Slide 27

Slide 27 text

Introduction to Time Series Finally!

Slide 28

Slide 28 text

What Will We Cover? ➔ Time Series Data ➔ Time Series Databases ➔ Getting to Know InfluxDB ➔ Value of Time Series Data ➔ Advancing Monitoring to Time Series

Slide 29

Slide 29 text

Time Series Data What is it?

Slide 30

Slide 30 text

Time Series Data Data with a timestamp

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

What is Time Series Data?

Slide 38

Slide 38 text

Irregular (Events) ➔ Unpredictable ➔ Inconsistent Intervals What is Time Series Data? Regular (Metrics) ➔ Predictable ➔ Evenly Distributed

Slide 39

Slide 39 text

Regular / Metrics ★ CPU Usage ★ Memory Usage ★ Ping Time for Google.com ★ Number of Processes

Slide 40

Slide 40 text

Irregular / Events ★ User Clicked Login ★ Authentication Failed ★ CI Published v1.3.1 ★ Network Cable Unplugged

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Time Series Data Use Cases

Slide 43

Slide 43 text

IoT / Sensor ➔ Thermostats ➔ Electric Engines ➔ Smart Things ➔ GPS ➔ Fitbits Real Time Analytics ➔ Website Tracking ➔ Stock Prices ➔ Currency Exchange Rates Use Cases for Time Series Monitoring ➔ Infrastructure ➔ Applications ➔ Third Party Services

Slide 44

Slide 44 text

Time Series Databases TSDB’s

Slide 45

Slide 45 text

Time Series databases are optimized for collecting, storing, retrieving, and processing of Time Series data. Time Series Databases

Slide 46

Slide 46 text

➔ High Write Frequency ➔ Reads are range scans ➔ TTL / Lifecycle Management ➔ Time Sensitive Time Series Databases

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

12% Are you in the 88%?

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

13% It’s Not Too Late!

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Disclaimer Most of this isn’t unique to InfluxDB

Slide 57

Slide 57 text

InfluxDB Introductions

Slide 58

Slide 58 text

InfluxDB ➔ TSDB ➔ Open-Source ➔ FullStack (Telegraf, InfluxDB, Chronograf, and Kapacitor) ➔ v2 …

Slide 59

Slide 59 text

Points At any point in time, this value was N

Slide 60

Slide 60 text

load,host=vm1 1m=6.32,5m=8.20,15m=9.55 123456789 Point ● Series ● Fields ● Timestamp

Slide 61

Slide 61 text

● load,host=vm1 ● stock_price,market=NASDAQ,ticker=GOOG ● users,service=comments Series ● Name ● Tag Keys ● Tag Values

Slide 62

Slide 62 text

Fields ➔ Not Indexed ➔ Multiple Data Types Tags & Fields Tags ➔ Indexed ➔ String Types

Slide 63

Slide 63 text

Value of Time Series Data Isn’t It Valuable Forever?

Slide 64

Slide 64 text

Resolution The predictable interval at which we will collect our time series data

Slide 65

Slide 65 text

The value of all time series data is directly correlated with the resolution that the data is available Value of Time Series Data

Slide 66

Slide 66 text

Cost of Time Series Data Wait, Isn’t It Free?!

Slide 67

Slide 67 text

Example cpu,machine=abc1 usage=1.66 timestamp

Slide 68

Slide 68 text

Resolution ➔ 1 Measurement ➔ 1 Series ➔ 1s Resolution 86400 Points Per Day

Slide 69

Slide 69 text

Resolution ➔ 1 Measurement ➔ 2 Series ➔ 1s Resolution 172800 Points Per Day

Slide 70

Slide 70 text

Resolution 4320000 Points Per Day ➔ 5 Measurement ➔ 10 Series ➔ 1s Resolution

Slide 71

Slide 71 text

Nasdaq 28512000 0000 Points Per Day ➔ 1 Measurement ➔ 3300 Series ➔ 1ms Resolution

Slide 72

Slide 72 text

Nasdaq 4752000 Points Per Day ➔ 1 Measurement ➔ 3300 Series ➔ 1m Resolution

Slide 73

Slide 73 text

Nasdaq 79200 Points Per Day ➔ 1 Measurement ➔ 3300 Series ➔ 1h Resolution

Slide 74

Slide 74 text

Nasdaq 13200 Points Per Day ➔ 1 Measurement ➔ 3300 Series ➔ 6h Resolution

Slide 75

Slide 75 text

Rollups Lowering the Resolution

Slide 76

Slide 76 text

Rollups with Continuous Queries CREATE CONTINUOUS QUERY "rollup_1h" ON "nasdaq" BEGIN SELECT mean(price) INTO yearly FROM weekly GROUP BY time(1h) END

Slide 77

Slide 77 text

Events? Outlier / Anomaly Detection InfluxDB Anomaly Detection

Slide 78

Slide 78 text

Advancing Monitoring to Time Series Taking Small Steps for Giant Leaps

Slide 79

Slide 79 text

Application Database CPU > 80% MEM > 80% Response Time > 300ms Black Friday

Slide 80

Slide 80 text

Service A Database A Service B Service B Service C Database B Database C Virtual Network Service Mesh Canary Ummm?

Slide 81

Slide 81 text

Cloud Native Architectures Convenience Vs. Cost You can treat the symptoms for a while … Upgrade Your Monitoring

Slide 82

Slide 82 text

Causality Treating the Disease

Slide 83

Slide 83 text

➔ Look at last weeks, months, and years of data ➔ Use tags to build correlation ➔ Get Statistical ◆ INTEGRAL() ◆ LINEAR_PREDICTION() ◆ DERIVATIVE() ◆ MOVING_AVERAGE() ◆ HOLT_WINTERS() Causality

Slide 84

Slide 84 text

Build Automation Through Causality, Historical Data, Prediction, and ML

Slide 85

Slide 85 text

➔ Use a TSDB ➔ Understand Cost / Select Tags Wisely ➔ Understand the resolution you need for 1m, 6m, > 12m Summary ➔ Rollup metrics ➔ Perform outlier detection on events ➔ Build automation, dashboarding, and reporting around your data (past, present, and future)

Slide 86

Slide 86 text

Thank You David McKay @rawkode Developer Relations Manager @InfluxDB | #InfluxDB That’s All Folks!