Tweet
Tweet

Slide 1

Slide 1 text

Storage in Prometheus 2.0 Goutham Veeramachaneni Intern @ CoreOS, Berlin Student at IIT Hyderabad, India : gouthamve : putadent

Slide 2

Slide 2 text

Thats a long way to come to give a talk!

Slide 3

Slide 3 text

The Problem Too many Time-Series! Time-Series??

Slide 4

Slide 4 text

Time Series time (t0, v0), (t1, v1), (t2, v2), (t3, v3), ....

Slide 5

Slide 5 text

Time Series time series

Slide 6

Slide 6 text

Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} ...

Slide 7

Slide 7 text

Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total

Slide 8

Slide 8 text

Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total{method="GET"}

Slide 9

Slide 9 text

Time Series time series

Slide 10

Slide 10 text

Time Series time series

Slide 11

Slide 11 text

Time Series time series

Slide 12

Slide 12 text

How do you get these time-series? What you expose: requests_total{path="/status", method="GET"} requests_total{path="/status", method="POST"} requests_total{path="/", method="GET"} What prometheus scrapes: requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.1:80"} requests_total{path="/", method="GET", instance="10.0.0.1:80"}

Slide 13

Slide 13 text

Scale 5 million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk 3,000 - 15,000 microservice instances

Slide 14

Slide 14 text

Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765

Slide 15

Slide 15 text

Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765 1496163646 +30 +30 +29 +30 Δ

Slide 16

Slide 16 text

Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765 1496163646 +30 +30 +29 +30 Δ 1496163646 +30 +0 -1 +1 ΔΔ

Slide 17

Slide 17 text

Compression - Value http://www.vldb.org/pvldb/vol8/p1816-teller.pdf

Slide 18

Slide 18 text

Compression - Value http://www.vldb.org/pvldb/vol8/p1816-teller.pdf bytes/sample Raw: 16 Compressed: 1.37 7TB => 600GB

Slide 19

Slide 19 text

Scale 5 million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 600GB on disk 3,000 - 15,000 microservice instances

Slide 20

Slide 20 text

Churn! time series

Slide 21

Slide 21 text

How do you get these time-series? What you expose: requests_total{path="/status", method="GET"} requests_total{path="/status", method="POST"} requests_total{path="/", method="GET"} What prometheus scrapes: requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.1:80"} requests_total{path="/", method="GET", instance="10.0.0.1:80"}

Slide 22

Slide 22 text

Churn! time series

Slide 23

Slide 23 text

Scale 5 million active time series 150 million total time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk

Slide 24

Slide 24 text

1.0 storage ┌──────────┬─────────┬─────────┬─────────┬─────────┐ series A └──────────┴─────────┴─────────┴─────────┴─────────┘ ┌──────────┬─────────┬─────────┬─────────┬─────────┐ series B └──────────┴─────────┴─────────┴─────────┴─────────┘ . . . ┌──────────┬─────────┬─────────┬─────────┬─────────┬─────────┐ series XYZ └──────────┴─────────┴─────────┴─────────┴─────────┴─────────┘ chunk 1 chunk 2 chunk 3 ...

Slide 25

Slide 25 text

Storage 1.0 1 file per series!

Slide 26

Slide 26 text

1.0 Querying 1. Get series labels 2. Calculate Fingerprint 3. Add the fingerprint against the label { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } Fingerprint: 3300 To identify each series.

Slide 27

Slide 27 text

1.0 Querying 1. Get series labels 2. Calculate Fingerprint 3. Add the fingerprint against the label-value pair { __name__=”reqs”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200” : 1000 500000 99 1 1500 2 1001 5 1502 method=”GET” : 2 999999 4 3 1502 9 6 5 10 ...

Slide 28

Slide 28 text

The problems ● Too many files ● The index gets huuuuge

Slide 29

Slide 29 text

Enter 2.0

Slide 30

Slide 30 text

Data and Queries time series

Slide 31

Slide 31 text

Time sharded blocks time series

Slide 32

Slide 32 text

2.0 Structure mutable prometheus write t0 t1 t2 t3 now query merge

Slide 33

Slide 33 text

2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 34

Slide 34 text

2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 35

Slide 35 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 36

Slide 36 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 37

Slide 37 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 38

Slide 38 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 39

Slide 39 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 40

Slide 40 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 41

Slide 41 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 42

Slide 42 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 43

Slide 43 text

2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● Assign block-scoped ID to each series ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations

Slide 44

Slide 44 text

Why time-shard? time series

Slide 45

Slide 45 text

The problems? ● Just a bunch of files. ● Time-sharding saves index

Slide 46

Slide 46 text

Benchmarks

Slide 47

Slide 47 text

Benchmarks Kubernetes cluster + dedicated Prometheus nodes 800 microservice instances + Kubernetes components 120,000 samples/second 300,000 active time series Swap out 50% of pods every 10 minutes

Slide 48

Slide 48 text

Benchmarks (Memory: GB)

Slide 49

Slide 49 text

Benchmarks (CPU: Cores used)

Slide 50

Slide 50 text

Benchmarks (disk writes: MB)

Slide 51

Slide 51 text

Benchmarks (on-disk size: GB)

Slide 52

Slide 52 text

Benchmarks (query latency: secs)

Slide 53

Slide 53 text

More 2.0

Slide 54

Slide 54 text

Blocks recap time series

Slide 55

Slide 55 text

Compaction time series

Slide 56

Slide 56 text

Compaction time series

Slide 57

Slide 57 text

Retention time series

Slide 58

Slide 58 text

Retention time series

Slide 59

Slide 59 text

Deletions

Slide 60

Slide 60 text

Deletion time series

Slide 61

Slide 61 text

Tombstones Series-Ref ---> Deleted Ranges { 190: [{100, 200}, {300, 600}], 250: [{100, 5000}], } When the querier, runs it pickups these ranges. If something is deleted, we skip that range in the query.

Slide 62

Slide 62 text

Tombstones Removal time series

Slide 63

Slide 63 text

Tombstones Removal time series time series

Slide 64

Slide 64 text

Backups

Slide 65

Slide 65 text

BACKUPS ARE HEEEERE! Step 1: Stop mutation and “save” the block (hard-link) Step 2: Persist in-memory blocks Step 3: Restart mutation Step 3: Backup the data lazily and safely Snapshots

Slide 66

Slide 66 text

How it works Data dir Backup dir

Slide 67

Slide 67 text

How it works Data dir Backup dir

Slide 68

Slide 68 text

How it works Data dir Backup dir

Slide 69

Slide 69 text

Demo (Based on time) ● Delete ● Backup