Storage in 2.0

Orchestration systems such as Kubernetes are rapidly gaining traction and unlock features of highly dynamic environments, such as frequent rolling updates and auto-scaling, for everyone. This causes the number of series to blow up, putting new strains on Prometheus’ one-file-per-series data layout. This talk will be laying out the problems with the 1.0 storage and how the new storage engine, built from the ground up, solves them while also producing a sharp increase in performance. Storage 2.0 also has additional features, like granular, time-ranged deletes and live backups whose implementation would be discussed.

Transcript

1. Storage in Prometheus 2.0 Goutham Veeramachaneni Intern @ CoreOS, Berlin Student at IIT Hyderabad, India : gouthamve : putadent
2. Thats a long way to come to give a talk!
3. The Problem Too many Time-Series! Time-Series??
4. Time Series time (t0, v0), (t1, v1), (t2, v2), (t3, v3), ....
5. Time Series time series
6. Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} ...
7. Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total
8. Time Series requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total{method="GET"}
9. Time Series time series
10. Time Series time series
11. Time Series time series
12. How do you get these time-series? What you expose: requests_total{path="/status", method="GET"} requests_total{path="/status", method="POST"} requests_total{path="/", method="GET"} What prometheus scrapes: requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.1:80"} requests_total{path="/", method="GET", instance="10.0.0.1:80"}
13. Scale 5 million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk 3,000 - 15,000 microservice instances
14. Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765
15. Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765 1496163646 +30 +30 +29 +30 Δ
16. Compression - Timestamp 1496163646 1496163676 1496163706 1496163735 1496163765 1496163646 +30 +30 +29 +30 Δ 1496163646 +30 +0 -1 +1 ΔΔ
17. Compression - Value http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
18. Compression - Value http://www.vldb.org/pvldb/vol8/p1816-teller.pdf bytes/sample Raw: 16 Compressed: 1.37 7TB => 600GB
19. Scale 5 million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 600GB on disk 3,000 - 15,000 microservice instances
20. Churn! time series
21. How do you get these time-series? What you expose: requests_total{path="/status", method="GET"} requests_total{path="/status", method="POST"} requests_total{path="/", method="GET"} What prometheus scrapes: requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.1:80"} requests_total{path="/", method="GET", instance="10.0.0.1:80"}
22. Churn! time series
23. Scale 5 million active time series 150 million total time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk
24. 1.0 storage ┌──────────┬─────────┬─────────┬─────────┬─────────┐ series A └──────────┴─────────┴─────────┴─────────┴─────────┘ ┌──────────┬─────────┬─────────┬─────────┬─────────┐ series B └──────────┴─────────┴─────────┴─────────┴─────────┘ . . . ┌──────────┬─────────┬─────────┬─────────┬─────────┬─────────┐ series XYZ └──────────┴─────────┴─────────┴─────────┴─────────┴─────────┘ chunk 1 chunk 2 chunk 3 ...
25. Storage 1.0 1 file per series!
26. 1.0 Querying 1. Get series labels 2. Calculate Fingerprint 3. Add the fingerprint against the label { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } Fingerprint: 3300 To identify each series.
27. 1.0 Querying 1. Get series labels 2. Calculate Fingerprint 3. Add the fingerprint against the label-value pair { __name__=”reqs”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200” : 1000 500000 99 1 1500 2 1001 5 1502 method=”GET” : 2 999999 4 3 1502 9 6 5 10 ...
28. The problems • Too many files • The index gets huuuuge
29. Enter 2.0
30. Data and Queries time series
31. Time sharded blocks time series
32. 2.0 Structure mutable prometheus write t0 t1 t2 t3 now query merge
33. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
34. 2.0 Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
35. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
36. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
37. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
38. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
39. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
40. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
41. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
42. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
43. 2.0 Index status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502 { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } • Assign block-scoped ID to each series • Maintain sorted lists from label pair to IDs • Efficient k-way set operations
44. Why time-shard? time series
45. The problems? • Just a bunch of files. • Time-sharding saves index
46. Benchmarks
47. Benchmarks Kubernetes cluster + dedicated Prometheus nodes 800 microservice instances + Kubernetes components 120,000 samples/second 300,000 active time series Swap out 50% of pods every 10 minutes
48. Benchmarks (Memory: GB)
49. Benchmarks (CPU: Cores used)
50. Benchmarks (disk writes: MB)
51. Benchmarks (on-disk size: GB)
52. Benchmarks (query latency: secs)
53. More 2.0
54. Blocks recap time series
55. Compaction time series
56. Compaction time series
57. Retention time series
58. Retention time series
59. Deletions
60. Deletion time series
61. Tombstones Series-Ref ---> Deleted Ranges { 190: [{100, 200}, {300, 600}], 250: [{100, 5000}], } When the querier, runs it pickups these ranges. If something is deleted, we skip that range in the query.
62. Tombstones Removal time series
63. Tombstones Removal time series time series
64. Backups
65. BACKUPS ARE HEEEERE! Step 1: Stop mutation and “save” the block (hard-link) Step 2: Persist in-memory blocks Step 3: Restart mutation Step 3: Backup the data lazily and safely Snapshots
66. How it works Data dir Backup dir
67. How it works Data dir Backup dir
68. How it works Data dir Backup dir
69. Demo (Based on time) • Delete • Backup