Thanos - Prometheus at Scale

Slides from CloudNative 2018 London.

Video: https://skillsmatter.com/app/conferences/10160-cloudnative-london-2018/skillscasts/12104-thanos-prometheus-at-scale

The CNCF's Prometheus Monitoring system has been thriving for several years. Along with its powerful data model, operational simplicity and reliability have been a key factor in its success. However, some questions were still largely unaddressed to this day. How can you store historical data at the order of petabytes in a reliable and cost-efficient way? Can you do so without sacrificing responsive query times? And what about a global view of all your metrics and transparent handling of HA setups? Thanos takes Prometheus' strong foundations and extends it into a clustered, yet coordination free, globally scalable metric system. It retains Prometheus’s simple operational model and even simplifies deployments further. Under the hood, Thanos uses highly cost-efficient object storage that’s available in virtually all environments today. By building directly on top of the storage format introduced with Prometheus 2.0, Thanos achieves near real-time responsiveness even for cold queries against historical data. All while having virtually no cost overhead beyond that of the underlying object storage. During this talk, Bartek will explore the theoretical concepts behind Thanos and demonstrate how it seamlessly integrates into existing Prometheus setups.

Transcript

1. Bartek Płotka Prometheus at Scale London, 26th September 2018 github.com/improbable-eng/thanos @bwplotka
2. Bartek Płotka Software Engineer bartek@improbable.io @bwplotka
3. 310 Founded: Games in Development: +19 2012 Employees: "Improbable’s platform, SpatialOS, is designed to let anyone build massive simulations, running in the cloud: imagine Minecraft with thousands of players in the same space or researchers creating simulated cities to model the behaviour of millions. Its ultimate goal: to create totally immersive, persistent virtual worlds." - WIRED, May 2017
4. @bwplotka Agenda Prometheus
5. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus
6. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus Thanos
7. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus Thanos + + + + +
8. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus Thanos + + + + + + + + +
9. @bwplotka Prometheus Query Engine Scrape Engine Compactor Rule & Alert Engine Prometheus Service X Service X Services /metrics every 15s HTTP Query API Grafana Alertmanager Local storage
10. @bwplotka Cluster Single Prometheus Prometheus Grafana Alertmanager Workloads you want to monitor
11. @bwplotka Cluster Single Prometheus Prometheus Grafana Alertmanager Workloads you want to monitor 200 000 samples / sec = 1 CPU, 13 GB RAM = 3 mln series with 15s scrape interval.
12. @bwplotka Cluster High Availability Prometheus Grafana Alertmanager Workloads you want to monitor Prometheus Prometheus
13. @bwplotka Cluster 1 Globally distributed clusters Cluster 2 Prometheus Cluster n Cluster n+1 Prometheus ...
14. @bwplotka Problem: Global View Prometheus Prometheus sum(rate(go_memstats_alloc_bytes_total[1m])) ? source = “a” source = “b”
15. @bwplotka Problem: Global View Prometheus Prometheus sum(go_memstats_alloc_bytes_total::rate1m) ✓/? Prometheus /federate /federate source = “a” source = “b” every 15s
16. @bwplotka go_memstats_alloc_bytes_total{..., replica=A) Problem: Global View + HA Prometheus Prometheus go_memstats_alloc_bytes_total{..., replica=B) go_memstats_alloc_bytes_total{...} [deduplicated]
17. @bwplotka Problem: Metric retention
18. @bwplotka Problem: Metric retention SSD Prometheus Prometheus Remote write
19. @bwplotka Thanos Goals • Have a global view • Have a HA in place • Increase retention
20. Global View See everything from a single place!
21. @bwplotka SSD Prometheus Prometheus Targets
22. @bwplotka SSD Sidecar Prometheus Sidecar Targets gRPC (Store API) spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - sidecar - --http-address=0.0.0.0:19190 - --grpc-address=0.0.0.0:19090 - --tsdb.path=/prometheus-data - --prometheus.url=http://localhost:9090
23. @bwplotka Store API service Store { rpc Series(SeriesRequest) returns (stream SeriesResponse); rpc LabelNames(LabelNamesRequest) returns (LabelNamesResponse); rpc LabelValues(LabelValuesRequest) returns (LabelValuesResponse); } message SeriesRequest { int64 min_time = 1; int64 max_time = 2; repeated LabelMatcher matchers = 3; } Sidecar Prometheus remote read Store API
24. @bwplotka SSD Querier Prometheus Sidecar Querier Store API Targets HTTP Query API
25. @bwplotka SSD Global View Prometheus Sidecar Querier Targets SSD Sidecar Targets Prometheus Merge Store API
26. @bwplotka SSD Global View + Availability Prometheus Sidecar Targets SSD Sidecar Targets Prometheus SSD Sidecar Prometheus “replica”:”A” “replica”:”B” Querier Merge Deduplicate Store API
27. @bwplotka Thanos Goals • Have a global view ✓ • Have a HA in place ✓ Prometheus Sidecar SSD Sidecar Prometheus Sidecar Prometheus Querier spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - query - --http-address=0.0.0.0:9090 - --grpc-address=0.0.0.0:19090 - --query.replica-label=replica - --store=sidecar1.default.svc:19090 - --store=sidecar2-a.default.svc:19090 - --store=sidecar2-b.default.svc:19090
28. @bwplotka Global View + Availability
29. @bwplotka Global View + Availability
30. Historical Metrics What exactly happened X months ago?
31. @bwplotka TSDB Layout Block 4 Block 3 Block 1 chunks chunks chunks chunks index T-10h T-16h T-4h T-2h T
32. @bwplotka SSD Data saving Prometheus Sidecar Targets Object Storage Blocks Blocks Block --gcs.bucket=...
33. @bwplotka SSD Data saving Prometheus Sidecar Targets Object Storage Blocks Blocks Block --storage.tsdb.max-block-duration=2h --storage.tsdb.retention=12h
34. @bwplotka Store Gateway Object Storage Blocks Cache Store Querier Store API spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - store - --http-address=0.0.0.0:19190 - --grpc-address=0.0.0.0:19090 - --store-dir=/store-data - --gcs.bucket=... - --index-cache-size=1GB - --chunk-pool-size=8GB
35. @bwplotka spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - rule - --http-address=0.0.0.0:19190 - --grpc-address=0.0.0.0:19090 - --data-dir=/rule-data - --gcs.bucket=... - --rule-file=... - --query-url=... Object Storage Blocks Ruler Ruler Querier Store API Query SSD Blocks
36. Compaction Density matters
37. @bwplotka Compaction Object Storage Blocks Disk Compactor
38. @bwplotka Compaction Object Storage Blocks Disk Compactor Block spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - compact - --http-address=0.0.0.0:80 - --data-dir=/compactor-data - --gcs.bucket=... - --wait Blocks Block
39. Downsampling
40. @bwplotka Downsampling Raw: 16 bytes/sample Compressed: 1.07 bytes/sample
41. @bwplotka Downsampling Decompressing one sample takes 10-40 nanoseconds.
42. @bwplotka Downsampling Decompressing one sample takes 10-40 nanoseconds. Decompressing 1000 series @ 30s scrape interval for 1 year data takes 10-40 seconds alone. Plus your actual computation over all those samples, e.g. rate()
43. @bwplotka Downsampling Block RAW Block @ 5m Block @ 1h 10x 12x
44. @bwplotka Thanos Goals • Have a global view ✓ • Have a HA in place ✓ • Increase retention ✓
45. @bwplotka Prometheus Query Engine Scrape Engine Compactor Rule & Alert Engine Prometheus
46. @bwplotka Thanos Scrape Engine Compactor Rule & Alert Engine Thanos Querier Thanos Querier Thanos Querier
47. @bwplotka Thanos Compactor Rule & Alert Engine Thanos Querier Thanos Querier Thanos Querier SSD Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar
48. @bwplotka Thanos Compactor Thanos Querier Thanos Querier Thanos Querier SSD Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar Thanos Ruler Thanos Ruler
49. @bwplotka Thanos Thanos Ruler Thanos Ruler Thanos Querier Thanos Querier Thanos Querier SSD Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar Global Compactor
50. @bwplotka Thanos Store Gateway Store Gateway Object Storage SSD Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar Thanos Ruler Thanos Ruler Global Compactor Thanos Querier Thanos Querier Thanos Querier
51. Deployment Models
52. @bwplotka Just Global Querier Querier Querier Querier Cluster A (master) Cluster B Cluster C + + + + + +
53. @bwplotka Federated Querier Querier Querier Querier Querier Querier Querier … Querier Querier Querier Cluster A (master) Cluster B Cluster C + + Federation (through Store API) + + + +
54. @bwplotka Federated Deployment Querier Querier Querier Store Bucket Querier Querier Querier … Store Bucket Querier Querier Querier Store Bucket Cluster A (master) Cluster B Cluster C + + Federation (through Store API) + + + +
55. @bwplotka Hierarchical Deployment Cluster 1 Cluster 2 + Cluster n Cluster n+1 + ... + Monitoring Cluster Grafana Alertmanager Bucket Compactor Querier Querier Querier Ruler Store Statically configured / global discovery +
56. @bwplotka Hierarchical Deployment ++++ ++ ++ ++++ ++ ++ ++++ ++ ++ Testing Staging Production Querier Querier Querier
57. @bwplotka Thanos Goals • Have a global view ✓ • Have a HA in place ✓ • Increase retention ✓
58. @bwplotka Next steps • New object store providers! • Memory optimizations • Prometheus: Stream remote read • Multi Tenancy extension • Query safeguards
59. @bwplotka Summary • Start small with just single Prometheus. • Extend your setup gradually. ◦ Just global querier. ◦ Federated global querier. ◦ Object storage. • Model your Thanos deployment.
60. github.com/improbable-eng/thanos Bartek Płotka @bwplotka improbable.io ...psst, join our slack workspace for more info! Credits: • Percona blog post about Prometheus 2 perf • Emojis designed by Freepik from Flaticon