Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Thanos - Prometheus at Scale

55cdaff2951048a85f370467b324ed2e?s=47 Bartek
September 26, 2018

Thanos - Prometheus at Scale

Slides from CloudNative 2018 London.

Video: https://skillsmatter.com/app/conferences/10160-cloudnative-london-2018/skillscasts/12104-thanos-prometheus-at-scale

The CNCF's Prometheus Monitoring system has been thriving for several years. Along with its powerful data model, operational simplicity and reliability have been a key factor in its success. However, some questions were still largely unaddressed to this day. How can you store historical data at the order of petabytes in a reliable and cost-efficient way? Can you do so without sacrificing responsive query times? And what about a global view of all your metrics and transparent handling of HA setups? Thanos takes Prometheus' strong foundations and extends it into a clustered, yet coordination free, globally scalable metric system. It retains Prometheus’s simple operational model and even simplifies deployments further. Under the hood, Thanos uses highly cost-efficient object storage that’s available in virtually all environments today. By building directly on top of the storage format introduced with Prometheus 2.0, Thanos achieves near real-time responsiveness even for cold queries against historical data. All while having virtually no cost overhead beyond that of the underlying object storage. During this talk, Bartek will explore the theoretical concepts behind Thanos and demonstrate how it seamlessly integrates into existing Prometheus setups.

55cdaff2951048a85f370467b324ed2e?s=128

Bartek

September 26, 2018
Tweet

Transcript

  1. Bartek Płotka Prometheus at Scale London, 26th September 2018 github.com/improbable-eng/thanos

    @bwplotka
  2. Bartek Płotka Software Engineer bartek@improbable.io @bwplotka

  3. 310 Founded: Games in Development: +19 2012 Employees: "Improbable’s platform,

    SpatialOS, is designed to let anyone build massive simulations, running in the cloud: imagine Minecraft with thousands of players in the same space or researchers creating simulated cities to model the behaviour of millions. Its ultimate goal: to create totally immersive, persistent virtual worlds." - WIRED, May 2017
  4. @bwplotka Agenda Prometheus

  5. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus

  6. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus Thanos

  7. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus Thanos +

    + + + +
  8. @bwplotka Agenda Prometheus Prometheus Prometheus Prometheus Prometheus Prometheus Thanos +

    + + + + + + + +
  9. @bwplotka Prometheus Query Engine Scrape Engine Compactor Rule & Alert

    Engine Prometheus Service X Service X Services /metrics every 15s HTTP Query API Grafana Alertmanager Local storage
  10. @bwplotka Cluster Single Prometheus Prometheus Grafana Alertmanager Workloads you want

    to monitor
  11. @bwplotka Cluster Single Prometheus Prometheus Grafana Alertmanager Workloads you want

    to monitor 200 000 samples / sec = 1 CPU, 13 GB RAM = 3 mln series with 15s scrape interval.
  12. @bwplotka Cluster High Availability Prometheus Grafana Alertmanager Workloads you want

    to monitor Prometheus Prometheus
  13. @bwplotka Cluster 1 Globally distributed clusters Cluster 2 Prometheus Cluster

    n Cluster n+1 Prometheus ...
  14. @bwplotka Problem: Global View Prometheus Prometheus sum(rate(go_memstats_alloc_bytes_total[1m])) ? source =

    “a” source = “b”
  15. @bwplotka Problem: Global View Prometheus Prometheus sum(go_memstats_alloc_bytes_total::rate1m) ✓/? Prometheus /federate

    /federate source = “a” source = “b” every 15s
  16. @bwplotka go_memstats_alloc_bytes_total{..., replica=A) Problem: Global View + HA Prometheus Prometheus

    go_memstats_alloc_bytes_total{..., replica=B) go_memstats_alloc_bytes_total{...} [deduplicated]
  17. @bwplotka Problem: Metric retention

  18. @bwplotka Problem: Metric retention SSD Prometheus Prometheus Remote write

  19. @bwplotka Thanos Goals • Have a global view • Have

    a HA in place • Increase retention
  20. Global View See everything from a single place!

  21. @bwplotka SSD Prometheus Prometheus Targets

  22. @bwplotka SSD Sidecar Prometheus Sidecar Targets gRPC (Store API) spec:

    containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - sidecar - --http-address=0.0.0.0:19190 - --grpc-address=0.0.0.0:19090 - --tsdb.path=/prometheus-data - --prometheus.url=http://localhost:9090
  23. @bwplotka Store API service Store { rpc Series(SeriesRequest) returns (stream

    SeriesResponse); rpc LabelNames(LabelNamesRequest) returns (LabelNamesResponse); rpc LabelValues(LabelValuesRequest) returns (LabelValuesResponse); } message SeriesRequest { int64 min_time = 1; int64 max_time = 2; repeated LabelMatcher matchers = 3; } Sidecar Prometheus remote read Store API
  24. @bwplotka SSD Querier Prometheus Sidecar Querier Store API Targets HTTP

    Query API
  25. @bwplotka SSD Global View Prometheus Sidecar Querier Targets SSD Sidecar

    Targets Prometheus Merge Store API
  26. @bwplotka SSD Global View + Availability Prometheus Sidecar Targets SSD

    Sidecar Targets Prometheus SSD Sidecar Prometheus “replica”:”A” “replica”:”B” Querier Merge Deduplicate Store API
  27. @bwplotka Thanos Goals • Have a global view ✓ •

    Have a HA in place ✓ Prometheus Sidecar SSD Sidecar Prometheus Sidecar Prometheus Querier spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - query - --http-address=0.0.0.0:9090 - --grpc-address=0.0.0.0:19090 - --query.replica-label=replica - --store=sidecar1.default.svc:19090 - --store=sidecar2-a.default.svc:19090 - --store=sidecar2-b.default.svc:19090
  28. @bwplotka Global View + Availability

  29. @bwplotka Global View + Availability

  30. Historical Metrics What exactly happened X months ago?

  31. @bwplotka TSDB Layout Block 4 Block 3 Block 1 chunks

    chunks chunks chunks index T-10h T-16h T-4h T-2h T
  32. @bwplotka SSD Data saving Prometheus Sidecar Targets Object Storage Blocks

    Blocks Block --gcs.bucket=...
  33. @bwplotka SSD Data saving Prometheus Sidecar Targets Object Storage Blocks

    Blocks Block --storage.tsdb.max-block-duration=2h --storage.tsdb.retention=12h
  34. @bwplotka Store Gateway Object Storage Blocks Cache Store Querier Store

    API spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args: - store - --http-address=0.0.0.0:19190 - --grpc-address=0.0.0.0:19090 - --store-dir=/store-data - --gcs.bucket=... - --index-cache-size=1GB - --chunk-pool-size=8GB
  35. @bwplotka spec: containers: - image: improbable/thanos:v0.1.0 command: - thanos args:

    - rule - --http-address=0.0.0.0:19190 - --grpc-address=0.0.0.0:19090 - --data-dir=/rule-data - --gcs.bucket=... - --rule-file=... - --query-url=... Object Storage Blocks Ruler Ruler Querier Store API Query SSD Blocks
  36. Compaction Density matters

  37. @bwplotka Compaction Object Storage Blocks Disk Compactor

  38. @bwplotka Compaction Object Storage Blocks Disk Compactor Block spec: containers:

    - image: improbable/thanos:v0.1.0 command: - thanos args: - compact - --http-address=0.0.0.0:80 - --data-dir=/compactor-data - --gcs.bucket=... - --wait Blocks Block
  39. Downsampling

  40. @bwplotka Downsampling Raw: 16 bytes/sample Compressed: 1.07 bytes/sample

  41. @bwplotka Downsampling Decompressing one sample takes 10-40 nanoseconds.

  42. @bwplotka Downsampling Decompressing one sample takes 10-40 nanoseconds. Decompressing 1000

    series @ 30s scrape interval for 1 year data takes 10-40 seconds alone. Plus your actual computation over all those samples, e.g. rate()
  43. @bwplotka Downsampling Block RAW Block @ 5m Block @ 1h

    10x 12x
  44. @bwplotka Thanos Goals • Have a global view ✓ •

    Have a HA in place ✓ • Increase retention ✓
  45. @bwplotka Prometheus Query Engine Scrape Engine Compactor Rule & Alert

    Engine Prometheus
  46. @bwplotka Thanos Scrape Engine Compactor Rule & Alert Engine Thanos

    Querier Thanos Querier Thanos Querier
  47. @bwplotka Thanos Compactor Rule & Alert Engine Thanos Querier Thanos

    Querier Thanos Querier SSD Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar
  48. @bwplotka Thanos Compactor Thanos Querier Thanos Querier Thanos Querier SSD

    Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar Thanos Ruler Thanos Ruler
  49. @bwplotka Thanos Thanos Ruler Thanos Ruler Thanos Querier Thanos Querier

    Thanos Querier SSD Prometheus Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar Global Compactor
  50. @bwplotka Thanos Store Gateway Store Gateway Object Storage SSD Prometheus

    Sidecar SSD Prometheus Sidecar SSD Prometheus Sidecar Thanos Ruler Thanos Ruler Global Compactor Thanos Querier Thanos Querier Thanos Querier
  51. Deployment Models

  52. @bwplotka Just Global Querier Querier Querier Querier Cluster A (master)

    Cluster B Cluster C + + + + + +
  53. @bwplotka Federated Querier Querier Querier Querier Querier Querier Querier …

    Querier Querier Querier Cluster A (master) Cluster B Cluster C + + Federation (through Store API) + + + +
  54. @bwplotka Federated Deployment Querier Querier Querier Store Bucket Querier Querier

    Querier … Store Bucket Querier Querier Querier Store Bucket Cluster A (master) Cluster B Cluster C + + Federation (through Store API) + + + +
  55. @bwplotka Hierarchical Deployment Cluster 1 Cluster 2 + Cluster n

    Cluster n+1 + ... + Monitoring Cluster Grafana Alertmanager Bucket Compactor Querier Querier Querier Ruler Store Statically configured / global discovery +
  56. @bwplotka Hierarchical Deployment ++++ ++ ++ ++++ ++ ++ ++++

    ++ ++ Testing Staging Production Querier Querier Querier
  57. @bwplotka Thanos Goals • Have a global view ✓ •

    Have a HA in place ✓ • Increase retention ✓
  58. @bwplotka Next steps • New object store providers! • Memory

    optimizations • Prometheus: Stream remote read • Multi Tenancy extension • Query safeguards
  59. @bwplotka Summary • Start small with just single Prometheus. •

    Extend your setup gradually. ◦ Just global querier. ◦ Federated global querier. ◦ Object storage. • Model your Thanos deployment.
  60. github.com/improbable-eng/thanos Bartek Płotka @bwplotka improbable.io ...psst, join our slack workspace

    for more info! Credits: • Percona blog post about Prometheus 2 perf • Emojis designed by Freepik from Flaticon