Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Thanos Deep Dive: Look into Distributed System

55cdaff2951048a85f370467b324ed2e?s=47 Bartek
November 20, 2019

Thanos Deep Dive: Look into Distributed System

KubeCon 2019

55cdaff2951048a85f370467b324ed2e?s=128

Bartek

November 20, 2019
Tweet

Transcript

  1. @ThanosMetrics Inside a Distributed Monitoring System San Diego, 20th November

    2019 Bartłomiej Płotka Frederic Branczyk brancz fredbrancz bwplotka
  2. @ThanosMetrics Speakers Frederic Branczyk Principal Software Engineer @ Red Hat;

    OpenShift Monitoring Team Prometheus Maintainer; Thanos Maintainer; SIG Instrumentation Lead Bartek Plotka Principal Software Engineer @ Red Hat; OpenShift Monitoring Team Prometheus Maintainer; Thanos Maintainer
  3. @ThanosMetrics Agenda • Quick intro, reiterate quickly on components •

    StoreAPI ◦ Querier (discovery, fanout, filtering) ◦ Producer vs Browser ◦ Integrations: OpenTSDB • Downsampling • Horizontal Query scaling • Summary
  4. @ThanosMetrics Thanos Community • Fully open source from start •

    Started in Nov 2017 • Part of CNCF Sandbox • 4600+ Github stars • 160+ contributors • ~500 slack users • 8 maintainers, 3 triagers from 7 different companies. • Transparent Governance • Prometheus Ecosystem
  5. @ThanosMetrics Let’s quickly reiterate on Thanos pull Querier Sidecar exposing

    StoreAPI
  6. @ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar StoreAPI

  7. @ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar StoreAPI StoreAPI

    Gateway
  8. @ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar Gateway Ruler

    StoreAPI StoreAPI StoreAPI
  9. @ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar Gateway Ruler

    StoreAPI Federated Querier
  10. @ThanosMetrics There was something common in all these architectures

  11. @ThanosMetrics StoreAPI

  12. @ThanosMetrics StoreAPI • Every component in Thanos serves data via

    gRPC StoreAPI ◦ sidecar ◦ store ◦ rule ◦ receive (experimental component) ◦ query • Integrations! https://thanos.io/integrations.md/ ◦ OpenTSDB as StoreAPI: https://github.com/G-Research/geras
  13. @ThanosMetrics StoreAPI From: rpc.proto

  14. @ThanosMetrics Thanos Query: Store Discovery • --store flag ◦ Exact

    endpoints ◦ DNS discovery: A, AAAA, SRV
  15. @ThanosMetrics Thanos Query: Store Infos • Every 10s requests Info

    endpoint • Healthiness • Metadata propagation
  16. @ThanosMetrics Thanos Query: Life of a query • Query ◦

    Select possible stores ◦ Fan out to gather data ◦ Process query
  17. @ThanosMetrics Thanos Query: Life of a query pull Querier {region=”us-east-1”}

    {region=”us-east-2”} {region=”us-west-1”}
  18. @ThanosMetrics ProxyStore

  19. @ThanosMetrics Challenges of Querying Years of Data

  20. @ThanosMetrics Query Resolution time • Typical scrape period of Prometheus

    is 15s • Querying 30 days means ~170k samples
  21. @ThanosMetrics Query Resolution time Scrape interval = ~15s step =

    1m Evaluation time
  22. @ThanosMetrics Query Resolution: 5h range time time Displayed Storage Step

    1m Samples: ~250 Fetched Samples: ~1k ...
  23. @ThanosMetrics Query Resolution: 30d range time time Displayed Storage Step

    3h Samples: ~250 Fetched Samples: ~170k ...
  24. @ThanosMetrics Chunks time Chunk Chunk Samples are stored in chunks

  25. @ThanosMetrics Chunks time Chunk Chunk 1.3 bytes/sample 16 bytes/sample Samples

    are stored in chunks
  26. @ThanosMetrics Chunk tradeoff Decompressing one sample takes 10-40 nanoseconds

  27. @ThanosMetrics Chunk tradeoff Query Range Samples for 1000 series Decompression

    latency Chunk data size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB Decompressing one sample takes 10-40 nanoseconds
  28. @ThanosMetrics Chunks tradeoff Query Range Samples for 1000 series Decompression

    latency Chunk data size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB 30d ~170 millions ~7s ~240MB Decompressing one sample takes 10-40 nanoseconds
  29. @ThanosMetrics Chunks tradeoff Query Range Samples for 1000 series Decompression

    latency Chunk data size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB 30d ~170 millions ~7s ~240MB 1y ~2 billions ~1m20s ~2GB Decompressing one sample takes 10-40 nanoseconds
  30. @ThanosMetrics Downsampling

  31. @ThanosMetrics Downsampling Block RAW Block @ 5m Block @ 1h

    10-20x 12x
  32. @ThanosMetrics Downsampling chunk count sum min max counter chunk ...

  33. @ThanosMetrics Downsampling count sum min max counter count(requests_total) count_over_time(requests_total[1h])

  34. @ThanosMetrics Downsampling count sum min max counter sum_over_time(requests_total[1h])

  35. @ThanosMetrics Downsampling count sum min max counter min(requests_total) min_over_time(requests_total[1h])

  36. @ThanosMetrics Downsampling count sum min max counter max(requests_total) max_over_time(requests_total[1h])

  37. @ThanosMetrics Downsampling count sum min max counter rate(requests_total[1h]) increase(requests_total[1h])

  38. @ThanosMetrics Downsampling count sum min max counter requests_total avg(requests_total) sum(requests_total)

    avg
  39. @ThanosMetrics Downsampling: What chunk to use on query? range query

    from t0 to t1, step 10s: rate(alerts_total[5m]) PromQL
  40. @ThanosMetrics Downsampling: What chunk to use on query? labels: __name__

    = “alerts_total” time: start: t0-5m end: t1 step: 10s read hints: func: “rate” range query from t0 to t1, step 10s: rate(alerts_total[5m]) PromQL Select
  41. @ThanosMetrics Downsampling: What chunk to use on query? labels: __name__

    = “alerts_total” time: start: t0-5m end: t1 step: 10s read hints: func: “rate” range query from t0 to t1, step 10s: rate(alerts_total[5m]) PromQL Select Fetch raw raw Fetch
  42. @ThanosMetrics Downsampling: What chunk to use on query? labels: __name__

    = “alerts_total” time: start: t0-5m end: t1 step: 30m read hints: func: “rate” range query from t0 to t1, step 30m: rate(alerts_total[1h]) PromQL Select Can we fit 5 samples for this step with lower resolution?
  43. @ThanosMetrics Downsampling: What chunk to use on query? labels: __name__

    = “alerts_total” time: start: t0-5m end: t1 step: 30m read hints: func: “rate” range query from t0 to t1, step 30m: rate(alerts_total[1h]) PromQL Select Fetch counter counter Fetch Can we fit 5 samples for this step with lower resolution? yes for 5m resolution!
  44. @ThanosMetrics Downsampling: What chunk to use on query? labels: __name__

    = “alerts” state = “active” time: start: t0 end: t1 step: 30m read hints: func: “avg” range query from t0 to t1, step 30m: avg(alerts{state=”active}) sum sum PromQL Select Fetch count count
  45. @ThanosMetrics Downsampling Query Range Samples for 1000 series Decompression latency

    Fetched chunks size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB 30d ~170 millions ~7s ~240MB 30d ~8 millions ~300ms ~9MB 1y ~2 billions ~80s ~2GB 1y ~8 millions ~300ms ~9MB 1h resolution [~50d+ queries] 5m resolution [~5d+ queries]
  46. @ThanosMetrics Downsampling: Caveats • Thanos/Prometheus UI: Step (evaluation interval in

    seconds) • Grafana: Resolutions (1/x samples per pixel) • rate[<5m] vs rate[1h] / rate[5h] / rate[$_interval] • Storing only downsampled data and trying to zoom-in
  47. @ThanosMetrics Downsampling: Caveats • Thanos/Prometheus UI: Step (evaluation interval in

    seconds) • Grafana: Resolutions (1/x samples per pixel) • rate[<5m] vs rate[1h] / rate[5h] / rate[$_interval] • Storing only downsampled data and trying to zoom-in Standardize downsampling?
  48. @ThanosMetrics Horizontal Scaling of Long Term Storage Read Path

  49. @ThanosMetrics Querying long term storage backend Querier Gateway

  50. @ThanosMetrics Time partitioning Querier Gateway: --min-time=1y --max-time=150d Gateway: --min-time=150d

  51. @ThanosMetrics Block Sharding Querier Gateway: --selector.relabel-config= - action: keep regex:

    "eu.*" source_labels: - region Gateway: --selector.relabel-config= - action: keep regex: "us.*" source_labels: - region
  52. @ThanosMetrics Block Sharding Querier Gateway: --selector.relabel-config= - action: keep regex:

    "eu.*" source_labels: - region Gateway: --selector.relabel-config= - action: keep regex: "us.*" source_labels: - region
  53. @ThanosMetrics Common StoreAPI Downsampling Horizontal Scaling of Long Term Storage

    Summary
  54. @ThanosMetrics Thank You! https://thanos.io

  55. @ThanosMetrics Bonus: Caching Caching Querier Sidecars Gateway Cortex Frontend

  56. @ThanosMetrics Response Caching: Challenges • Extremely useful for rolling windows

    (e.g Grafana “last 1h”) • Dynamically changing StoreAPIs • Downsampling • Partial Response • Backfilling/Deletion