Tweet
Tweet

Slide 1

Slide 1 text

@ThanosMetrics Inside a Distributed Monitoring System San Diego, 20th November 2019 Bartłomiej Płotka Frederic Branczyk brancz fredbrancz bwplotka

Slide 2

Slide 2 text

@ThanosMetrics Speakers Frederic Branczyk Principal Software Engineer @ Red Hat; OpenShift Monitoring Team Prometheus Maintainer; Thanos Maintainer; SIG Instrumentation Lead Bartek Plotka Principal Software Engineer @ Red Hat; OpenShift Monitoring Team Prometheus Maintainer; Thanos Maintainer

Slide 3

Slide 3 text

@ThanosMetrics Agenda ● Quick intro, reiterate quickly on components ● StoreAPI ○ Querier (discovery, fanout, filtering) ○ Producer vs Browser ○ Integrations: OpenTSDB ● Downsampling ● Horizontal Query scaling ● Summary

Slide 4

Slide 4 text

@ThanosMetrics Thanos Community ● Fully open source from start ● Started in Nov 2017 ● Part of CNCF Sandbox ● 4600+ Github stars ● 160+ contributors ● ~500 slack users ● 8 maintainers, 3 triagers from 7 different companies. ● Transparent Governance ● Prometheus Ecosystem

Slide 5

Slide 5 text

@ThanosMetrics Let’s quickly reiterate on Thanos pull Querier Sidecar exposing StoreAPI

Slide 6

Slide 6 text

@ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar StoreAPI

Slide 7

Slide 7 text

@ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar StoreAPI StoreAPI Gateway

Slide 8

Slide 8 text

@ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar Gateway Ruler StoreAPI StoreAPI StoreAPI

Slide 9

Slide 9 text

@ThanosMetrics Let’s quickly reiterate on Thanos Querier Sidecar Gateway Ruler StoreAPI Federated Querier

Slide 10

Slide 10 text

@ThanosMetrics There was something common in all these architectures

Slide 11

Slide 11 text

@ThanosMetrics StoreAPI

Slide 12

Slide 12 text

@ThanosMetrics StoreAPI ● Every component in Thanos serves data via gRPC StoreAPI ○ sidecar ○ store ○ rule ○ receive (experimental component) ○ query ● Integrations! https://thanos.io/integrations.md/ ○ OpenTSDB as StoreAPI: https://github.com/G-Research/geras

Slide 13

Slide 13 text

@ThanosMetrics StoreAPI From: rpc.proto

Slide 14

Slide 14 text

@ThanosMetrics Thanos Query: Store Discovery ● --store flag ○ Exact endpoints ○ DNS discovery: A, AAAA, SRV

Slide 15

Slide 15 text

@ThanosMetrics Thanos Query: Store Infos ● Every 10s requests Info endpoint ● Healthiness ● Metadata propagation

Slide 16

Slide 16 text

@ThanosMetrics Thanos Query: Life of a query ● Query ○ Select possible stores ○ Fan out to gather data ○ Process query

Slide 17

Slide 17 text

@ThanosMetrics Thanos Query: Life of a query pull Querier {region=”us-east-1”} {region=”us-east-2”} {region=”us-west-1”}

Slide 18

Slide 18 text

@ThanosMetrics ProxyStore

Slide 19

Slide 19 text

@ThanosMetrics Challenges of Querying Years of Data

Slide 20

Slide 20 text

@ThanosMetrics Query Resolution time ● Typical scrape period of Prometheus is 15s ● Querying 30 days means ~170k samples

Slide 21

Slide 21 text

@ThanosMetrics Query Resolution time Scrape interval = ~15s step = 1m Evaluation time

Slide 22

Slide 22 text

@ThanosMetrics Query Resolution: 5h range time time Displayed Storage Step 1m Samples: ~250 Fetched Samples: ~1k ...

Slide 23

Slide 23 text

@ThanosMetrics Query Resolution: 30d range time time Displayed Storage Step 3h Samples: ~250 Fetched Samples: ~170k ...

Slide 24

Slide 24 text

@ThanosMetrics Chunks time Chunk Chunk Samples are stored in chunks

Slide 25

Slide 25 text

@ThanosMetrics Chunks time Chunk Chunk 1.3 bytes/sample 16 bytes/sample Samples are stored in chunks

Slide 26

Slide 26 text

@ThanosMetrics Chunk tradeoff Decompressing one sample takes 10-40 nanoseconds

Slide 27

Slide 27 text

@ThanosMetrics Chunk tradeoff Query Range Samples for 1000 series Decompression latency Chunk data size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB Decompressing one sample takes 10-40 nanoseconds

Slide 28

Slide 28 text

@ThanosMetrics Chunks tradeoff Query Range Samples for 1000 series Decompression latency Chunk data size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB 30d ~170 millions ~7s ~240MB Decompressing one sample takes 10-40 nanoseconds

Slide 29

Slide 29 text

@ThanosMetrics Chunks tradeoff Query Range Samples for 1000 series Decompression latency Chunk data size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB 30d ~170 millions ~7s ~240MB 1y ~2 billions ~1m20s ~2GB Decompressing one sample takes 10-40 nanoseconds

Slide 30

Slide 30 text

@ThanosMetrics Downsampling

Slide 31

Slide 31 text

@ThanosMetrics Downsampling Block RAW Block @ 5m Block @ 1h 10-20x 12x

Slide 32

Slide 32 text

@ThanosMetrics Downsampling chunk count sum min max counter chunk ...

Slide 33

Slide 33 text

@ThanosMetrics Downsampling count sum min max counter count(requests_total) count_over_time(requests_total[1h])

Slide 34

Slide 34 text

@ThanosMetrics Downsampling count sum min max counter sum_over_time(requests_total[1h])

Slide 35

Slide 35 text

@ThanosMetrics Downsampling count sum min max counter min(requests_total) min_over_time(requests_total[1h])

Slide 36

Slide 36 text

@ThanosMetrics Downsampling count sum min max counter max(requests_total) max_over_time(requests_total[1h])

Slide 37

Slide 37 text

@ThanosMetrics Downsampling count sum min max counter rate(requests_total[1h]) increase(requests_total[1h])

Slide 38

Slide 38 text

@ThanosMetrics Downsampling count sum min max counter requests_total avg(requests_total) sum(requests_total) avg

Slide 39

Slide 39 text

@ThanosMetrics Downsampling: What chunk to use on query? range query from t0 to t1, step 10s: rate(alerts_total[5m]) PromQL

Slide 40

Slide 40 text

@ThanosMetrics Downsampling: What chunk to use on query? labels: __name__ = “alerts_total” time: start: t0-5m end: t1 step: 10s read hints: func: “rate” range query from t0 to t1, step 10s: rate(alerts_total[5m]) PromQL Select

Slide 41

Slide 41 text

@ThanosMetrics Downsampling: What chunk to use on query? labels: __name__ = “alerts_total” time: start: t0-5m end: t1 step: 10s read hints: func: “rate” range query from t0 to t1, step 10s: rate(alerts_total[5m]) PromQL Select Fetch raw raw Fetch

Slide 42

Slide 42 text

@ThanosMetrics Downsampling: What chunk to use on query? labels: __name__ = “alerts_total” time: start: t0-5m end: t1 step: 30m read hints: func: “rate” range query from t0 to t1, step 30m: rate(alerts_total[1h]) PromQL Select Can we fit 5 samples for this step with lower resolution?

Slide 43

Slide 43 text

@ThanosMetrics Downsampling: What chunk to use on query? labels: __name__ = “alerts_total” time: start: t0-5m end: t1 step: 30m read hints: func: “rate” range query from t0 to t1, step 30m: rate(alerts_total[1h]) PromQL Select Fetch counter counter Fetch Can we fit 5 samples for this step with lower resolution? yes for 5m resolution!

Slide 44

Slide 44 text

@ThanosMetrics Downsampling: What chunk to use on query? labels: __name__ = “alerts” state = “active” time: start: t0 end: t1 step: 30m read hints: func: “avg” range query from t0 to t1, step 30m: avg(alerts{state=”active}) sum sum PromQL Select Fetch count count

Slide 45

Slide 45 text

@ThanosMetrics Downsampling Query Range Samples for 1000 series Decompression latency Fetched chunks size 30m ~120 000 ~5ms ~160KB 1d ~6 millions ~240ms ~8MB 30d ~170 millions ~7s ~240MB 30d ~8 millions ~300ms ~9MB 1y ~2 billions ~80s ~2GB 1y ~8 millions ~300ms ~9MB 1h resolution [~50d+ queries] 5m resolution [~5d+ queries]

Slide 46

Slide 46 text

@ThanosMetrics Downsampling: Caveats ● Thanos/Prometheus UI: Step (evaluation interval in seconds) ● Grafana: Resolutions (1/x samples per pixel) ● rate[<5m] vs rate[1h] / rate[5h] / rate[$_interval] ● Storing only downsampled data and trying to zoom-in

Slide 47

Slide 47 text

@ThanosMetrics Downsampling: Caveats ● Thanos/Prometheus UI: Step (evaluation interval in seconds) ● Grafana: Resolutions (1/x samples per pixel) ● rate[<5m] vs rate[1h] / rate[5h] / rate[$_interval] ● Storing only downsampled data and trying to zoom-in Standardize downsampling?

Slide 48

Slide 48 text

@ThanosMetrics Horizontal Scaling of Long Term Storage Read Path

Slide 49

Slide 49 text

@ThanosMetrics Querying long term storage backend Querier Gateway

Slide 50

Slide 50 text

@ThanosMetrics Time partitioning Querier Gateway: --min-time=1y --max-time=150d Gateway: --min-time=150d

Slide 51

Slide 51 text

@ThanosMetrics Block Sharding Querier Gateway: --selector.relabel-config= - action: keep regex: "eu.*" source_labels: - region Gateway: --selector.relabel-config= - action: keep regex: "us.*" source_labels: - region

Slide 52

Slide 52 text

@ThanosMetrics Block Sharding Querier Gateway: --selector.relabel-config= - action: keep regex: "eu.*" source_labels: - region Gateway: --selector.relabel-config= - action: keep regex: "us.*" source_labels: - region

Slide 53

Slide 53 text

@ThanosMetrics Common StoreAPI Downsampling Horizontal Scaling of Long Term Storage Summary

Slide 54

Slide 54 text

@ThanosMetrics Thank You! https://thanos.io

Slide 55

Slide 55 text

@ThanosMetrics Bonus: Caching Caching Querier Sidecars Gateway Cortex Frontend

Slide 56

Slide 56 text

@ThanosMetrics Response Caching: Challenges ● Extremely useful for rolling windows (e.g Grafana “last 1h”) ● Dynamically changing StoreAPIs ● Downsampling ● Partial Response ● Backfilling/Deletion