Tweet
Tweet

Slide 1

Slide 1 text

Blazin’ Fast PromQL Prometheus London Meetup, August 2019 @tom_wilkie

Slide 2

Slide 2 text

Cortex is a time-series store built on Prometheus that is: - Horizontally scalable - Highly Available - Long-term storage - Multi-tenant - Multi-tenant Cortex: horizontally scalable Prometheus 2 Cortex gives you: - A global view of as many metrics as you need - With no gaps in the charts - On durable, long term storage - Across multiple tenants Cortex is a CNCF Sandbox project: github.com/cortexproject/cortex

Slide 3

Slide 3 text

3 Querier PromQL Engine Chunk Store Ingester Client Ingester Ingester Ingester Ingester Ingester >1 yr ago NoSQL Index Blob Store

Slide 4

Slide 4 text

4 Querier PromQL Engine Chunk Store Ingester Client Ingester Ingester Ingester Ingester Ingester Index Memcached Chunk Memcached NoSQL Index Blob Store Caching

Slide 5

Slide 5 text

5 Querier Ingester Ingester Ingester Ingester Ingester Index Memcached Chunk Memcached NoSQL Index Blob Store Query Frontend Results Memcached More Caching

Slide 6

Slide 6 text

6 Query Frontend rate(http_duration_seconds_count{job="shipping"}[1m]) rate... rate... rate... rate... 2. Split by day rate... rate... 3. Cache lookup .. 4. Queue & Parallel Dispatch rate(request_durations_seconds_count[1m]) 1. Step align rate(http_duration_seconds_count{job="shipping"}[1m])

Slide 7

Slide 7 text

But wait! One more thing... 7

Slide 8

Slide 8 text

8 https://github.com/cortexproject/cortex/pull/1441

Slide 9

Slide 9 text

9 $ ./cortex \ -config.file=./docs/prometheus-frontend.yml \ -frontend.downstream-url=http://demo.robustperception.io:9090 ... Try this query over 7 days: histogram_quantile(0.50, sum by (job, le) ( rate(prometheus_http_request_duration_seconds_bucket[1m]) ) )

Slide 10

Slide 10 text

- Start sharding aggregations by series to accelerate high-cardinality queries (design doc). - Automatically replace with recording rules where appropriate? - Embed this as a library in Thanos... - Handle gaps from HA pairs... - What do you want to see? What does the future hold? 10

Slide 11

Slide 11 text

Thank You! 11 @tom_wilkie https://github.com/cortexproject/cortex

Slide 12

Slide 12 text

How do we compare to Trickster? - Reusable: set of HTTP middlewares, useable as a library - Memcached for “external” cache (vs Redis for Trickster) - We are multi-tenant - We split by day and execute in parallel - We have some rudimentary QOS / queueing / scheduling However: - No “Fast Forward” like Trickster - Trickster more widely used 12 https://github.com/Comcast/trickster