(Presented at London Prometheus Meetup 20/09/2019)
PromQL, the Prometheus Query Language, is a concise, powerful and increasingly popular language for querying time series data. But PromQL queries can take a long time when they have to consider >100k series and months of data. Even with Prometheus’ compression, a 90-day query over 200k series can touch ~100GB of data.
In this talk, we will present a series of techniques employed by Cortex (a CNCF project for clustered Prometheus) for accelerating PromQL queries - namely query results caching, time slice parallelisation, aggregation sharding, and automatic recoding rule substitutions.
But there’s more: we will show how you can use this technology to get these improvements with Thanos and Prometheus.
Blazin’ Fast PromQL
Prometheus London Meetup, August 2019
Cortex is a time-series store built on
Prometheus that is:
- Horizontally scalable
- Highly Available
- Long-term storage
Cortex: horizontally scalable Prometheus
Cortex gives you:
- A global view of as many
metrics as you need
- With no gaps in the charts
- On durable, long term storage
- Across multiple tenants
Cortex is a CNCF Sandbox project: github.com/cortexproject/cortex
>1 yr ago
Query Frontend Results
rate... rate... rate... rate...
2. Split by day
3. Cache lookup
4. Queue & Parallel Dispatch
1. Step align
But wait! One more thing...
$ ./cortex \
Try this query over 7 days:
sum by (job, le) (
- Start sharding aggregations by series to accelerate
high-cardinality queries (design doc).
- Automatically replace with recording rules where appropriate?
- Embed this as a library in Thanos...
- Handle gaps from HA pairs...
- What do you want to see?
What does the future hold?
How do we compare to Trickster?
- Reusable: set of HTTP middlewares, useable as a library
- Memcached for “external” cache (vs Redis for Trickster)
- We are multi-tenant
- We split by day and execute in parallel
- We have some rudimentary QOS / queueing / scheduling
- No “Fast Forward” like Trickster
- Trickster more widely used