CloudWatch Read Adapter - A selective archive for CloudWatch metrics

Transcript

1. CloudWatch Read Adapter A selective archive for CloudWatch metrics Mitsuhiro Tanda (@mtanda) Grafana Labs PromCon EU 2026, Lightning Talk
2. CLOUDWATCH READ ADAPTER A small remote read adapter Uses Prometheus’s own remote read protocol Lets Prometheus / Grafana query CloudWatch metrics with PromQL Keeps high-precision data around longer than CloudWatch itself does Built in 2017, and it ran in production for several years PromQL Grafana remote read Prometheus CloudWatch Read Adapter GetMetricStatistics Amazon CloudWatch Grafana asks Prometheus → Prometheus asks the adapter → the adapter asks CloudWatch
3. Why I built it
4. WHY I BUILT IT Copying everything costs too much PromQL Grafana scrape Exporter poll every metric, every 1 minute Prometheus CloudWatch I wanted to keep all metrics, not a hand-picked subset API calls scale with every metric, every minute, and the cost increases accordingly CloudWatch only keeps high-precision data for about 2 weeks, so without copying it in time, I’d lose it forever
5. WHY I BUILT IT Not every metric needs to be live Rarely queried Frequently queried no real-time need alerts & dashboards ↓ ARCHIVING IS ENOUGH ↓ CAN'T WAIT FOR THAT Served once archived Also needs to be live a little delay is fine live, before it's archived every metric gets archived once it's old enough, and the question is whether that's enough on its own
6. How it works
7. HOW IT WORKS One query, split by range already archived remote read query Archived data CloudWatch Read Adapter Prometheus not archived yet CloudWatch Data that’s already archived is served fast and free Data that’s too new to be archived yet is read live from CloudWatch, every time
8. HOW IT WORKS API cost went down a lot Before: poll every metric, every 1 minute 1,000 metrics × 43,200 calls/metric/month = 43,200,000 calls/month → $432/month After: poll every 6 hours instead 1,000 metrics × 120 calls/metric/month = 120,000 calls/month → $1.20/month → 1/360th the cost archive path only, at $0.01 per 1,000 CloudWatch API calls
9. What didn’t go well
10. WHAT DIDN'T GO WELL Maintenance cost v2 SD K v1 to ts db AW S ts db do w ng ra de up gr ad e .. . main I embedded Prometheus’s own TSDB, probably more than it was built for, since it’s meant to evolve with Prometheus, not as a standalone library One upgrade broke things badly enough to force a downgrade, a real maintenance risk for a company relying on this
11. LESSONS LEARNED What I learned Remote read turned out to be a great way to extend what Prometheus can actually reach Using it well, I could cut AWS cost by a huge amount But embedding a TSDB like that takes real internal knowledge
12. Thank you! Mitsuhiro Tanda (@mtanda) Grafana Labs github.com/mtanda/cloudwatch_read_adapter