Performance Metrics and Cassandra

This details why Cassandra is a great fit for storing performance metrics. It briefly describes a suggested schema and gives advice on aggregating metrics.

This talk was presented for the Big Data Benchmarking Community (BDBC) on November 29th, 2012: http://clds.ucsd.edu/bdbc/community

Transcript

1. ©2012 DataStax 1 Metrics and Cassandra Tyler Hobbs tyler@datastax.com
2. ©2012 DataStax 2 • Problems with alternatives • Particular advantages of Cassandra Why store metrics in Cassandra?
3. ©2012 DataStax 3 • Random writes, random reads • Write amplification on SSDs • Lock contention • Availability • Difficulty of horizontal scaling RDBMS Issues
4. ©2012 DataStax 4 • Same problems as RDBMS, lower throughput RRDTool
5. ©2012 DataStax 5 • Same data model as Cassandra • Master/Slave vs Fully Distributed OpenTSDB
6. ©2012 DataStax 6 • Open Source • Fully Distributed • Non-Relational • Log-structured Merge-Tree About Apache Cassandra
7. ©2012 DataStax 7 • Sequential writes • Mostly sequential reads • Supports high parallelism • Partitions data automatically, no distributed joins • Block-based compression Cassandra does Time-Series Data Very Well
8. ©2012 DataStax 8 • ~30k writes/sec per node • Linear scalability • (see Netflix's 1,000,000 writes/sec benchmark) Cassandra does Time-Series Data Very Well
9. ©2012 DataStax 9 Cassandra Schema CREATE TABLE metrics ( metric_id text, time timestamp, value float, PRIMARY KEY (metric_id, time) ) WITH CLUSTERING ORDER BY (time DESC);
10. ©2012 DataStax 10 Cassandra Schema CREATE TABLE metrics ( metric_id text, time timestamp, value float, PRIMARY KEY (metric_id, time) ) WITH CLUSTERING ORDER BY (time DESC); Partition Key Clustering Key
11. ©2012 DataStax 11 Cassandra Schema INSERT INTO metrics (metric_id, time, value) VALUES ('node12­load', now(), 1.2);
12. ©2012 DataStax 12 Cassandra Schema SELECT time, value FROM metrics WHERE metric_id = 'node12­load' AND time > '2012­11­28';
13. ©2012 DataStax 13 • Write Time • Read Time Metric Aggregation
14. ©2012 DataStax 14 • Entirely Optional • Good for reducing total volume of data stored Write-Time Aggregation
15. ©2012 DataStax 15 • Primarily useful for rolling up a single metric at different time granularities • 1 min avg, 3 hour max, etc • Use a strategy similar to RRDTool in memory Write-Time Aggregation
16. ©2012 DataStax 16 • Complex analysis of data • Read individual metrics, combine client-side • Potentially store the results Read-Time Aggregation
17. ©2012 DataStax 17 • Simple API, few hidden costs • Primarily benchmarking reads • Writes always have the same cost • Reads depend on data model quality, caching, and disk seek times Benchmarking Cassandra
18. ©2012 DataStax 18 • Primary benchmarking mistake? • Not enough client-side threads/processes. Benchmarking Cassandra
19. ©2012 DataStax 19 Questions? Tyler Hobbs tyler@datastax.com