Time Series and Events: Storage and Querying Strategies with Cassandra

Time Series and Events: Storage and Querying Strategies with Cassandra

Cassandra IoT meetup, London

76bd3a3821f3bf531c2eeb445a04cbf3?s=128

Tareq Abedrabbo

October 23, 2014
Tweet

Transcript

  1. 1.

    Time Series and Events: Storage and Querying Strategies David Borsos

    & Tareq Abedrabbo C* London Meetup, 23 October 2014
  2. 2.

    About Us - Software consultancy and delivery company - We’ve

    been building systems with Cassandra for 3 years - DataStax Solutions Partner
  3. 3.

    Events and Time Series - Data streams (events, measurements) from

    a variety of sources - Use data effectively i.e. querying and analytics - Events can be more complex than simple measurements - Event based architecture - a natural fit
  4. 4.

    Essential C* concepts - Assuming CQL - Partition - Partition

    key - Clustering column - (Primary key = partition key + clustering columns)
  5. 5.

    Simple Time Series Modeling Simplest case: using timestamps as clustering

    columns key timestamps / values id1 ts11 ts12 ts13 ts14 ts15 v11 v12 v13 v14 v15 id2 ts21 ts22 ts23 ts24 ts25 v21 v22 v23 v24 v25
  6. 6.

    Pros - Simple - Works well for simple data structures

    - E.g. measurements - Good read and write performance
  7. 7.

    Cons - Hard limit on partition size (2 billion cells)

    - Limited flexibility - Limited querying
  8. 8.

    Time bucketing Adding a time bucket to the partition key

    key timestamps / values ts11 ts12 ts13 ts14 ts15 id1 bucket1 v11 v12 id1 bucket2 v12 v14 v15 ts21 ts22 ts23 ts24 ts25 id2 bucket1 v21 v22 id2 bucket2 v23 v25 v25
  9. 9.

    Time bucketing - Mitigates the partition size issue - Queries

    become slightly more complicated - Write performance is not affected - Reads may be slower - Potentially hitting multiple buckets
  10. 10.

    Denormalise for each new query query key id1 ts11 id1

    ts12 id2 ts21 id2 ts22 qp1 bucket1 v11 qp2 bucket1 v12 v21 qp2 bucket2 v22
  11. 11.

    Denormalise for each new query query key id1 ts11 id1

    ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22
  12. 12.

    Denormalise for each new query query key id1 ts11 id1

    ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22
  13. 13.

    Denormalise for each new query query key id1 ts11 id1

    ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22
  14. 14.

    Denormalise for each new query query key id1 ts11 id1

    ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22 query key id1 ts11 id1 ts12 id2 ts21 id2 ts22 qp1 b1 v11 qp2 b1 v12 v21 qp2 b2 v22
  15. 15.

    Denormalise for each new query - Allow richer queries -

    Higher disk usage - Disk space is cheap, but not free - Write latency may be impacted - Writing to the denormalised indexes - Time-bucket indexes may create hotspots - “Hot shards”
  16. 17.

    Motivation - Capture (more) structured events - Support complex querying

    - Support analytics - Changes in the data model - Allow for extension in the architecture - Guarantees vs. performance - Consistency vs. latency - Should be tunable
  17. 18.

    Go ReSTful - Event Store Event Service - Simple HTTP

    / ReST interface - Create events: POST - Query events: GET with parameters - Clear service contract, easy to use - Accessible anywhere you have http! - Decouples interface from implementation
  18. 19.

    Go ReSTful - create an event POST /api/events { "type"

    : "SOME.EVENT.TYPE", "source" : "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } }
  19. 20.

    Go ReSTful - read a single event GET /api/events/{id} {

    "type" : "SOME.EVENT.TYPE", "source" : "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } }
  20. 21.

    Go ReSTful - read a single event GET /api/events/{id} {

    "type" : "SOME.EVENT.TYPE", "source" : "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } } Not extremely useful
  21. 22.

    Go ReSTful - query events GET /api/events?<parameters> - start absolute

    start time (range) - startOffset X milliseconds ago (range) - end absolute end time (range) - limit number of items returned - order ASC or DESC - tag indexed values - type indexed values
  22. 23.

    Go ReSTful - query events GET /api/events?start=141..&type=X&limit=10 { “events” :

    [ { “type” : “SOME.EVENT.TYPE”, “source: “some-component:instance”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?...” }
  23. 24.

    Sync/async indexing ReST API Write AMQP Primary event store Index1

    / asc Index1 / desc Index2 / asc Index2 / desc Publish IndexN / desc IndexN / asc Write Write Write …. Async
  24. 25.

    Data model: Main Event Table create table events ( id

    timeuuid primary key, source text, type text, meta_t map<text, text>, meta_v map<text, blob>, payload_t map<text, text>, payload_v map<text, blob> );
  25. 26.

    Data model: Query Indexes events_by_type_asc ( tbucket text, type text,

    eventid timeuuid, primary key ((type, tbucket), eventid) ) with clustering order by (eventid asc);
  26. 27.

    Data model: Query Indexes events_by_type_asc ( tbucket text, type text,

    eventid timeuuid, primary key ((type, tbucket), eventid) ) with clustering order by (eventid asc); - No actual values, timeuuid of event is enough - There are multiple index tables; one for each query type - Ascending/descending versions
  27. 28.

    Pros - Client is unaware of the implementation - We

    could use MySQL or MongoDB (or txt files) - Client only needs to be able to - Understand JSON data structures - Follow URIs - GET parameters work in an intuitive way - Disk consumption is reasonable - Extensible - More than just a data store - Driving a pub/sub mechanism on top of events
  28. 29.

    Cons - Not optimized for latency only - Don’t always

    need that - Still quick enough for our use-cases - More complex service code - Needs to execute multiple CQL queries in sequence - Using CQL batches when possible - Cluster hotspots can still occur
  29. 30.

    Implementing pagination Want to query - Events - From a

    specified start time - Until now - Of type X - In ascending order - Each “page” to return 5 items - With a continuation URL to iterate
  30. 31.

    Implementing pagination Want to query - Events - From a

    specified start time - Until now (*) - Of type X - In ascending order (*) - Each “page” to return 5 items - With a continuation URL to iterate GET /api/events?start=141..&type=X&limit=5
  31. 32.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 i_type time → type1 bucket1 type1

    bucket2 uuid1 uuid2 type1 bucket3 uuid3 uuid4 type1 bucket4 uuid5 uuid6 uuid7 uuid8 type1 bucket5
  32. 33.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 i_type time → type1 bucket1 ↑

    query range ↓ type1 bucket2 uuid1 uuid2 type1 bucket3 uuid3 uuid4 type1 bucket4 uuid5 uuid6 uuid7 uuid8 type1 bucket5
  33. 34.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 i_type time → ← query range

    → type1 bucket1 ↑ query range ↓ type1 bucket2 uuid1 uuid2 type1 bucket3 uuid3 uuid4 type1 bucket4 uuid5 uuid6 uuid7 uuid8 type1 bucket5
  34. 35.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 i_type time → ← query range

    → type1 bucket1 ↑ query range ↓ type1 bucket2 uuid1 uuid2 type1 bucket3 uuid3 uuid4 type1 bucket4 uuid5 uuid6 uuid7 uuid8 type1 bucket5
  35. 36.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 i_type time → ← query range

    → type1 bucket1 ↑ query range ↓ type1 bucket2 uuid1 uuid2 type1 bucket3 uuid3 uuid4 type1 bucket4 uuid5 uuid6 uuid7 uuid8 type1 bucket5 Lookup results in primary event table Continuation URL
  36. 37.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 { “events” : [ { “id”

    : “uuid1”, “type” : “X”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?continueFrom=uuid7&type=X&limit=5” }
  37. 38.

    Implementing pagination GET /api/events?start=141..&type=X&limit=5 { “events” : [ { “id”

    : “uuid1”, “type” : “X”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?continueFrom=uuid7&type=X&limit=5” }
  38. 39.

    Future improvements - User defined types - Solve cluster hotspots

    for bucket indexes - Solr integration - Hadoop or Spark for analytics - Testing with Stubbed Cassandra
  39. 40.

    Lessons learnt - Scalability is not only about raw performance

    - Experiment to determine the best fit for your use case - Simplify when you can - Understand Thrift, use CQL
  40. 42.

    Pushing the data model further - Still prone for cluster

    hotspots (buckets) - Sharding within a time bucket - Pay with read-time latency - Writes are not affected
  41. 43.

    Sharded/bucketed index events_by_type_asc ( tbucket text, shard text, type text,

    eventid timeuuid, primary key ((type, shard, tbucket), eventid) ) with clustering order by (eventid asc); where shard = eventid % n