Building a Scalable Event Service with Cassandra: Design to Code

Building a Scalable Event Service with Cassandra: Design to Code

Cassandra Summit Europe 2014

76bd3a3821f3bf531c2eeb445a04cbf3?s=128

Tareq Abedrabbo

December 04, 2014
Tweet

Transcript

  1. 1.

    Building a Scalable Event Service with Cassandra Design to Code

    David Borsos & Tareq Abedrabbo Cassandra Summit Europe 2014
  2. 3.

    This talk is about… ‣ What we built ‣ Why

    we built it ‣ How we built it
  3. 6.
  4. 8.

    ✓ Capture millions of platform and business events ✓ Trigger

    downstream processes asynchronously ✓ Customise standard processes in a non-intrusive way ✓ Provide a system-wide transaction log ✓ Analytics ✓ Auditing ✓ System testing
  5. 9.

    However… - Ambiguous requirements - New paradigm and emerging architecture

    - We need to look at the problem as a whole - We need to avoid building useless features - We need to avoid accumulating technical debt
  6. 15.

    • A simple event is an opaque value, typically a

    time series item • meter reading • A structured event can have an arbitrarily complex structure that can evolve over time • user registration event
  7. 17.

    Evolution of Event • Payload and meta-data as simple collections

    of key/value • The type is persisted with each event • to make events readable • to avoid managing schemas
  8. 19.

    An event store should be - Simple request/response paradigm with

    clear guarantees - Accessible, ideally even from legacy services - Ability to query for events
  9. 23.

    • Store an event • POST /api/events/ • Read an

    event • GET /api/events/{eventId}
  10. 24.

    Anatomy of an Event { "type" : "SOME.EVENT.TYPE", "source" :

    "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } }
  11. 26.
  12. 27.

    The Event Table Key Event id1 timestamp type payload …

    123 X <<blob>> … id2 timestamp type payload … 456 Y <<blob>> …
  13. 29.

    • Query events • GET /api/events?{parameters} • {queryString} can consist

    of the following fields: • start, end, startOffset, limit, tag, type, order
  14. 32.

    Querying One denormalised table for each query Query Key id1

    ts11 id1 ts12 id2 ts21 id2 ts22 p1 b1 v11 p2 b2 v12 v21 p2 b2 v22
  15. 33.

    • Cons: • Denormalise for each query, again and again

    • Higher disk usage • Disk space is cheap, but not free • Write latency is affected • Time-bucketed indexes can create hot spots (hot shards)
  16. 36.
  17. 37.

    • Same service contract • Indices are updated synchronously or

    asynchronously • Basic client guarantee: if a POST is successful the event has been persisted “sufficiently” • Events can be published to a message broker
  18. 38.

    • Pro • Decoupling • client are unaware of the

    implementation details • Intuitive ReSTful interface • Disk consumption is more reasonable • Easily extensible • Pub/sub
  19. 39.

    • Cons • Not primarily optimised for latency • Still

    sufficiently performant for our use-cases • More complex service code • Needs to execute multiple CQL queries in sequence • Cluster hotspots can still occur, in theory
  20. 40.

    Indices • Ascending and descending time buckets for each query

    type • Index value references an event stored in the main table by its id
  21. 41.

    Indices events_by_type_asc ( tbucket text, type text, eventid timeuuid, primary

    key ((type, tbucket), eventid)) with clustering order by (eventid asc); events_by_type_desc ( tbucket text, type text, eventid timeuuid, primary key ((type, tbucket), eventid)) with clustering order by (eventid desc); Ascending and descending time buckets for each query type
  22. 43.

    Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 type1 bucket2

    id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  23. 44.

    Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 ▲ query

    range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  24. 45.

    Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎

    type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  25. 46.

    Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎

    type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  26. 47.

    Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎

    type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  27. 48.

    GET /api/events?start=141..&type=X&limit=5 { “events” : [ { “id” : “uuid1”,

    “type” : “X”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?continueFrom=uuid7&type=X&limit=5” }
  28. 49.

    Performance Characteristics • 70 to 85 million events per day

    • Client latency increases moderately with increased parallel load (40ms to 60ms, +10ms on the client) • Current behaviour exceeds by far current target volumes
  29. 50.

    Lessons learnt • Scalability is not only about raw performance

    and latency • Experiment • Simplify • Understand Thrift, use CQL
  30. 53.

    • Data model improvements: User Defined Types • DateTiered compaction

    • Analytics with Spark • Add other data views