Building a Scalable Event Service with Cassandra: Design to Code

Cassandra Summit Europe 2014

Transcript

1. Building a Scalable Event Service with Cassandra Design to Code David Borsos & Tareq Abedrabbo Cassandra Summit Europe 2014
2. About Us David Borsos Senior Consultant at OpenCredo Tareq Abedrabbo CTO at OpenCredo
3. This talk is about… ‣ What we built ‣ Why we built it ‣ How we built it
4. Project background
5. • High street retailer • Microservices, event-driven architecture • Java, Cassandra, Cloud Foundry, RabbitMQ
7. Why do we need an event service?
8. ✓ Capture millions of platform and business events ✓ Trigger downstream processes asynchronously ✓ Customise standard processes in a non-intrusive way ✓ Provide a system-wide transaction log ✓ Analytics ✓ Auditing ✓ System testing
9. However… - Ambiguous requirements - New paradigm and emerging architecture - We need to look at the problem as a whole - We need to avoid building useless features - We need to avoid accumulating technical debt
10. Design principles
11. Simplicity
12. Decoupling
13. Design for a distributed system
14. What is an event?
15. • A simple event is an opaque value, typically a time series item • meter reading • A structured event can have an arbitrarily complex structure that can evolve over time • user registration event
16. Simplify the data model
17. Evolution of Event • Payload and meta-data as simple collections of key/value • The type is persisted with each event • to make events readable • to avoid managing schemas
18. What does the event store look like?
19. An event store should be - Simple request/response paradigm with clear guarantees - Accessible, ideally even from legacy services - Ability to query for events
20. Event Store ☛ Event Service
21. Resource-driven Design
22. Event service API, version 1: store and read an event
23. • Store an event • POST /api/events/ • Read an event • GET /api/events/{eventId}
24. Anatomy of an Event { "type" : "SOME.EVENT.TYPE", "source" : "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } }
25. and the architecture to support the requirements…
27. The Event Table Key Event id1 timestamp type payload … 123 X <<blob>> … id2 timestamp type payload … 456 Y <<blob>> …
28. Event service API, version 2: querying events and notifications
29. • Query events • GET /api/events?{parameters} • {queryString} can consist of the following fields: • start, end, startOffset, limit, tag, type, order
30. • Examples: • GET /api/events?start={startTime} &end={endTime} • GET /api/events? startOffset=3600000&type=someType
31. Modelling time series and queries
32. Querying One denormalised table for each query Query Key id1 ts11 id1 ts12 id2 ts21 id2 ts22 p1 b1 v11 p2 b2 v12 v21 p2 b2 v22
33. • Cons: • Denormalise for each query, again and again • Higher disk usage • Disk space is cheap, but not free • Write latency is affected • Time-bucketed indexes can create hot spots (hot shards)
34. Flexible, adaptable architecture
35. Adapt the data model to real-world constraints
37. • Same service contract • Indices are updated synchronously or asynchronously • Basic client guarantee: if a POST is successful the event has been persisted “sufficiently” • Events can be published to a message broker
38. • Pro • Decoupling • client are unaware of the implementation details • Intuitive ReSTful interface • Disk consumption is more reasonable • Easily extensible • Pub/sub
39. • Cons • Not primarily optimised for latency • Still sufficiently performant for our use-cases • More complex service code • Needs to execute multiple CQL queries in sequence • Cluster hotspots can still occur, in theory
40. Indices • Ascending and descending time buckets for each query type • Index value references an event stored in the main table by its id
41. Indices events_by_type_asc ( tbucket text, type text, eventid timeuuid, primary key ((type, tbucket), eventid)) with clustering order by (eventid asc); events_by_type_desc ( tbucket text, type text, eventid timeuuid, primary key ((type, tbucket), eventid)) with clustering order by (eventid desc); Ascending and descending time buckets for each query type
42. Example: Implementing Pagination
43. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
44. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
45. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎ type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
46. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎ type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
47. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎ type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
48. GET /api/events?start=141..&type=X&limit=5 { “events” : [ { “id” : “uuid1”, “type” : “X”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?continueFrom=uuid7&type=X&limit=5” }
49. Performance Characteristics • 70 to 85 million events per day • Client latency increases moderately with increased parallel load (40ms to 60ms, +10ms on the client) • Current behaviour exceeds by far current target volumes
50. Lessons learnt • Scalability is not only about raw performance and latency • Experiment • Simplify • Understand Thrift, use CQL
51. Links • http://www.opencredo.com/blog • @davib0 • @tareq_abedrabbo Thank you! Any questions?
52. Future improvements
53. • Data model improvements: User Defined Types • DateTiered compaction • Analytics with Spark • Add other data views