Building a Scalable Event Service with Cassandra: Design to Code

Slide 1

Slide 1 text

Building a Scalable Event Service with Cassandra Design to Code David Borsos & Tareq Abedrabbo Cassandra Summit Europe 2014

Slide 2

Slide 2 text

About Us David Borsos Senior Consultant at OpenCredo Tareq Abedrabbo CTO at OpenCredo

Slide 3

Slide 3 text

This talk is about… ‣ What we built ‣ Why we built it ‣ How we built it

Slide 4

Slide 4 text

Project background

Slide 5

Slide 5 text

• High street retailer • Microservices, event-driven architecture • Java, Cassandra, Cloud Foundry, RabbitMQ

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Why do we need an event service?

Slide 8

Slide 8 text

✓ Capture millions of platform and business events ✓ Trigger downstream processes asynchronously ✓ Customise standard processes in a non-intrusive way ✓ Provide a system-wide transaction log ✓ Analytics ✓ Auditing ✓ System testing

Slide 9

Slide 9 text

However… - Ambiguous requirements - New paradigm and emerging architecture - We need to look at the problem as a whole - We need to avoid building useless features - We need to avoid accumulating technical debt

Slide 10

Slide 10 text

Design principles

Slide 11

Slide 11 text

Simplicity

Slide 12

Slide 12 text

Decoupling

Slide 13

Slide 13 text

Design for a distributed system

Slide 14

Slide 14 text

What is an event?

Slide 15

Slide 15 text

• A simple event is an opaque value, typically a time series item • meter reading • A structured event can have an arbitrarily complex structure that can evolve over time • user registration event

Slide 16

Slide 16 text

Simplify the data model

Slide 17

Slide 17 text

Evolution of Event • Payload and meta-data as simple collections of key/value • The type is persisted with each event • to make events readable • to avoid managing schemas

Slide 18

Slide 18 text

What does the event store look like?

Slide 19

Slide 19 text

An event store should be - Simple request/response paradigm with clear guarantees - Accessible, ideally even from legacy services - Ability to query for events

Slide 20

Slide 20 text

Event Store ☛ Event Service

Slide 21

Slide 21 text

Resource-driven Design

Slide 22

Slide 22 text

Event service API, version 1: store and read an event

Slide 23

Slide 23 text

• Store an event • POST /api/events/ • Read an event • GET /api/events/{eventId}

Slide 24

Slide 24 text

Anatomy of an Event { "type" : "SOME.EVENT.TYPE", "source" : "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } }

Slide 25

Slide 25 text

and the architecture to support the requirements…

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

The Event Table Key Event id1 timestamp type payload … 123 X <> … id2 timestamp type payload … 456 Y <> …

Slide 28

Slide 28 text

Event service API, version 2: querying events and notiﬁcations

Slide 29

Slide 29 text

• Query events • GET /api/events?{parameters} • {queryString} can consist of the following ﬁelds: • start, end, startOffset, limit, tag, type, order

Slide 30

Slide 30 text

• Examples: • GET /api/events?start={startTime} &end={endTime} • GET /api/events? startOffset=3600000&type=someType

Slide 31

Slide 31 text

Modelling time series and queries

Slide 32

Slide 32 text

Querying One denormalised table for each query Query Key id1 ts11 id1 ts12 id2 ts21 id2 ts22 p1 b1 v11 p2 b2 v12 v21 p2 b2 v22

Slide 33

Slide 33 text

• Cons: • Denormalise for each query, again and again • Higher disk usage • Disk space is cheap, but not free • Write latency is affected • Time-bucketed indexes can create hot spots (hot shards)

Slide 34

Slide 34 text

Flexible, adaptable architecture

Slide 35

Slide 35 text

Adapt the data model to real-world constraints

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

• Same service contract • Indices are updated synchronously or asynchronously • Basic client guarantee: if a POST is successful the event has been persisted “sufﬁciently” • Events can be published to a message broker

Slide 38

Slide 38 text

• Pro • Decoupling • client are unaware of the implementation details • Intuitive ReSTful interface • Disk consumption is more reasonable • Easily extensible • Pub/sub

Slide 39

Slide 39 text

• Cons • Not primarily optimised for latency • Still sufﬁciently performant for our use-cases • More complex service code • Needs to execute multiple CQL queries in sequence • Cluster hotspots can still occur, in theory

Slide 40

Slide 40 text

Indices • Ascending and descending time buckets for each query type • Index value references an event stored in the main table by its id

Slide 41

Slide 41 text

Indices events_by_type_asc ( tbucket text, type text, eventid timeuuid, primary key ((type, tbucket), eventid)) with clustering order by (eventid asc); events_by_type_desc ( tbucket text, type text, eventid timeuuid, primary key ((type, tbucket), eventid)) with clustering order by (eventid desc); Ascending and descending time buckets for each query type

Slide 42

Slide 42 text

Example: Implementing Pagination

Slide 43

Slide 43 text

Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5

Slide 44

Slide 44 text

Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5

Slide 45

Slide 45 text

Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎ type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

GET /api/events?start=141..&type=X&limit=5 { “events” : [ { “id” : “uuid1”, “type” : “X”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?continueFrom=uuid7&type=X&limit=5” }

Slide 49

Slide 49 text

Performance Characteristics • 70 to 85 million events per day • Client latency increases moderately with increased parallel load (40ms to 60ms, +10ms on the client) • Current behaviour exceeds by far current target volumes

Slide 50

Slide 50 text

Lessons learnt • Scalability is not only about raw performance and latency • Experiment • Simplify • Understand Thrift, use CQL

Slide 51

Slide 51 text

Links • http://www.opencredo.com/blog • @davib0 • @tareq_abedrabbo Thank you! Any questions?

Slide 52

Slide 52 text

Future improvements

Slide 53

Slide 53 text

• Data model improvements: User Deﬁned Types • DateTiered compaction • Analytics with Spark • Add other data views