Building a Scalable Event Service with Cassandra

DataStax Tech Days - London - 30 September 2014

Transcript

1. Building a Scalable Event Service with Cassandra Tareq Abedrabbo - Open Credo DataStax Tech Days
2. About Us • CTO at Open Credo • We are a software consultancy and delivery company • Open source, NoSQL/Big Data, cloud • DataStax Solutions Partner
3. Agenda • Project background • The event service • API • Architecture • Implementation • Future improvements and lessons learnt
4. Project background
5. • Large retail organisation • Decoupled micro services architecture • Java-based, event-driven platform • Cassandra, Cloud Foundry, RabbitMQ
6. Why do we need an event service?
7. • Capture platform and business events • Trigger downstream processes asynchronously • Customisation standard processes in a non- intrusive way • Analytics • System testing
8. Design principles
9. • Simplicity (yes, really!) • Decouple API and implementation (contract-first) • Scalability and fault-tolerance • Flexibility in the implementation • Ability to evolve while minimising changes
10. Event service API
11. • Store an event • POST /api/events/ • Read an event • GET /api/events/{eventId}
12. { "type" : "DEMOENTITY.DEMOPROCESS.DEMOTASK", "source" : "demoapp1:2.5:981a24b3-2860-40ba-90d4-c47ef1a70abe", "clientTimestamp" : 1401895567594, "serverTimestamp" : 1401895568616, "platformContext" : { "id" : "demoapp1", "version" : "2.5" }, "businessContext" : { "channel" : "WEB", }, "payload" : { "message" : "foo", "anInteger" : 33, "bool" : false } }
13. • Query events • GET /api/events?{queryString} • {queryString} can consist of the following fields: • start, end, startOffset, limit, tag, type, order
14. • Examples: • GET /api/events?start={startTime} &end={endTime} • GET /api/events? startOffset=3600000&type=someType
15. { "count" : 1, "continuation" : "http://event-service-location/api/events?continueFrom=9f00e9d0- ebfc-11e3-81c5-09597eb bf2cb&end=1401965827000&limit=1", "events" : [ { "type" : "DEMOENTITY.DEMOPROCESS.DEMOTASK", "source" : "demoapp1:2.5:981a24b3-2860-40ba-90d4-c47ef1a70abe", "clientTimestamp" : 1401895567594, "serverTimestamp" : 1401895568616, "platformContext" : { "id" : "demoapp1", "version" : "2.5" }, "businessContext" : { }, "payload" : { }, “self" : ”http://event-service-location/api/events/8d4ce680- ebfc-11e3-81c5-09597ebbf2cb" } ] }
16. V1: create and read an event
18. V2: querying events and notifications
20. Anatomy of an Event public class Event { ! private final UUID id; private final EventType type; private final String source; private final Date clientTimestamp; private final Date serverTimestamp; private final Map<String, Object> platformContext; private final Map<String, Object> businessContext; private final Map<String, Object> payload; private final List<String> tags; }
21. Primary Event Store create table events (id timeuuid primary key, source text, type text, cts timestamp, sts timestamp, bct map<text, text>, bcv map<text, blob>, pct map<text, text>, pcv map<text, blob>, plt map<text, text>, plv map<text, blob>); Events are simply keyed by id
22. Indices create table events_by_time_asc (tbucket text, eventid timeuuid, primary key (tbucket, eventid)) with clustering order by (eventid asc); create table events_by_time_desc (tbucket text, eventid timeuuid, primary key (tbucket, eventid)) with clustering order by (eventid desc); Ascending and descending time buckets for each query type
23. Future improvements • User Defined Types • DataStax SLOR integration • Spark integration • Testing with Stubbed Cassandra
24. Some Lessons learnt • Experiment • Simplify • Understand Thrift, use CQL
25. Links • Twitter: @tareq_abedrabbo • OpenCredo: http://www.opencredo.com/blog Thank you! Questions?