Reactive Micro-Services with CQRS/DDD/ES and Kafka

Transcript

1. Reactive Micro-Services with CQRS/DDD/ES and Kafka Ben Goldin
 @vgoldin
2. Reactive? React to events - Event Driven React to load - Scalable React to failures - Resilient React to users - Responsive No surprises…most are quality attributes of any good system
3. Micro-Services? Do one thing, and do it well Maintain independently Evolve independently Scale independently This is not a Micro-Services meet up anyway :)
4. CQRS? Separate read model from the write model Decouple read side from the write side Provide foundation for Reactive Works well with Micro- Services No frameworks, please…
5. Event Sourcing? Track changes to the domain aggregate root as a sequence of events Publish events to a multiple subscribers Including denormaliser of the query side of CQRS
6. Domain-Driven-Design?
7. Micro-Services Architecture around CQRS concepts API [Container: Micro-Service] - RESTful Service Boundary - Projection Query Service - Denormalisers - Command Dispatcher Domain [Container: Micro-Service] - Aggregate Root - Command Handlers - Event Handlers - Event Store Commands Events Actor Business Capability [System] command / query
8. API Micro-Service API Micro-Service RESTfull API [JAXRS Resource] Description of functionality component provides. Denormaliser [Event Handler] Updates the projection based on Event Data Command Dispatcher [Component] Dispatch user/api commands to the CommandBus Query Service [Component] Description of functionality component provides. query projection Projection / Read DB query upsert dispatch command dispatch command publish event Messaging User
9. Domain Micro-Service Domain Micro-Service system boundary Command Handler [Component] Validates the commands originated from the API and delegates action to the aggregate root API Micro-Service [System] Aggregate Root [Component] Manages the state of the aggregate root and performs business actions on it execute (2.2) Repository [Component] Restores the Aggregate Root state from the events / Saves the new events to the event store load AR (2.1) save AR (2.3) Event Store [Component] Tracks the changes to the aggregate root as a sequence of events Event Publisher [Component] Publishes events, stored in the event store, to the external subscribers load events (2.1.1) / store events (2.3.1) publish (2.3.2) Messaging event (4) command (2) command (1) event (3) failure events (3a)
10. Kafka? Distributed and ordered commit log Pull-based Publish/Subscribe Messaging with message retention on disk Fault-tolerant arbitrary scalability Isolated topics and partitions per consumer group Binary TCP-based communication protocol
11. Kafka at LinkedIn In Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. At the busiest times of day, we are receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organised into more than 60 clusters.
12. Kafka for Messaging in CQRS? Guaranties ordering within one partition Partition per key, where key equals aggregate root id Catch-Up Subscriptions for event reply and ordering assurance Offset per consumer group for N to N models and durable subscriptions - similar to VirtualTopic in ActiveMQ
13. Kafka for Event Sourcing? Kafka is not a database
 not designed to query Hence not possible to reply events, filtering particular aggregate root Creating Topic per Aggregate Root will lead to a huge number of topics / partitions “Comfortable” number of partitions (hence topics) is 10k [*] https://cwiki.apache.org/ confluence/display/KAFKA/ FAQ#FAQ- HowmanytopicscanIhave?
14. Reference Implementation [*] based on SimpleCQRS from Greg Young
 https://github.com/gregoryyoung/m-r https://github.com/vgoldin/cqrs-eventsourcing-kafka