Kafka for Rubyists: Topics and Publishing Messages

Transcript

1. Kafka - Modelling Topics, Publishing Events
2. Modelling Topics & Partitions - There is no silver bullet - Very dependent on the applications - Two extreme opposite sides of the spectrum: one topic for everything and separate topic for every event - don’t do this
3. Modelling Topics & Partitions - some good rules of thumb - Topic: a concept of grouping things together that are close to each other - Number of topics depends on the ecosystem - for example not all consumer groups might be interested in all events, having them go through the events they don’t care might not be ideal - Producers shouldn’t be too much coupled to the logic behind multiple applications/consumer groups, strive for the balance
4. Modelling Topics & Partitions - some good rules of thumb - Things that belong to the same “tenant” should be published to the same partition under a given topic (again, causality of the events)
5. What to publish? And what not to publish? - What not to publish: unless you only care about database replication or are building some archive service, don’t do paper-trail- versions style of serialization & publishing model.as_json etc. - this can lead to a “distributed monolith“
6. What to publish? - Domain events - for example, “payment_paid” event with a payload containing things that will be useful for the consumers to do something with the event, so maybe amount, currency, payment time, transaction ID etc. - Events for activity tracking, metrics and analytics, e.g. “page_visited” with user’s ID, timestamp and URL of the page etc.
7. What to publish? - “Resources” projections - when you need some data to be available in other services. Think more like HTTP Resources in REST API that don’t map 1:1 to a database, including sideloading (e.g. invoice together with its services in a single event instead of having separate events for invoice and services).
8. What to publish? - Even better analogy - DDD aggregate, respecting transactional boundaries, for example, “invoice_created” event that would contain invoice data itself, customer’s data, services etc. Like a MongoDB document.
9. Example of topics for the invoicing app - Topic 1: Companies - the actual users of the application - Topic 2: Invoices - everything related to the invoices, including services/items, customer’s data, amounts, payment terms etc. - Topic 3: Billing - whatever is related to charging Companies
10. Thanks!