Share
Embed
Kafka for Rubyists: Topics and Publishing Messages
karol.galanciak
December 20, 2020
Transcript
10 slides
-
Modelling Topics & Partitions - There is no silver bullet
- Very dependent on the applications - Two extreme opposite sides of the spectrum: one topic for everything and separate topic for every event - don’t do this -
Modelling Topics & Partitions - some good rules of thumb
- Topic: a concept of grouping things together that are close to each other - Number of topics depends on the ecosystem - for example not all consumer groups might be interested in all events, having them go through the events they don’t care might not be ideal - Producers shouldn’t be too much coupled to the logic behind multiple applications/consumer groups, strive for the balance -
Modelling Topics & Partitions - some good rules of thumb
- Things that belong to the same “tenant” should be published to the same partition under a given topic (again, causality of the events) -
What to publish? And what not to publish? - What
not to publish: unless you only care about database replication or are building some archive service, don’t do paper-trail- versions style of serialization & publishing model.as_json etc. - this can lead to a “distributed monolith“ -
What to publish? - Domain events - for example, “payment_paid”
event with a payload containing things that will be useful for the consumers to do something with the event, so maybe amount, currency, payment time, transaction ID etc. - Events for activity tracking, metrics and analytics, e.g. “page_visited” with user’s ID, timestamp and URL of the page etc. -
What to publish? - “Resources” projections - when you need
some data to be available in other services. Think more like HTTP Resources in REST API that don’t map 1:1 to a database, including sideloading (e.g. invoice together with its services in a single event instead of having separate events for invoice and services). -
What to publish? - Even better analogy - DDD aggregate,
respecting transactional boundaries, for example, “invoice_created” event that would contain invoice data itself, customer’s data, services etc. Like a MongoDB document. -
Example of topics for the invoicing app - Topic 1:
Companies - the actual users of the application - Topic 2: Invoices - everything related to the invoices, including services/items, customer’s data, amounts, payment terms etc. - Topic 3: Billing - whatever is related to charging Companies