Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka for Rubyists: Topics and Publishing Messages

Kafka for Rubyists: Topics and Publishing Messages



December 20, 2020


  1. Kafka - Modelling Topics, Publishing Events

  2. Modelling Topics & Partitions - There is no silver bullet

    - Very dependent on the applications - Two extreme opposite sides of the spectrum: one topic for everything and separate topic for every event - don’t do this
  3. Modelling Topics & Partitions - some good rules of thumb

    - Topic: a concept of grouping things together that are close to each other - Number of topics depends on the ecosystem - for example not all consumer groups might be interested in all events, having them go through the events they don’t care might not be ideal - Producers shouldn’t be too much coupled to the logic behind multiple applications/consumer groups, strive for the balance
  4. Modelling Topics & Partitions - some good rules of thumb

    - Things that belong to the same “tenant” should be published to the same partition under a given topic (again, causality of the events)
  5. What to publish? And what not to publish? - What

    not to publish: unless you only care about database replication or are building some archive service, don’t do paper-trail- versions style of serialization & publishing model.as_json etc. - this can lead to a “distributed monolith“
  6. What to publish? - Domain events - for example, “payment_paid”

    event with a payload containing things that will be useful for the consumers to do something with the event, so maybe amount, currency, payment time, transaction ID etc. - Events for activity tracking, metrics and analytics, e.g. “page_visited” with user’s ID, timestamp and URL of the page etc.
  7. What to publish? - “Resources” projections - when you need

    some data to be available in other services. Think more like HTTP Resources in REST API that don’t map 1:1 to a database, including sideloading (e.g. invoice together with its services in a single event instead of having separate events for invoice and services).
  8. What to publish? - Even better analogy - DDD aggregate,

    respecting transactional boundaries, for example, “invoice_created” event that would contain invoice data itself, customer’s data, services etc. Like a MongoDB document.
  9. Example of topics for the invoicing app - Topic 1:

    Companies - the actual users of the application - Topic 2: Invoices - everything related to the invoices, including services/items, customer’s data, amounts, payment terms etc. - Topic 3: Billing - whatever is related to charging Companies
  10. Thanks!