Kafka for Rubyists: Topics and Publishing Messages

Transcript

1. Kafka - Modelling Topics, Publishing
   Events
   

   View Slide

2. Modelling Topics & Partitions
   - There is no silver bullet
   - Very dependent on the applications
   - Two extreme opposite sides of the spectrum:
   one topic for everything and separate topic for
   every event - don’t do this
   

   View Slide

3. Modelling Topics & Partitions - some good rules of thumb
   - Topic: a concept of grouping things together that
   are close to each other
   - Number of topics depends on the ecosystem - for
   example not all consumer groups might be
   interested in all events, having them go through
   the events they don’t care might not be ideal
   - Producers shouldn’t be too much coupled to the
   logic behind multiple applications/consumer
   groups, strive for the balance
   

   View Slide

4. Modelling Topics & Partitions - some good rules of thumb
   - Things that belong to the same “tenant”
   should be published to the same partition
   under a given topic (again, causality of the
   events)
   

   View Slide

5. What to publish? And what not to publish?
   - What not to publish: unless you only care
   about database replication or are building
   some archive service, don’t do paper-trail-
   versions style of serialization & publishing
   model.as_json etc. - this can lead to a
   “distributed monolith“
   

   View Slide

6. What to publish?
   - Domain events - for example, “payment_paid”
   event with a payload containing things that will
   be useful for the consumers to do something
   with the event, so maybe amount, currency,
   payment time, transaction ID etc.
   - Events for activity tracking, metrics and
   analytics, e.g. “page_visited” with user’s ID,
   timestamp and URL of the page etc.
   

   View Slide

7. What to publish?
   - “Resources” projections - when you need
   some data to be available in other services.
   Think more like HTTP Resources in REST API
   that don’t map 1:1 to a database, including
   sideloading (e.g. invoice together with its
   services in a single event instead of having
   separate events for invoice and services).
   

   View Slide

8. What to publish?
   - Even better analogy - DDD aggregate,
   respecting transactional boundaries, for
   example, “invoice_created” event that would
   contain invoice data itself, customer’s data,
   services etc. Like a MongoDB document.
   

   View Slide

9. Example of topics for the invoicing app
   - Topic 1: Companies - the actual users of the
   application
   - Topic 2: Invoices - everything related to the
   invoices, including services/items, customer’s
   data, amounts, payment terms etc.
   - Topic 3: Billing - whatever is related to
   charging Companies
   

   View Slide

10. Thanks!
    

    View Slide