Kafka for Rubyists: Consuming and Performance

Transcript

1. Consuming and performance
   

   View Slide

2. Consumers’ performance
   - It’s easier to have poorly performing
   consumers than you think
   - Every DB query matters when consuming
   messages
   - Rule of thumb: make handlers as lightweight
   as possible
   

   View Slide

3. Lightweight Consumers
   - Database queries limited to absolute minimum
   - Handle side-effects as background jobs
   (Sidekiq)
   

   View Slide

4. Consumers’ Number
   - The more consumers, the better
   - But there is a limit: having more consumers
   than topics/partition is useless (a partition can
   be assigned to only one consumer within a
   consumer group)
   - More topics/partitions makes it easy to
   optimize throughput (although it might cause
   other problems, be careful)
   

   View Slide

5. Further ways to optimize throughput
   - Custom Partition Assignment Strategy
   - A default one: round-robin
   - Requires having a good idea about the
   number of messages in different topics/
   partitions and how fast they can be processed
   - Good monitoring is your friend
   

   View Slide

6. Further ways to optimize throughput
   - karafka-sidekiq-backend
   - Messages are going to be enqueued from
   Karafka workers to be processed in Sidekiq
   workers
   - Can result in a huge throughput increase
   (more Sidekiq workers, more throughput)
   - Messages are no longer ordered :(
   - Be careful about Redis/Sidekiq issues
   

   View Slide

7. Thanks!
   

   View Slide