Kafka for Rubyists: Consuming and Performance

Transcript

1. Consuming and performance

2. Consumers’ performance - It’s easier to have poorly performing consumers

   than you think - Every DB query matters when consuming messages - Rule of thumb: make handlers as lightweight as possible

3. Lightweight Consumers - Database queries limited to absolute minimum -

   Handle side-effects as background jobs (Sidekiq)

4. Consumers’ Number - The more consumers, the better - But

   there is a limit: having more consumers than topics/partition is useless (a partition can be assigned to only one consumer within a consumer group) - More topics/partitions makes it easy to optimize throughput (although it might cause other problems, be careful)

5. Further ways to optimize throughput - Custom Partition Assignment Strategy

   - A default one: round-robin - Requires having a good idea about the number of messages in different topics/ partitions and how fast they can be processed - Good monitoring is your friend

6. Further ways to optimize throughput - karafka-sidekiq-backend - Messages are

   going to be enqueued from Karafka workers to be processed in Sidekiq workers - Can result in a huge throughput increase (more Sidekiq workers, more throughput) - Messages are no longer ordered :( - Be careful about Redis/Sidekiq issues

7. Thanks!