Kafka for Rubyists: Consuming and Performance

Transcript

1. Consuming and performance
2. Consumers’ performance - It’s easier to have poorly performing consumers than you think - Every DB query matters when consuming messages - Rule of thumb: make handlers as lightweight as possible
3. Lightweight Consumers - Database queries limited to absolute minimum - Handle side-effects as background jobs (Sidekiq)
4. Consumers’ Number - The more consumers, the better - But there is a limit: having more consumers than topics/partition is useless (a partition can be assigned to only one consumer within a consumer group) - More topics/partitions makes it easy to optimize throughput (although it might cause other problems, be careful)
5. Further ways to optimize throughput - Custom Partition Assignment Strategy - A default one: round-robin - Requires having a good idea about the number of messages in different topics/ partitions and how fast they can be processed - Good monitoring is your friend
6. Further ways to optimize throughput - karafka-sidekiq-backend - Messages are going to be enqueued from Karafka workers to be processed in Sidekiq workers - Can result in a huge throughput increase (more Sidekiq workers, more throughput) - Messages are no longer ordered :( - Be careful about Redis/Sidekiq issues
7. Thanks!