Consuming and performance 

Consumers’ performance - It’s easier to have poorly performing consumers than you think - Every DB query matters when consuming messages - Rule of thumb: make handlers as lightweight as possible 

Lightweight Consumers - Database queries limited to absolute minimum - Handle side-effects as background jobs (Sidekiq) 

Consumers’ Number - The more consumers, the better - But there is a limit: having more consumers than topics/partition is useless (a partition can be assigned to only one consumer within a consumer group) - More topics/partitions makes it easy to optimize throughput (although it might cause other problems, be careful) 

Further ways to optimize throughput - Custom Partition Assignment Strategy - A default one: round-robin - Requires having a good idea about the number of messages in different topics/ partitions and how fast they can be processed - Good monitoring is your friend 

Further ways to optimize throughput - karafka-sidekiq-backend - Messages are going to be enqueued from Karafka workers to be processed in Sidekiq workers - Can result in a huge throughput increase (more Sidekiq workers, more throughput) - Messages are no longer ordered :( - Be careful about Redis/Sidekiq issues 

Thanks!