Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka for Rubyists: Consuming and Performance

Kafka for Rubyists: Consuming and Performance


December 27, 2020

More Decks by karol.galanciak

Other Decks in Programming


  1. Consumers’ performance - It’s easier to have poorly performing consumers

    than you think - Every DB query matters when consuming messages - Rule of thumb: make handlers as lightweight as possible
  2. Lightweight Consumers - Database queries limited to absolute minimum -

    Handle side-effects as background jobs (Sidekiq)
  3. Consumers’ Number - The more consumers, the better - But

    there is a limit: having more consumers than topics/partition is useless (a partition can be assigned to only one consumer within a consumer group) - More topics/partitions makes it easy to optimize throughput (although it might cause other problems, be careful)
  4. Further ways to optimize throughput - Custom Partition Assignment Strategy

    - A default one: round-robin - Requires having a good idea about the number of messages in different topics/ partitions and how fast they can be processed - Good monitoring is your friend
  5. Further ways to optimize throughput - karafka-sidekiq-backend - Messages are

    going to be enqueued from Karafka workers to be processed in Sidekiq workers - Can result in a huge throughput increase (more Sidekiq workers, more throughput) - Messages are no longer ordered :( - Be careful about Redis/Sidekiq issues