Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka for Rubyists: Consuming and Performance

Kafka for Rubyists: Consuming and Performance

347a1b64f3a6e38981bc99b53919e2b1?s=128

karol.galanciak

December 27, 2020
Tweet

Transcript

  1. Consuming and performance

  2. Consumers’ performance - It’s easier to have poorly performing consumers

    than you think - Every DB query matters when consuming messages - Rule of thumb: make handlers as lightweight as possible
  3. Lightweight Consumers - Database queries limited to absolute minimum -

    Handle side-effects as background jobs (Sidekiq)
  4. Consumers’ Number - The more consumers, the better - But

    there is a limit: having more consumers than topics/partition is useless (a partition can be assigned to only one consumer within a consumer group) - More topics/partitions makes it easy to optimize throughput (although it might cause other problems, be careful)
  5. Further ways to optimize throughput - Custom Partition Assignment Strategy

    - A default one: round-robin - Requires having a good idea about the number of messages in different topics/ partitions and how fast they can be processed - Good monitoring is your friend
  6. Further ways to optimize throughput - karafka-sidekiq-backend - Messages are

    going to be enqueued from Karafka workers to be processed in Sidekiq workers - Can result in a huge throughput increase (more Sidekiq workers, more throughput) - Messages are no longer ordered :( - Be careful about Redis/Sidekiq issues
  7. Thanks!