$30 off During Our Annual Pro Sale. View Details »

Kafka for Rubyists: Consuming and Performance

Kafka for Rubyists: Consuming and Performance

karol.galanciak

December 27, 2020
Tweet

More Decks by karol.galanciak

Other Decks in Programming

Transcript

  1. Consuming and performance

    View Slide

  2. Consumers’ performance
    - It’s easier to have poorly performing
    consumers than you think
    - Every DB query matters when consuming
    messages
    - Rule of thumb: make handlers as lightweight
    as possible

    View Slide

  3. Lightweight Consumers
    - Database queries limited to absolute minimum
    - Handle side-effects as background jobs
    (Sidekiq)

    View Slide

  4. Consumers’ Number
    - The more consumers, the better
    - But there is a limit: having more consumers
    than topics/partition is useless (a partition can
    be assigned to only one consumer within a
    consumer group)
    - More topics/partitions makes it easy to
    optimize throughput (although it might cause
    other problems, be careful)

    View Slide

  5. Further ways to optimize throughput
    - Custom Partition Assignment Strategy
    - A default one: round-robin
    - Requires having a good idea about the
    number of messages in different topics/
    partitions and how fast they can be processed
    - Good monitoring is your friend

    View Slide

  6. Further ways to optimize throughput
    - karafka-sidekiq-backend
    - Messages are going to be enqueued from
    Karafka workers to be processed in Sidekiq
    workers
    - Can result in a huge throughput increase
    (more Sidekiq workers, more throughput)
    - Messages are no longer ordered :(
    - Be careful about Redis/Sidekiq issues

    View Slide

  7. Thanks!

    View Slide