Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka for Rubyists: Intro to Kafka

Kafka for Rubyists: Intro to Kafka

347a1b64f3a6e38981bc99b53919e2b1?s=128

karol.galanciak

December 20, 2020
Tweet

Transcript

  1. Intro to Kafka

  2. What is Kafka? - Distributed event streaming platform - Publish-subscribe

    model - Implemented as persistent and durable append- only log(s) - Dumb broker, smart consumer (contrary to standard message queues) - Highly performant and scalable (if it’s good enough for LinkedIn, it’s gonna be just fine for you)
  3. A Message queue or a persistent storage? - Messages are

    not removed from the log once consumed - Controlled via retention policy - Messages are stored in topics that can be partitioned - Messages are identified by “offsets” in a given topic/partition
  4. None
  5. Practical considerations - Semantics: at-most-once delivery vs. at-least- once delivery

    - Exactly-once delivery? - Log compaction
  6. Practical considerations - Partitions as parallelization unit (one consumer within

    a consumer group for partition) - More partitions - more throughput - More partitions - potentially more problems as well :( (increased unavailability, latency) - Ordering of the messages is preserved only in a given partition of a topic. This is critical if you care about causality
  7. Kafka vs. RabbitMQ - Two different things! - Different system

    for a different purpose (one is not better than another) - RabbitMQ is a smart broker/dumb consumer type of system - No persistent/durable storage in Rabbit
  8. Kafka killer features summary - Performance - Durability - Ability

    to replay events (even indefinitely, durability is for real) - Log compaction - Heart of the event-driven ecosystems (Kafka Streams, Apache Samza/Spark/Flink, oh my!)
  9. Kafka use cases - Pub-Sub system for (micro)services architecture -

    Activity Tracking - Metrics and Analytics - Stream processing - Event Sourcing - Cross-database replication (if you know what you are doing)
  10. Thanks!