Introduction to Kafka

Introduction to Kafka

This is an intro to Kafka lightning talk I gave at the June 2014 LA DevOps Meetup: http://www.meetup.com/ladevops/events/189617262/

675e2b6f653233a3a4d4e04f34610e1d?s=128

Brandon Burton

July 01, 2014
Tweet

Transcript

  1. introduction to kafka

  2. who am i? • brandon burton • @solarce • Operations

    Lead @ lookout.com • We use Kafka
  3. what is kafka?

  4. persistent

  5. distributed

  6. replicated

  7. publish/subscribe

  8. messaging system

  9. persistent, distributed, replicated, pub/sub messaging system

  10. None
  11. apache kafka • top level apache project • written in

    scala • written at and open source by LinkedIn in 2011 • primary developers still employed at LinkedIn • uses zookeeper for state and leader election
  12. goals • fast • durable • simple design • simple

    api
  13. key terms • broker • topic • partition • publisher

    • consumer
  14. All-in-one system with one architecture and one API • Kafka

    is a specialized system and overlaps uses cases for both offline and real-time log processing. 9
  15. broker ! Receive messages from Producers (push), deliver messages to

    Consumers (pull)
  16. topic ! Topics are queues. They are logical collections of

    partitions.
  17. partition ! partitions are replicated append-only log files a partition

    stores a subset of the messages for a topic
  18. partition /RJ)RUPDW DSSHQG /RJILOH 0HVVDJH6HW 0HVVDJH6HW 0HVVDJH6HW 0HVVDJH6HW WPSNDINDORJVNDIND DSSHQG

    DSSHQG DSSHQG %URNHU %\WH%XIIHU0HVVDJH6HWZULWH7R )LOH&KDQQHO ,QGH[    0HVVDJHRIIVHW LQGH[
  19. publisher ! publishers write messages to a topic, using a

    hashing function to determine which partition to write a message to
  20. consumer ! consumers request a range of messages on a

    specific topic from the broker. !
  21. further reading • https://kafka.apache.org/ documentation.html#monitoring • https://cwiki.apache.org/confluence/display/KAFKA/ Replication+tools • https://engineering.linkedin.com/kafka/benchmarking-

    apache-kafka-2-million-writes-second-three-cheap- machines • Enterprise Kafka as a Service: https:// www.youtube.com/watch?v=7dkSze52i-o
  22. None