Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Kafka

Introduction to Kafka

This is an intro to Kafka lightning talk I gave at the June 2014 LA DevOps Meetup: http://www.meetup.com/ladevops/events/189617262/

Brandon Burton

July 01, 2014
Tweet

More Decks by Brandon Burton

Other Decks in Technology

Transcript

  1. who am i? • brandon burton • @solarce • Operations

    Lead @ lookout.com • We use Kafka
  2. apache kafka • top level apache project • written in

    scala • written at and open source by LinkedIn in 2011 • primary developers still employed at LinkedIn • uses zookeeper for state and leader election
  3. All-in-one system with one architecture and one API • Kafka

    is a specialized system and overlaps uses cases for both offline and real-time log processing. 9
  4. partition /RJ)RUPDW DSSHQG /RJILOH 0HVVDJH6HW 0HVVDJH6HW 0HVVDJH6HW 0HVVDJH6HW WPSNDINDORJVNDIND DSSHQG

    DSSHQG DSSHQG %URNHU %\WH%XIIHU0HVVDJH6HWZULWH7R )LOH&KDQQHO ,QGH[    0HVVDJHRIIVHW LQGH[
  5. publisher ! publishers write messages to a topic, using a

    hashing function to determine which partition to write a message to
  6. consumer ! consumers request a range of messages on a

    specific topic from the broker. !
  7. further reading • https://kafka.apache.org/ documentation.html#monitoring • https://cwiki.apache.org/confluence/display/KAFKA/ Replication+tools • https://engineering.linkedin.com/kafka/benchmarking-

    apache-kafka-2-million-writes-second-three-cheap- machines • Enterprise Kafka as a Service: https:// www.youtube.com/watch?v=7dkSze52i-o