Introduction to Kafka

This is an intro to Kafka lightning talk I gave at the June 2014 LA DevOps Meetup: http://www.meetup.com/ladevops/events/189617262/

Transcript

1. introduction to kafka
2. who am i? • brandon burton • @solarce • Operations Lead @ lookout.com • We use Kafka
3. what is kafka?
4. persistent
5. distributed
6. replicated
7. publish/subscribe
8. messaging system
9. persistent, distributed, replicated, pub/sub messaging system
11. apache kafka • top level apache project • written in scala • written at and open source by LinkedIn in 2011 • primary developers still employed at LinkedIn • uses zookeeper for state and leader election
12. goals • fast • durable • simple design • simple api
13. key terms • broker • topic • partition • publisher • consumer
14. All-in-one system with one architecture and one API • Kafka is a specialized system and overlaps uses cases for both offline and real-time log processing. 9
15. broker ! Receive messages from Producers (push), deliver messages to Consumers (pull)
16. topic ! Topics are queues. They are logical collections of partitions.
17. partition ! partitions are replicated append-only log files a partition stores a subset of the messages for a topic
18. partition /RJ)RUPDW DSSHQG /RJILOH 0HVVDJH6HW 0HVVDJH6HW 0HVVDJH6HW 0HVVDJH6HW WPSNDINDORJVNDIND DSSHQG DSSHQG DSSHQG %URNHU %\WH%XIIHU0HVVDJH6HWZULWH7R )LOH&KDQQHO ,QGH[    0HVVDJHRIIVHW LQGH[
19. publisher ! publishers write messages to a topic, using a hashing function to determine which partition to write a message to
20. consumer ! consumers request a range of messages on a specific topic from the broker. !
21. further reading • https://kafka.apache.org/ documentation.html#monitoring • https://cwiki.apache.org/confluence/display/KAFKA/ Replication+tools • https://engineering.linkedin.com/kafka/benchmarking- apache-kafka-2-million-writes-second-three-cheap- machines • Enterprise Kafka as a Service: https:// www.youtube.com/watch?v=7dkSze52i-o