How Plumbr uses Kafka

Transcript

1. Eating Kafka Nikita Salnikov-Tarnovski @iNikem
2. Intro to Kafka
3. What is Kafka • Distributed streaming platform • It lets you publish and subscribe to streams of records • It lets you store streams of records in a fault-tolerant way.
4. What is Kafka • Kafka runs as a cluster on one or more servers. • The Kafka cluster stores streams of records in categories called topics. • Each record consists of a key, a value, and a timestamp.
5. Four APIs http://kafka.apache.org/documentation/
6. Append log http://kafka.apache.org/documentation/
7. Brokers • Several brokers form a cluster • Coordinated with Zookeeper • All partitions are distributed among brokers
8. Producers • Producer sends record to a topic • Based on a key, partition is chosen • Leader broker is found • Wait for requested acks
9. Fast writes • Brokers cheat and don’t write to disk • They write to disk cache • And let OS care about flushing to disk
10. Replication • Each topic can be replicated among brokers • So for each partition there are X copies • Brokers just consume messages from leader
11. Consumer groups (c) Confluent
12. Consumer rebalance (c) Confluent
13. Commit • Consumer has to commit offsets he consumed • You have to decide, when and how!
14. Delivery semantics • At least once • At most once • Exactly once
15. Kafka Connect • Off-the-shelf solution to pipe data to or from Kafka • E.g. DB, Elasticsearch, files, etc…
16. Kafka Streams • DSL and platform for writing data processing streams • If you squint enough, very similar to Java8 streams and Fork-Join pool • But across multiple jvms and servers
17. Kafka in Plumbr
18. Kafka cluster • 5 brokers • 2x replication • 20T data for last 90 days • Inflow ~125G per day
19. Data processing pipeline
20. Spring Cloud Stream • Greatly simplifies development of Kafka based apps • Couple of annotations and data flows :)
21. Solving performance problems is hard. We don’t think it needs to be. @JavaPlumbr/@iNikem http://plumbr.eu