How It Works - Kafka

A series of talks on data engineering

Transcript

1. HOW IT WORKS: KAFKA 1
2. PLAN use-cases concepts architecture demo 2 . 1
3. CAPABILITIES publish-subscribe storage stream processing 3 . 1
4. APPLICATIONS streaming data pipelines reactive applications 4 . 1
5. USE-CASES Messaging: traditional message broker pattern of data processing Website Activity Tracking: real-time publish-subscribe feeds in domains of page views, searches, and other user interactions. Metrics: operational monitoring data processing. Log Aggregation: collecting physical log files and store them for further processing. Stream Processing: multi-stage data processing pipelines. Event Sourcing: support of apps built with stored event sequences that can be replayed and applied again for deriving a consistent system state. Commit Log: the type of data stored in distributed system that ensures the re-syncing mechanism. 5 . 1
6. CORE CONCEPTS
7. 6 . 1
8. MESSAGE key value timestamp 6 . 2
9. TOPICS & PARTITIONS & OFFSETS 6 . 3
10. PRODUCERS & CONSUMERS
11. 6 . 4
12. PARTITIONING & REPLICATION FACTOR 6 . 5
13. COMPONENTS core brokers zookeeper producer & consumer api admin api kafka streams kafka connect 7 . 1
14. 7 . 2
15. ARCHITECTURE 8 . 1
17. 8 . 2
18. 8 . 3
19. replication & fault tolerance brokers & leader election zookeeper 8 . 4
20. PRODUCER API leader balancing partition control ack (message acknowledge) async & batching 9 . 1
21. CONSUMER API message access patterns queue pub/sub 10 . 1
22. CONSUMER GROUP 10 . 2
23. CONSUMER API consumer vs consumer group offset control & offset storage push vs pull and optimal batching 10 . 3
24. ADMIN, MONITORING, CONFIGURATION UI & monitoring CLI admin api configs: topic, broker, consumer, producer, admin, (connect, streams) 11 . 1
25. LOG STORAGE commit log instead of transient queue starting/ending offsets acknowledgement & replication topic ttl sequential disk access vs random-memory access log compaction 12 . 1
26. 12 . 2
27. DELIVERY SEMANTICS ordering at producer ordering at consumer delivery guarantees at-most-once vs. at-least-once vs. exactly- once 13 . 1
28. <..> VS. KAFKA yes flexibility scaling, parallelism pub/sub or queue msg ordering encryption integration streaming connect protocols persistance (commit log vs. trasient queue) HA, fault-tolerance (tolerate N-1 server failures) 14 . 1
29. no msg priority transactions (no out of the box) consumer ack producer-consumer routing get msg by key (offset-based consuming) batch processing 14 . 2
30. PARALLELISM number of partitions 15 . 1
31. KAFKA STREAMS 16 . 1
32. 16 . 2
33. streams-architecture-example-02.png 16 . 3
34. KAFKA CONNECT 17 . 1
35. DEMO 18 . 1
36. STANDALONE 18 . 2
37. TIM WALKTHROUGH 18 . 3
38. ui & monitoring & zoo partitioning, replication, ttl brokers & skew consumers cli std kafkacat 18 . 4
39. TIPS & TRICKS deployment: usually ec2, packer/ansible AWS MKS interface: kafkacat, kafka ui confluent platform and schema management offset storage partitioning strategies 19 . 1
40. COMMON ARCHITECTURES s3/sns/hdfs + kafka simple producer + kafka kafka + spark kafka + akka kafka connect (s3, http, hdfs, sns, kinesis, jdbc, hbase, elastic, cassandra,…) confluent platform: ksql + schema registry + control center + rest proxy (+ connect + streams) kafka from presto/hive .. 19 . 2
41. audio_pipeline.png 19 . 3
42. ACTION? getting hands dirty download kafka, start and play with topics produce text file - row by row RTFM streams: word-count 20 . 1
43. 😃 😃 😃 😃 😃 😃 😃 😃 😃 😃 😃 😃 21 . 1