Apache Kafka

Build solid architecture using apache kafka

Transcript

1. Apache Kafka Build a solid application architecture TEBOURBI Maher @mtebourbi
2. !
3. Client App Browser Web App DB
4. Cache Web App HDFS Search Graph DB …
5. Web App Search Graph DB ETL …
6. Ok Set X = 1 Set X = 1 Set X = 3 Set X = 3 X = 3 X = 1 Ok Ok Ok
7. Other drawbacks • Database schema evolution. • Impact of some bugs on data. • System debugging.
8. Change Change
9. Cache Web App Jobs MQ Search Graph DB … Same data
10. F(x) Raw Data Derived Data
11. Log to the rescue
12. DB 3 2 1 4 5 6 7 8 9 10 11 Command log Web App Search Cache
13. 3 2 1 4 5 6 7 8 9 10 11 Topic Producer Consumer Offset
14. 3 2 1 4 5 6 7 8 9 10 11 3 2 1 4 5 6 7 3 2 1 4 5 6 7 8 9 P1 P2 P3 Producer Topic Partitions :
15. P1 P2 P3 P4 C1 C2 C3 C4 C5 Consumer Group CG1 CG2
16. Pull vs Push
17. 3 2 1 4 5 6 7 8 9 10 11 Partition Consumer Offset ZooKeeper Topic DB Vs Vs
18. 3 2 1 4 5 6 7 8 9 10 11 1 2 1 3 4 5 4 3 1 6 9 F B C X K D H nil Y W V Key Value Offset Key Value Offset 4 5 2 1 6 9 7 6 2 9 10 11 H D B Y W V Compaction
19. App time window S1 S2 B1 B2 Rt. Services Batch Services
20. Advantages • Eventual consistency. • More agile and maintainable. • Scalability. • Fault tolerance.
21. Other use cases • Messaging • Website Activity Tracking • Metring • Stream processing
22. Some annoyances • Require Zookeeper. • Official java client API changes. • Log compaction not possible with compressed topics. • Partition limited to single machine.
23. Performance • Use of OS page cache. • Linux: sendfile() syscall : pagecache -> socket. • 1.1 Trillion Messages Per Day at Linkedin • 2 Million Writes Per Second: https:/ /goo.gl/acmi00 3 Servers : Intel Xeon 2.5 GHz processor with six cores Six 7200 RPM SATA drives 32GB of RAM 1Gb Ethernet Kafka version : 0.8.1
25. ?