Realtime, Fault Tolerant Systems with kafka

Presented at ThoughtWorks XConf '16 Bangalore, Gurgaon.

Transcript

1. Manan Bharara Abhishek Khare Throughput REALTIME, FAULT TOLERANT SYSTEMS WITH KAFKA
2. Manan Bharara Abhishek Khare Throughput REALTIME, FAULT TOLERANT SYSTEMS WITH KAFKA
3. MONOLITH tail / grep / more / less RATIONALE
4. RATIONALE
5. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE
6. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE
7. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE AGGREGATE
8. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS MICROSERVICE AGGREGATE
9. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE
10. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE
11. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE
12. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
13. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
14. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
15. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
16. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE DB K A F K A
17. POPULAR USE CASES • Data ingestion • Log aggregation • User activity tracking • Message Queue
18. ARCHITECTURE PRODUCER PRODUCER PRODUCER PRODUCER PRODUCER BROKER BROKER BROKER ZOOKEEPER CONSUMER CONSUMER BROKER T1 P1 T2 P1 T1 P2 T1 P1 T1 P2 T2 P1
19. • Replication • One leader. Multiple followers. • Leader -> Followers • Leader election FAULT TOLERANCE
20. FAULT TOLERANCE DEMO
21. SCALABILITY
22. • Horizontal Scalability SCALABILITY
23. • Horizontal Scalability • Partition Reassignment SCALABILITY
24. EXAMPLE APPLICATION Clicks Page Views Custom events {visitorId: “10239”, productId: “prd-18789”, …} SPARK STREAMING APPLICATION K A F K A Product - Products Visitor Recommendations USER FACING APPLICATION