Tweet
Tweet

Slide 1

Slide 1 text

Manan Bharara Abhishek Khare Throughput REALTIME, FAULT TOLERANT SYSTEMS WITH KAFKA

Slide 2

Slide 2 text

Manan Bharara Abhishek Khare Throughput REALTIME, FAULT TOLERANT SYSTEMS WITH KAFKA

Slide 3

Slide 3 text

MONOLITH tail / grep / more / less RATIONALE

Slide 4

Slide 4 text

RATIONALE

Slide 5

Slide 5 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE

Slide 6

Slide 6 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE

Slide 7

Slide 7 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE AGGREGATE

Slide 8

Slide 8 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS MICROSERVICE AGGREGATE

Slide 9

Slide 9 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE

Slide 10

Slide 10 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE

Slide 11

Slide 11 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE

Slide 12

Slide 12 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER

Slide 13

Slide 13 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER

Slide 14

Slide 14 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER

Slide 15

Slide 15 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER

Slide 16

Slide 16 text

RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS ACTIVITY LOGS Clicks Page views MICROSERVICE DB K A F K A

Slide 17

Slide 17 text

POPULAR USE CASES • Data ingestion • Log aggregation • User activity tracking • Message Queue

Slide 18

Slide 18 text

ARCHITECTURE PRODUCER PRODUCER PRODUCER PRODUCER PRODUCER BROKER BROKER BROKER ZOOKEEPER CONSUMER CONSUMER BROKER T1 P1 T2 P1 T1 P2 T1 P1 T1 P2 T2 P1

Slide 19

Slide 19 text

• Replication • One leader. Multiple followers. • Leader -> Followers • Leader election FAULT TOLERANCE

Slide 20

Slide 20 text

FAULT TOLERANCE DEMO

Slide 21

Slide 21 text

SCALABILITY

Slide 22

Slide 22 text

• Horizontal Scalability SCALABILITY

Slide 23

Slide 23 text

• Horizontal Scalability • Partition Reassignment SCALABILITY

Slide 24

Slide 24 text

EXAMPLE APPLICATION Clicks Page Views Custom events {visitorId: “10239”, productId: “prd-18789”, …} SPARK STREAMING APPLICATION K A F K A Product - Products Visitor Recommendations USER FACING APPLICATION

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content