Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Realtime, Fault Tolerant Systems with kafka

Manan
August 28, 2016

Realtime, Fault Tolerant Systems with kafka

Presented at ThoughtWorks XConf '16 Bangalore, Gurgaon.

Manan

August 28, 2016
Tweet

More Decks by Manan

Other Decks in Technology

Transcript

  1. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS

    ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
  2. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS

    ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
  3. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS

    ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
  4. RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS

    ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
  5. POPULAR USE CASES • Data ingestion • Log aggregation •

    User activity tracking • Message Queue
  6. ARCHITECTURE PRODUCER PRODUCER PRODUCER PRODUCER PRODUCER BROKER BROKER BROKER ZOOKEEPER

    CONSUMER CONSUMER BROKER T1 P1 T2 P1 T1 P2 T1 P1 T1 P2 T2 P1
  7. • Replication • One leader. Multiple followers. • Leader ->

    Followers • Leader election FAULT TOLERANCE
  8. EXAMPLE APPLICATION Clicks Page Views Custom events {visitorId: “10239”, productId:

    “prd-18789”, …} SPARK STREAMING APPLICATION K A F K A Product - Products Visitor Recommendations USER FACING APPLICATION