Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Realtime, Fault Tolerant Systems with kafka
Search
Manan
August 28, 2016
Technology
0
98
Realtime, Fault Tolerant Systems with kafka
Presented at ThoughtWorks XConf '16 Bangalore, Gurgaon.
Manan
August 28, 2016
Tweet
Share
More Decks by Manan
See All by Manan
Accelerated growth through incentive-less referrals
manan
0
49
Necessary tooling and monitoring for performance critical applications - RootConf 2017
manan
0
310
Necessary tooling and monitoring for performance critical applications
manan
1
840
Needle in a haystack - Advanced DevTools usage
manan
0
120
Optimizing JS Apps to increase performance
manan
0
170
JavaScript DevOPS - Automating everything JS
manan
1
1.4k
Offline website testing
manan
0
63
Memory optimizations for your JS app to increase performance
manan
0
200
Offline in the browser with IndexedDB, AppCache and LocalStorage
manan
0
1.3k
Other Decks in Technology
See All in Technology
Ask-LLM論文紹介: How to Train Data-Efficient LLMs
s_ota
0
110
データ品質をコード化! LINEヤフーのMLOpsを最適化する "ACP Data Quality" の紹介
lycorptech_jp
PRO
2
160
サービスメッシュ環境における OpenTelemetry 活用 / OpenTelemetry in Service Mesh
k6s4i53rx
2
830
現実世界の事象から学ぶSOLID原則
h0r15h0
24
10k
ビジネスとコード品質の接合点 そしてコード品質がそこに及ぼす影響 / The Intersections of Business and Engineering, and The Impact of Code Quality There
mtx2s
10
1k
中央集権体制からDataOpsへの転換 / centralized-to-dataops-transformation
pei0804
7
1.4k
Tohoku.Tech #1 「Cursorを使ったRaspberry Piの開発」by ねこまた
jun2882
0
250
Vos logs méritent mieux que la config par défaut
lyrixx
2
310
やっていきテスト
k6s4i53rx
0
160
大規模なアジャイル開発の現場と技術負債 / Technical Debt
yoshiitaka
20
4k
Introduction to Jetpack Compose
pohjus
1
110
HoneycombとOpenTelemetryでオブザーバビリティに入門してみる
sumiren
2
150
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
1
1.2k
Testing 201, or: Great Expectations
jmmastey
27
6.3k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
124
32k
The Invisible Customer
myddelton
114
12k
Documentation Writing (for coders)
carmenintech
59
3.8k
How to Ace a Technical Interview
jacobian
272
22k
Product Roadmaps are Hard
iamctodd
43
9.6k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
240
1.2M
Building Better People: How to give real-time feedback that sticks.
wjessup
350
18k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
19
1.9k
Fireside Chat
paigeccino
19
2.6k
RailsConf 2023
tenderlove
0
510
Transcript
Manan Bharara Abhishek Khare Throughput REALTIME, FAULT TOLERANT SYSTEMS WITH
KAFKA
Manan Bharara Abhishek Khare Throughput REALTIME, FAULT TOLERANT SYSTEMS WITH
KAFKA
MONOLITH tail / grep / more / less RATIONALE
RATIONALE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE AGGREGATE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS MICROSERVICE
AGGREGATE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS ACTIVITY
LOGS Clicks Page views MICROSERVICE AGGREGATE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HDFS ACTIVITY
LOGS Clicks Page views MICROSERVICE AGGREGATE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS
ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS
ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS
ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS
ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS
ACTIVITY LOGS Clicks Page views MICROSERVICE AGGREGATE AGGREGATE AGGREGATE FILTER
RATIONALE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE MICROSERVICE HBASE HDFS
ACTIVITY LOGS Clicks Page views MICROSERVICE DB K A F K A
POPULAR USE CASES • Data ingestion • Log aggregation •
User activity tracking • Message Queue
ARCHITECTURE PRODUCER PRODUCER PRODUCER PRODUCER PRODUCER BROKER BROKER BROKER ZOOKEEPER
CONSUMER CONSUMER BROKER T1 P1 T2 P1 T1 P2 T1 P1 T1 P2 T2 P1
• Replication • One leader. Multiple followers. • Leader ->
Followers • Leader election FAULT TOLERANCE
FAULT TOLERANCE DEMO
SCALABILITY
• Horizontal Scalability SCALABILITY
• Horizontal Scalability • Partition Reassignment SCALABILITY
EXAMPLE APPLICATION Clicks Page Views Custom events {visitorId: “10239”, productId:
“prd-18789”, …} SPARK STREAMING APPLICATION K A F K A Product - Products Visitor Recommendations USER FACING APPLICATION
None
None
None
None
None