Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
370
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
これまでの計測・開発・デプロイ方法全部見せます! / Findy ISUCON 2024-11-14
tohutohu
3
370
社内で最大の技術的負債のリファクタリングに取り組んだお話し
kidooonn
1
550
AGIについてChatGPTに聞いてみた
blueb
0
130
IBC 2024 動画技術関連レポート / IBC 2024 Report
cyberagentdevelopers
PRO
1
110
Evangelismo técnico: ¿qué, cómo y por qué?
trishagee
0
360
TypeScript、上達の瞬間
sadnessojisan
46
13k
Amplify Gen2 Deep Dive / バックエンドの型をいかにしてフロントエンドへ伝えるか #TSKaigi #TSKaigiKansai #AWSAmplifyJP
tacck
PRO
0
390
サイバーセキュリティと認知バイアス:対策の隙を埋める心理学的アプローチ
shumei_ito
0
390
複雑なState管理からの脱却
sansantech
PRO
1
150
強いチームと開発生産性
onk
PRO
35
11k
【Startup CTO of the Year 2024 / Audience Award】アセンド取締役CTO 丹羽健
niwatakeru
0
1.3k
Lambda10周年!Lambdaは何をもたらしたか
smt7174
2
110
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
65
11k
Faster Mobile Websites
deanohume
305
30k
The Language of Interfaces
destraynor
154
24k
How GitHub (no longer) Works
holman
310
140k
Imperfection Machines: The Place of Print at Facebook
scottboms
265
13k
Become a Pro
speakerdeck
PRO
25
5k
Making Projects Easy
brettharned
115
5.9k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
6.9k
Building Better People: How to give real-time feedback that sticks.
wjessup
364
19k
Rails Girls Zürich Keynote
gr2m
94
13k
The Invisible Side of Design
smashingmag
298
50k
Gamification - CAS2011
davidbonilla
80
5k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com