Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
390
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
これがLambdaレス時代のChatOpsだ!実例で学ぶAmazon Q Developerカスタムアクション活用法
iwamot
PRO
6
940
Azure Well-Architected Framework入門
tomokusaba
1
350
Modern_Data_Stack最新動向クイズ_買収_AI_激動の2025年_.pdf
sagara
0
240
from Sakichi Toyoda to Agile
kawaguti
PRO
1
110
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
11
79k
成長自己責任時代のあるきかた/How to navigate the era of personal responsibility for growth
kwappa
4
310
[Keynote] What do you need to know about DevEx in 2025
salaboy
0
160
速習AGENTS.md:5分で精度を上げる "3ブロック" テンプレ
ismk
1
460
【Oracle Cloud ウェビナー】クラウド導入に「専用クラウド」という選択肢、Oracle AlloyとOCI Dedicated Region とは
oracle4engineer
PRO
3
130
ACA でMAGI システムを社内で展開しようとした話
mappie_kochi
1
310
AWS 잘하는 개발자 되기 - AWS 시작하기: 클라우드 개념부터 IAM까지
kimjaewook
0
130
Vibe Coding Year in Review. From Karpathy to Real-World Agents by Niels Rolland, CEO Paatch
vcoisne
0
120
Featured
See All Featured
Rails Girls Zürich Keynote
gr2m
95
14k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
BBQ
matthewcrist
89
9.8k
Scaling GitHub
holman
463
140k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
860
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Balancing Empowerment & Direction
lara
4
680
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
The World Runs on Bad Software
bkeepers
PRO
72
11k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com