Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
390
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
FOSS4G 2025 KANSAI QGISで点群データをいろいろしてみた
kou_kita
0
410
AWS CDKの仕組み / how-aws-cdk-works
gotok365
9
420
Delta airlines Customer®️ USA Contact Numbers: Complete 2025 Support Guide
deltahelp
0
970
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
ymrl
2
250
SRE不在の開発チームが障害対応と 向き合った100日間 / 100 days dealing with issues without SREs
shin1988
1
480
american aa airlines®️ USA Contact Numbers: Complete 2025 Support Guide
aaguide
0
400
ABEMAの本番環境負荷試験への挑戦
mk2taiga
5
370
【あのMCPって、どんな処理してるの?】 AWS CDKでの開発で便利なAWS MCP Servers特集
yoshimi0227
6
430
Getting to Know Your Legacy (System) with AI-Driven Software Archeology (WeAreDevelopers World Congress 2025)
feststelltaste
1
160
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
460
AWS認定を取る中で感じたこと
siromi
1
220
データ基盤からデータベースまで?広がるユースケースのDatabricksについて教えるよ!
akuwano
3
140
Featured
See All Featured
Adopting Sorbet at Scale
ufuk
77
9.5k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Designing for humans not robots
tammielis
253
25k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
20k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
980
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
6
310
Facilitating Awesome Meetings
lara
54
6.4k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com