Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
FOSS4G 2025 KANSAI　QGISで点群データをいろいろしてみた
Avatar for kou_kita kou_kita
0
410
AWS CDKの仕組み / how-aws-cdk-works
Avatar for k.goto gotok365
9
420
Delta airlines Customer®️ USA Contact Numbers: Complete 2025 Support Guide
Avatar for michal deltahelp
0
970
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
Avatar for ymrl ymrl
2
250
SRE不在の開発チームが障害対応と 向き合った100日間 / 100 days dealing with issues without SREs
Avatar for katsukamaru shin1988
1
480
american aa airlines®️ USA Contact Numbers: Complete 2025 Support Guide
Avatar for jackal5647 aaguide
0
400
ABEMAの本番環境負荷試験への挑戦
Avatar for Miyazaki Taiga mk2taiga
5
370
【あのMCPって、どんな処理してるの？】 AWS CDKでの開発で便利なAWS MCP Servers特集
Avatar for yoshimi0227 yoshimi0227
6
430
Getting to Know Your Legacy (System) with AI-Driven Software Archeology (WeAreDevelopers World Congress 2025)
Avatar for Markus Harrer feststelltaste
1
160
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
Avatar for FUJIWARA Shunichiro fujiwara3
3
460
AWS認定を取る中で感じたこと
Avatar for Siromi siromi
1
220
データ基盤からデータベースまで？広がるユースケースのDatabricksについて教えるよ！
Avatar for Akihiro Kuwano akuwano
3
140

Featured

See All Featured
Adopting Sorbet at Scale
Avatar for Ufuk Kayserilioglu ufuk
77
9.5k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
695
190k
Designing for humans not robots
Avatar for Tammie Lister tammielis
253
25k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
35
2.4k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.1k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
351
20k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
48
2.9k
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
54
13k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
18
980
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
77
5.9k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
6
310
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
54
6.4k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com