Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Google Cloud Next '26 の裏でこっそりリリースされたCloud Number Registry & Cloud Hub コスト分析 を試してみた
Avatar for hikaru hikaru1001
0
150
Fabric MCPの紹介と使い分け
Avatar for Ryoma Nagata ryomaru0825
1
110
AgentCore×VPCでの設計パターンn選と勘所
Avatar for Har1101 har1101
4
370
Microsoft 365 / Microsoft 365 Copilot : 自分の状態を確認する「ラベル」について
Avatar for Taichi Nakamura taichinakamura
0
440
（きっとたぶん）人材育成や教育のような何かの話
Avatar for Takanori Sejima sejima
0
190
Oracle Cloud Infrastructure：2026年4月度サービス・アップデート
Avatar for oracle4engineer oracle4engineer
PRO
0
280
世界の中心でApp Runnerを叫ぶ FINAL
Avatar for つくぼし tsukuboshi
0
210
MySQL 9.7がやってきた ～これまでのあらすじと基本情報～ @ 日本MySQLユーザ会会2026年04月 / mysql97-yattekita
Avatar for sakaik sakaik
0
160
プラットフォームエンジニアリングの実践 - AWS コンテナサービスで構築する社内プラットフォーム / AWS Containers Platform Meetup #1
Avatar for Masatoshi Hayashi literalice
1
240
古今東西SRE
Avatar for Kaoru okaru
1
110
GitHub Copilot Dev Days
Avatar for tomokusaba tomokusaba
0
140
AI活用時代の事業判断高度化を導くエンジニアリング基盤 / 20260424 Atsushi Funahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
2
120

Featured

See All Featured
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.6k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
110
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
65
54k
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
1k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
2
1k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
250
1.3M
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
130
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
0
270
Winning Ecommerce Organic Search in an AI Era - #searchnstuff2025
Avatar for Aleyda Solis aleyda
1
2k
Fireside Chat
Avatar for paigeccino paigeccino
42
3.9k
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
490

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com