Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
210
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
vLLM Community Meetup Tokyo #3 オープニングトーク
Avatar for jpishikawa jpishikawa
0
320
スクリプトの先へ！AIエージェントと組み合わせる モバイルE2Eテスト
Avatar for Ryo WATANABE error96num
0
160
[JAWSDAYS2026]Who is responsible for IAM
Avatar for Mizuki TSUJI mizukibbb
0
440
聲の形にみるアクセシビリティ
Avatar for tomokusaba tomokusaba
0
170
AI は "道具" から "同僚" へ 自律型 AI エージェントの最前線と、AI 時代の人材の在り方 / Colleague in the AI Era - Autonomous AI Seminar 2026 at Niigata
Avatar for geeawa gawa
0
110
OpenClawで回す組織運営
Avatar for Kazuto Kusama jacopen
3
690
マルチロールEMが実践する「組織のレジリエンス」を高めるための組織構造と人材配置戦略
Avatar for coconala_engineer coconala_engineer
3
720
IBM Bobを使って、PostgreSQLのToDoアプリをDb2へ変換してみよう/202603_Dojo_Bob
Avatar for Mayumi Hirano mayumihirano
1
320
20260311 ビジネスSWG活動報告（デジタルアイデンティティ人材育成推進WG Ph2 活動報告会）
Avatar for OpenID Foundation Japan oidfj
0
260
NewSQL_ ストレージ分離と分散合意を用いたスケーラブルアーキテクチャ
Avatar for hacomono Inc. hacomono
PRO
2
250
Kubernetesにおける推論基盤
Avatar for ry ry
1
310
事例に見るスマートファクトリーへの道筋〜工場データをAI Readyにする実践ステップ〜
Avatar for 濱田孝治 hamadakoji
1
290

Featured

See All Featured
KATA
Avatar for Megan Lloyd mclloyd
PRO
35
15k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
250
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
2.6M
AI in Enterprises - Java and Open Source to the Rescue
Avatar for ivargrimstad ivargrimstad
0
1.2k
The Mindset for Success: Future Career Progression
Avatar for Greg Gifford greggifford
PRO
0
270
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
55
12k
Done Done
Avatar for chrislema chrislema
186
16k
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
30k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
408
66k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
2
980
More Than Pixels: Becoming A User Experience Designer
Avatar for Michelle Schulp Hunt marktimemedia
3
350

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com