Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
430
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Pavlokで始める電撃駆動開発
Avatar for 佐藤陽威 sgrsn
0
170
Agent 時代の Kaggle 展望 / kaggle-in-the-agentic-era
Avatar for Shotaro Ishihara upura
0
490
AIエージェントを前提としたプラットフォーム エンジニアリング：GKEで作るAgent-Ready Golden Path
Avatar for LegalOn Technologies, Inc legalontechnologies
PRO
1
200
PLaMo 3.0 Primeの構造化出力サポート
Avatar for Preferred Networks pfn
PRO
0
190
制約理論（ToC）入門　2026版
Avatar for Recruit recruitengineers
PRO
5
1.6k
攻撃と防御で学ぶAI時代のプロダクトセキュリティ演習
Avatar for Recruit recruitengineers
PRO
1
430
[MIRU26] Open-Vocabulary Intention-Guided Object Detection in Diverse Scenes
Avatar for Semantic Machine Intelligence Lab., Keio Univ. keio_smilab
PRO
0
140
20260807_第6回_関東kaggler会LT_claw系bot xangiと始める、"寂しくない" kaggle
Avatar for suguuuuuすぐー sugupoko
0
140
Master Dataグループ紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
4.8k
toio・myCobotでフィジカルAIっぽいことを行うための検討（とりあえず調査） / フィジカルAI LT（IoTLTによる開催）
Avatar for you（@youtoy） you
PRO
0
290
Forza Horizon 6 のテレメトリ機能で 自動運転に使えそうな学習データを集める話
Avatar for henjin0 henjin0
0
130
【CEDEC2026】次世代デジタルカードゲームのサーバー設計と運用 〜『Shadowverse: Worlds Beyond』の舞台裏～
Avatar for Cygames, Inc. cygames
PRO
0
910

Featured

See All Featured
The Impact of AI in SEO - AI Overviews June 2024 Edition
Avatar for Aleyda Solis aleyda
5
1.2k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
680
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
35k
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
430
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
560
Heart Work Chapter 1 - Part 1
Avatar for Lake Fama lfama
PRO
8
36k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
12
1.2k
Building the Perfect Custom Keyboard
Avatar for Naoto Takai takai
2
830
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
56k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
3k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
Avatar for Greg Gifford greggifford
PRO
0
250
WCS-LA-2024
Avatar for Leonardo Collado-Torres lcolladotor
0
790

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com