Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
経験がないことを言い訳にしない、 AI時代の他領域への染み出し方
Avatar for Yuki Harayama parayama0625
0
140
Data Engineering Study#30 LT資料
Avatar for tetsuroito tetsuroito
1
560
東京海上日動におけるセキュアな開発プロセスの取り組み
Avatar for miyabit miyabit
0
130
本当にわかりやすいAIエージェント入門
Avatar for segavvy segavvy
10
5.9k
Jitera Company Deck / JP
Avatar for Jitera Inc. jitera
0
140
組織内、組織間の資産保護に必要なアイデンティティ基盤と関連技術の最新動向
Avatar for Naohiro Fujie fujie
0
510
DatabricksのOLTPデータベース『Lakebase』に詳しくなろう！
Avatar for Takeru Ino inoutk
0
110
PHPからはじめるコンピュータアーキテクチャ / From Scripts to Silicon: A Journey Through the Layers of Computing
Avatar for HASEGAWA Tomoki tomzoh
2
380
20150719_Amazon Nova Canvas Virtual try-onアプリ 作成裏話
Avatar for Rika.I riz3f7
0
130
BEYOND THE RAG🚀 ~とりあえずRAG？を超えていけ！ 本当に使えるAIエージェント＆生成AIプロダクトを目指して~ / BEYOND-THE-RAG-Toward Practical-GenerativeAI-Products-AOAI-DevDay-2025
Avatar for Junya Miyake jnymyk
4
230
M365アカウント侵害時の初動対応
Avatar for LHazy lhazy
7
4.5k
エンジニアリングマネージャー“お悩み相談”パネルセッション
Avatar for ar_tama ar_tama
1
650

Featured

See All Featured
Making Projects Easy
Avatar for Brett Harned brettharned
116
6.3k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
53
2.9k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
40
1.9k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
2.9k
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
160
23k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
1
490
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
351
21k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
282
13k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com