Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
390
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
11k
Other Decks in Technology
See All in Technology
生成AI時代 文字コードを学ぶ意義を見出せるか?
hrsued
1
730
Model Mondays S2E03: SLMs & Reasoning
nitya
0
240
OpenHands🤲にContributeしてみた
kotauchisunsun
1
500
WordPressから ヘッドレスCMSへ! Storyblokへの移行プロセス
nyata
0
340
なぜ私はいま、ここにいるのか? #もがく中堅デザイナー #プロダクトデザイナー
bengo4com
0
1.3k
rubygem開発で鍛える設計力
joker1007
2
270
2025-06-26_Lightning_Talk_for_Lightning_Talks
_hashimo2
2
110
事業成長の裏側:エンジニア組織と開発生産性の進化 / 20250703 Rinto Ikenoue
shift_evolve
PRO
1
240
Core Audio tapを使ったリアルタイム音声処理のお話
yuta0306
0
150
Liquid Glass革新とSwiftUI/UIKit進化
fumiyasac0921
0
300
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
shift_evolve
PRO
1
140
自律的なスケーリング手法FASTにおけるVPoEとしてのアカウンタビリティ / dev-productivity-con-2025
yoshikiiida
0
390
Featured
See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
800
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Facilitating Awesome Meetings
lara
54
6.4k
Testing 201, or: Great Expectations
jmmastey
42
7.6k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.3k
Visualization
eitanlees
146
16k
The Invisible Side of Design
smashingmag
300
51k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Bash Introduction
62gerente
614
210k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com