Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Streaming Ingestion & Processing at Flipkart
Search
Siddhartha Reddy
May 15, 2015
Technology
0
340
Streaming Ingestion & Processing at Flipkart
Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Siddhartha Reddy
May 15, 2015
Tweet
Share
More Decks by Siddhartha Reddy
See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
sids
1
170
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
sids
6
10k
Other Decks in Technology
See All in Technology
20分で完全に理解するGrafanaダッシュボード
hamadakoji
3
630
データベース02: データベースの概念
trycycle
0
160
require(ESM)とECMAScript仕様
uhyo
3
670
Java EE/Jakarta EEの現状と将来―クラウドネイティブ時代にJava EEは対応できるのか?―
takakiyo
1
160
MySQL の SQL クエリチューニングの要所を掴む勉強会
andpad
3
6.3k
GrafanaMeetup_AmazonManagedGrafanaのアクセス制御機能とマルチテナント環境下でのアクセス制御について
daitak
0
230
生成AIの変革の時代に、直近1年で直面した課題とその解決策
ktc_wada
0
200
検証を通して見えてきたTiDBの性能特性
lycorptech_jp
PRO
6
3.8k
Google Cloud Next '24 Recap(Cloud Run/k8s)
mokocm
0
210
LLM開発・活用の舞台裏@2024.04.25
yushin_n
1
210
Google Cloud の AI を支える裏側のインフラを垣間見る!
maroon1st
0
350
本当のAWS基礎
toru_kubota
0
520
Featured
See All Featured
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
227
16k
Imperfection Machines: The Place of Print at Facebook
scottboms
260
12k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
21
1.6k
What the flash - Photography Introduction
edds
64
11k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
187
16k
Fontdeck: Realign not Redesign
paulrobertlloyd
76
4.9k
The Straight Up "How To Draw Better" Workshop
denniskardys
227
130k
The Invisible Side of Design
smashingmag
294
49k
Docker and Python
trallard
34
2.7k
Robots, Beer and Maslow
schacon
PRO
155
7.9k
Statistics for Hackers
jakevdp
789
220k
Transcript
Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids
Flipkart Data Platform (an oversimplified view)
Streaming Ingestion
Choices • push, not pull • schemas & validations
Streaming Ingestion v1.0
None
• Push 㱺 accountability (with source teams) • good call!
• Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data
Streaming Ingestion v2.0
Stream Processing
An Example
Streaming Joins: Example It works! But… how do we deal
with lookup failures?
Streaming Joins: Handling Failures
None
None
Streaming Joins: Bootstrapping With a little help from MR friends
Streaming Joins: But… The example that doesn’t really work correctly
Streaming Joins
In summary • Streaming Ingestion: push, schemas & validation, HTTP
service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com