Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
共有と分離 - Compose Multiplatform "本番導入" の設計指針
Avatar for Ryo WATANABE error96num
1
370
AIのグローバルトレンド2025 #scrummikawa / global ai trend
Avatar for kyonmm kyonmm
PRO
1
270
サラリーマンの小遣いで作るtoCサービス - Cloudflare Workersでスケールする開発戦略
Avatar for shinaps shinaps
2
410
【初心者向け】ローカルLLMの色々な動かし方まとめ
Avatar for Aratako aratako
7
3.4k
KotlinConf 2025_イベントレポート
Avatar for ソニー株式会社 sony
1
120
品質視点から考える組織デザイン/Organizational Design from Quality
Avatar for miisan mii3king
0
200
allow_retry と Arel.sql / allow_retry and Arel.sql
Avatar for Shintani Teppei euglena1215
1
160
複数サービスを支える マルチテナント型 Batch MLプラットフォーム
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
300
La gouvernance territoriale des données grâce à la plateforme Terreze
Avatar for BlueHats bluehats
0
150
Autonomous Database - Dedicated 技術詳細 / adb-d_technical_detail_jp
Avatar for oracle4engineer oracle4engineer
PRO
4
10k
AWSを利用する上で知っておきたい名前解決のはなし（10分版）
Avatar for nagisa_53 nagisa53
10
3k
Rustから学ぶ 非同期処理の仕組み
Avatar for skanehira skanehira
1
130

Featured

See All Featured
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
96
6.2k
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
272
27k
It's Worth the Effort
Avatar for 3n 3n
187
28k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
140
7.1k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
236
140k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
36
2.5k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.5k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.9k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.1k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.2k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
3
620
KATA
Avatar for Megan Lloyd mclloyd
32
14k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com