Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
440
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
WAF 運用改善の承認サイクル/SRE_BizReach_MIXI_1
Avatar for Visional Engineering ＆ Design visional_engineering_and_design
0
150
プロダクトエンジニアに必要な「いい感じ」に作る能力 〜たくさん作れる時代に、どこまで作るかの決め方〜
Avatar for Jotaro Nishime jnishime_dresscode
2
1.4k
設計の世代交代を乗り越える、14年続くAndroidアプリの開発戦略
Avatar for SansanTech sansantech
PRO
1
150
Nav2、Nav3 ... はたまた自作？〜 作って理解する Nav3 の設計意図 〜 / Nav2, Nav3 ... or Build Your Own? — Understanding Nav3's design intent by building it from scratch
Avatar for Yuki Anzai yanzm
0
250
AIに丸投げしないトイル削減 / Eliminating Toil Without Leaving It All to AI
Avatar for kohbis kohbis
2
600
書籍『生成AIの安全性入門』の入門
Avatar for Koki Wataoka wataoka
0
220
Sony-DroidKaigi2026
Avatar for ソニー株式会社 sony
1
320
KAEN Company Deck
Avatar for 株式会社KAEN kaen
PRO
0
320
enechainの内製セルフサービスプラットフォーム
Avatar for Tomoya Usami hiyosi
0
120
多摩川(.dev)ランニング入門 / Tamagawa.dev#3
Avatar for FUJIWARA Shunichiro fujiwara3
3
380
Backstageでつくるセルフサービスな社内開発基盤
Avatar for Sakazaki Kikunosuke kikunosuke75
1
340
動画配信アプリでの Engage SDK 導入 — TVer Android が Play ストアにコンテンツを届けるまで
Avatar for TVer Inc. techtver
PRO
0
120

Featured

See All Featured
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.6k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
35
2.6k
Code Review Best Practice
Avatar for Trisha Gee trishagee
74
20k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
7k
Designing for Performance
Avatar for Lara Hogan lara
611
70k
Agile Leadership in an Agile Organization
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
220
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8.3k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
6.3k
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
810
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
270
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
3k
What’s in a name? Adding method to the madness
Avatar for The Alliance productmarketing
PRO
24
4.2k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com