Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
210
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Oracle AI Database@Google Cloud：サービス概要のご紹介
Avatar for oracle4engineer oracle4engineer
PRO
5
1.3k
40代からのアウトプット ― 経験は価値ある学びに変わる / 20260404 Naoki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
140
フルカイテン株式会社 エンジニア向け採用資料
Avatar for FULL KAITEN fullkaiten
0
11k
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
16
410k
TUNA Camp 2026 京都Stage ヒューリスティックアルゴリズム入門
Avatar for terry-u16 terryu16
0
640
RGBに陥らないために -プロダクトの価値を届けるまで-
Avatar for RightTouch righttouch
PRO
0
130
SaaSに宿る21g
Avatar for Yamakan kanyamaguc
2
180
Navigation APIと見るSvelteKitのWeb標準志向
Avatar for Okuto Oyama yamanoku
2
130
SaaSの操作主体は人間からAIへ - 経理AIエージェントが目指す深い自動化
Avatar for Motoshi Nishihira nishihira
0
120
CloudFrontのHost Header転送設定でパケットの中身はどう変わるのか？
Avatar for nagisa_53 nagisa53
1
230
Why we keep our community?
Avatar for Yasunobu Kawaguchi kawaguti
PRO
0
350
20260323_データ分析基盤でGeminiを使う話
Avatar for yuichi 1210yuichi0
0
210

Featured

See All Featured
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
440
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
260
SERP Conf. Vienna - Web Accessibility: Optimizing for Inclusivity and SEO
Avatar for Sara Fernández Carmona sarafernandez
2
1.4k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
432
67k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
287
14k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
6k
The innovator’s Mindset - 
Leading Through an Era of Exponential Change - McGill University 2025
Avatar for Jean-Marc De Jonghe jdejongh
PRO
1
140
職位にかかわらず全員がリーダーシップを発揮するチーム作り / Building a team where everyone can demonstrate leadership regardless of position
Avatar for MADOX madoxten
62
53k
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
1
2.1k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.6k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.2k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com