$30 off During Our Annual Pro Sale. View Details »

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
pmconf2025 - 他社事例を"自社仕様化"する技術_iRAFT法
Avatar for yamachi｜@yamapiya_ daichi_yamashita
0
790
プロダクトマネージャーが押さえておくべき、ソフトウェア資産とAIエージェント投資効果 / pmconf2025
Avatar for Masato Ishigaki / 石垣雅人 i35_267
2
590
regrowth_tokyo_2025_securityagent
Avatar for h-ashisan hiashisan
0
180
Oracle Technology Night #95 GoldenGate 26ai の実装に迫る1
Avatar for oracle4engineer oracle4engineer
PRO
0
150
[JAWS-UG 横浜支部 #91]DevOps Agent vs CloudWatch Investigations -比較と実践-
Avatar for sh_fk2 sh_fk2
1
240
「Managed Instances」と「durable functions」で広がるAWS Lambdaのユースケース
Avatar for Lamaglama39 lamaglama39
0
280
WordPress は終わったのか ～今のWordPress の制作手法ってなにがあんねん？～ / Is WordPress Over? How We Build with WordPress Today
Avatar for tbshiki tbshiki
1
480
新 Security HubがついにGA！仕組みや料金を深堀り #AWSreInvent #regrowth / AWS Security Hub Advanced GA
Avatar for MasahiroKawahara masahirokawahara
1
1.5k
第4回 「メタデータ通り」　リアル開催
Avatar for データ横丁 datayokocho
0
120
Kiro Autonomous AgentとKiro Powers の紹介 / kiro-autonomous-agent-and-powers
Avatar for tomoki10 tomoki10
0
330
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
11
390k
文字列の並び順 / Unicode Collation
Avatar for とみたまさひろ tmtms
0
130

Featured

See All Featured
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
35
3.3k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.3k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
180
10k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
2.9k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
355
21k
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
0
500
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
9.8k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com