Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
430
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
SRE Lounge Hiroshimaへの招待
Avatar for gr1m0h grimoh
0
250
知見・人・API・DB・予算 ─ ナイナイ尽くしだった人事データ整備 with dbt、5年間の学び
Avatar for ken ken6377
1
150
アカウントが増えてからでは遅い? ~ マルチアカウント統制の勘所 ~
Avatar for KenichiNakamura kenichinakamura
0
140
『AIに負けない』より『AIと遊ぶ』」〜ワクワクが最強のテスト・QA学習戦略_公開用
Avatar for odan tomohiro odan611
1
400
組織における AI-DLC 実践
Avatar for ASKUL Engineer askul
0
310
Kotlin 開発のツラミを爆破した話！ / Explode the difficulty of Kotlin dev!
Avatar for Kengo TODA eller86
0
150
クラウド上のデータ復旧で見落としがちな制約: 医療系 SaaS の BCP 設計から得た教訓
Avatar for KAKEHASHI kakehashi
PRO
0
210
金融の未来を考える / Thinking About the Future of Finance
Avatar for Kenji Saito ks91
PRO
0
150
Mastraエージェント、どのクラウドにデプロイする？
Avatar for みのるん minorun365
PRO
2
170
AIに「使われる」時代のSaaS戦略 〜既存WebAPIのMCPサーバー化における開発ノウハウ〜
Avatar for 駅すぱあと API ekispert_api
0
290
Foxgloveについて 実際にExtensionを開発して公開するまでの話 / About Foxglove: The Story of Developing and Releasing an Extension
Avatar for Ryo Kabutan ry0_ka
0
150
事業価値を⽣み出すSREへ SREが担うべき意思決定の5層
Avatar for kenta_hi kenta_hi
0
350

Featured

See All Featured
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
260
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
460
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
260
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
3
430
How to Talk to Developers About Accessibility
Avatar for Jason CranfordTeague jct
2
280
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
480
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.2k
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
Visualization
Avatar for Eitan Lees eitanlees
152
17k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
12
1.2k
sira's awesome portfolio website redesign presentation
Avatar for ElsiraPls elsirapls
0
290
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
440

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com