Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
440
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Software Supply Chain Attackからクラウド環境を守るためにできること
Avatar for LHazy lhazy
2
290
名刺メーカーDevグループ 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
1.2k
MulticaとPi Coding Agentで、小規模OSSを30本同時運用した流れ
Avatar for eiei114 eiei114
0
130
AI・HPC開発を支えるGPU環境の新しい選択肢 液冷GPUシステム「AquSys」の取り組み
Avatar for GPU UNITE gpuunite_official
0
170
MIXIで活躍できるエンジニアを 若手社員目線で考えてみる
Avatar for MIXI ENGINEERS mixi_engineers
PRO
0
110
Model Studio CLI × Token Plan
Avatar for 尾関将樹 | めい maigo999
0
200
AI for Science時代を切り開く、政府の次世代HPC戦略の展望
Avatar for GPU UNITE gpuunite_official
0
130
Bill One 開発エンジニア 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
7
20k
同じWAFが、攻撃の“形”は弾く── 正当な“形”の不正は通す
Avatar for kuroneko kuroneko13
0
260
Genie Codeハンズオン応用編
Avatar for Takaaki Yayoi taka_aki
0
130
長期運営で肥大化したExcelマスターデータの解消に向けた移行事例
Avatar for gree_tech gree_tech
PRO
0
100
AIペネトレーションテスト・ セキュリティ検証「AgenticSec」紹介資料
Avatar for Sho Nakatani laysakura
2
9.3k

Featured

See All Featured
The Language of Interfaces
Avatar for Des Traynor destraynor
162
27k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
408
67k
AI: The stuff that nobody shows you
Avatar for John Nunemaker jnunemaker
PRO
9
930
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
6.2k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
1k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8.3k
Lightning talk: Run Django tests with GitHub Actions
Avatar for Sarah Abderemane sabderemane
0
240
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2.3k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.4k
Between Models and Reality
Avatar for mayunak mayunak
4
400

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com