Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
スクラムマスターが
スクラムチームに入って取り組む5つのこと
- スクラムガイドには書いてないけど入った当初から取り組んでおきたい大切なこと -
Avatar for scrummasudar scrummasudar
3
2.2k
迷わない！AI×MCP連携のリファレンスアーキテクチャ完全ガイド
Avatar for CData Software Japan cdataj
0
570
【Oracle Cloud ウェビナー】ランサムウェアが突く「侵入の隙」とバックアップの「死角」 ～ 過去の教訓に学ぶ — 侵入前提の防御とデータ保護 ～
Avatar for oracle4engineer oracle4engineer
PRO
0
140
Eight Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
6.3k
たかがボタン、されどボタン ～button要素から深ぼるボタンUIの定義について～ / BuriKaigi 2026
Avatar for Okuto Oyama yamanoku
1
280
モノタロウ x クリエーションラインで実現する チームトポロジーにおける プラットフォームチーム・ ストリームアラインドチームの 効果的なコラボレーション
Avatar for Creationline creationline
0
950
名刺メーカーDevグループ 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
1k
AWS Amplify Conference 2026 - 仕様からリリースまで一気通貫 生成 AI 時代のフルスタック開発
Avatar for Riku INADA inariku
1
190
Git Training GitHub
Avatar for Yuki Hattori yuhattor
1
130
手軽に作れる電卓を作って イベントソーシングに親しもう CQRS+ESカンファレンス2026
Avatar for Akinori Takigawa akinoriakatsuka
0
450
ファインディにおけるフロントエンド技術選定の歴史
Avatar for puku0x puku0x
2
1.5k
歴史から学ぶ、Goのメモリ管理基礎
Avatar for Takuto Nagami logica0419
14
2.8k

Featured

See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.1k
エンジニアに許された特別な時間の終わり
Avatar for watany watany
106
230k
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.3k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.4k
Agile Actions for Facilitating Distributed Teams - ADO2019
Avatar for Mark Kilby mkilby
0
100
Test your architecture with Archunit
Avatar for Yoan thirion
1
2.1k
Paper Plane
Avatar for Katie Cordina Lindsey katiecoart
PRO
0
45k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
7.9k
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
0
120
Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf
Avatar for Kenny Baas-Schwegler baasie
0
430
The innovator’s Mindset - 
Leading Through an Era of Exponential Change - McGill University 2025
Avatar for Jean-Marc De Jonghe jdejongh
PRO
1
82
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
34k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com