Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
OpenClawとHermesAgentでAI新入社員を作った話
Avatar for yanada takanoriyanada
0
120
AIが変えた"品質の守り方"
Avatar for kkakizaki kkakizaki
13
4.8k
TypeScriptはどのようにどこまで推論できるのか ─ とにかく as は禁止で
Avatar for ypresto ypresto
3
450
インフラが苦手でも大丈夫！ 紙芝居 Kubernetes -WWGT 10周年編-
Avatar for Aoi Takahashi aoi1
1
260
checker.tsにチキンレースを仕掛けてみた：型エラー(TS2589)が発生する境界線を求めて
Avatar for Hal hal_spidernight
1
210
NFLコンペ2026 解法
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
120
組織の中で自分を経営する技術
Avatar for shoota shoota
0
170
CloudFront VPCオリジンとVPC Latticeサービスの内部ALBをマルチアカウントで一元利用しよう
Avatar for k-kun duelist2020jp
5
240
Spring AI × MCP 入門〜AIエージェントへのツール公開、境界設計から始める最小構成 〜
Avatar for 宮本裕也 yuyamiyamoto
0
120
Copilot CLI・IDE・Web・スマホで途切れない開発フローを目指して / One Copilot flow - CLI IDE Web Mobile
Avatar for AEON aeonpeople
1
1.1k
Generative UI × A2UI で AI エージェントを作った話　AI-DLC も使ってみた！
Avatar for たけのこ kmiya84377
1
250
Javaコミュニティをもっと楽しむための9箇条
Avatar for Sugiyama Takaaki takasyou
0
270

Featured

See All Featured
Jess Joyce - The Pitfalls of Following Frameworks
Avatar for Tech SEO Connect techseoconnect
PRO
1
150
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.9k
How to Talk to Developers About Accessibility
Avatar for Jason CranfordTeague jct
2
210
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
370
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
830
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
1
330
KATA
Avatar for Megan Lloyd mclloyd
PRO
35
15k
Agile Actions for Facilitating Distributed Teams - ADO2019
Avatar for Mark Kilby mkilby
0
190
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.9k
Ruling the World: When Life Gets Gamed
Avatar for Sebastian Deterding codingconduct
0
240
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
The innovator’s Mindset - 
Leading Through an Era of Exponential Change - McGill University 2025
Avatar for Jean-Marc De Jonghe jdejongh
PRO
1
180

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com