Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
StrandsとNeptuneを使ってナレッジグラフを構築する
Avatar for やくも yakumo
1
120
茨城の思い出を振り返る ～CDKのセキュリティを添えて～ / 20260201 Mitsutoshi Matsuo
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
320
配列に見る bash と zsh の違い
Avatar for kazzpapa3 kazzpapa3
3
160
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
15
400k
Embedded SREの終わりを設計する　「なんとなく」から計画的な自立支援へ
Avatar for SansanTech sansantech
PRO
3
2.5k
CDK対応したAWS DevOps Agentを試そう_20260201
Avatar for モブエンジニア（Masaki Okuda） masakiokuda
1
330
制約が導く迷わない設計 〜 信頼性と運用性を両立するマイナンバー管理システムの実践 〜
Avatar for ないとー bwkw
3
960
Sansan Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
3.9k
SREじゃなかった僕らがenablingを通じて「SRE実践者」になるまでのリアル / SRE Kaigi 2026
Avatar for AEON aeonpeople
6
2.5k
We Built for Predictability; The Workloads Didn’t Care
Avatar for Michael Stahnke stahnma
0
140
クレジットカード決済基盤を支えるSRE - 厳格な監査とSRE運用の両立 (SRE Kaigi 2026)
Avatar for capytan capytan
6
2.8k
ブロックテーマ、WordPress でウェブサイトをつくるということ / 2026.02.07 Gifu WordPress Meetup
Avatar for Toro_Unit (Hiroshi Urabe) torounit
0
190

Featured

See All Featured
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
32k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
4.6k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
24k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
27k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
698
190k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
6.3k
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
240
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
71k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
34k
B2B Lead Gen: Tactics, Traps & Triumph
Avatar for Sophie Logan marketingsoph
0
55
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
2.3M

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com