Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
膨大なデータをどうさばく？ Java × MQで作るPub/Subアーキテクチャ
Avatar for igu zenta
0
110
AIエージェントによるエンタープライズ向けスライド検索！
Avatar for shibuiwilliam shibuiwilliam
4
590
OSだってコンテナしたい❗Image Modeが切り拓くLinux OS運用の新時代
Avatar for Masataka Tsukamoto tsukaman
0
110
LINEヤフー バックエンド組織・体制の紹介
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
810
QAを"自動化する"ことの本質
Avatar for Keisuke Shinozaki kshino
1
140
ある編集者のこれまでとこれから —— 開発者コミュニティと歩んだ四半世紀
Avatar for 稲尾尚徳 inao
5
3.4k
旧から新へ: 大規模ウェブクローラの Perl から Go への移行 / YAPC::Fukuoka 2025
Avatar for motemen motemen
3
1.1k
ECS組み込みのBlue/Greenデプロイを動かしてELB側の動きを観察してみる
Avatar for 矢儀丈博 yuki_ink
1
130
事業状況で変化する最適解。進化し続ける開発組織とアーキテクチャ
Avatar for recruiting@caddi.jp caddi_eng
1
900
AI エージェントを評価するための温故知新と Spec Driven Evaluation
Avatar for Takahiro Kubo icoxfog417
PRO
1
320
仕様は“書く”より“語る” - 分断を超えたチーム開発の実践 / 20251115 Naoki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
1.1k
Rubyist入門: The Way to The Timeless Way of Programming
Avatar for Koji SHIMADA snoozer05
PRO
7
520

Featured

See All Featured
Side Projects
Avatar for Sacha Greif sachag
455
43k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
6.6k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
355
21k
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
46
7.8k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
57k
Scaling GitHub
Avatar for Zach Holman holman
463
140k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
57
6.1k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
127
54k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com