Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.
Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
180
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
解析の定理証明実践＠Lean 4
Avatar for H Segawa dec9ue
0
180
Navigation3でViewModelにデータを渡す方法
Avatar for mikan mikanichinose
0
220
SalesforceArchitectGroupOsaka#20_CNX'25_Report
Avatar for atomica7sei atomica7sei
0
170
ひとり情シスなCTOがLLMと始めるオペレーション最適化 / CTO's LLM-Powered Ops
Avatar for Mitsuki Ogasahara yamitzky
0
430
Understanding_Thread_Tuning_for_Inference_Servers_of_Deep_Models.pdf
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
120
“社内”だけで完結していた私が、AWS Community Builder になるまで
Avatar for nagisa_53 nagisa53
1
390
Node-RED × MCP 勉強会 vol.1
Avatar for 1ft-seabass 1ftseabass
PRO
0
140
AWS テクニカルサポートとエンドカスタマーの中間地点から見えるより良いサポートの活用方法
Avatar for kazzpapa3 kazzpapa3
2
540
BrainPadプログラミングコンテスト記念LT会2025_社内イベント&問題解説
Avatar for BrainPad brainpadpr
1
160
PHPでWebブラウザのレンダリングエンジンを実装する
Avatar for ディップ株式会社 dip_tech
PRO
0
200
Snowflake Summit 2025全体振り返り / Snowflake Summit 2025 Overall Review
Avatar for k.muguruma mtpooh
2
400
低レイヤを知りたいPHPerのためのCコンパイラ作成入門 完全版 / Building a C Compiler for PHPers Who Want to Dive into Low-Level Programming - Expanded
Avatar for HASEGAWA Tomoki tomzoh
4
3.2k

Featured

See All Featured
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
134
9.3k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
17
940
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
82
9.1k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
55
5.6k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
231
18k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
512
110k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
8
670
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.7k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
34
3k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
229
22k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
281
13k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com