Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Streaming Ingestion & Processing at Flipkart

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
440
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
Oracle Cloud Infrastructure(OCI)：Onboarding Session(はじめてのOCI/Oracle Supportご利⽤ガイド)
Avatar for oracle4engineer oracle4engineer
PRO
3
21k
Azure Serverless 2026：Production-ready な AI エージェント基盤 / Azure Serverless 2026: Production-Ready AI Agent Platform
Avatar for miyake miyake
2
660
3人で1000GPU超を統合運用する？マルチクラウド＆オンプレを跨ぐ、構築と運用のリアル！
Avatar for kazuki odo kazukun0716
2
740
Coil3を内部実装から読み解く～キャッシュ戦略とAVIF画像の描画〜/nikkei-tech-talk50
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
0
140
AI感のないAWS構成図をAIエージェントに描かせたい！
Avatar for さご sagochiko
2
480
Databricksメトリクスビューはじめてのもくもく会
Avatar for Takaaki Yayoi taka_aki
0
130
あなたの知らないAmazon VPC Route Server／Amazon VPC Route Server you don't know about
Avatar for モブエンジニア（Masaki Okuda） masakiokuda
0
140
生成AIを使って「人が」考える技術 ― AI時代の人機共想と実践ノウハウ｜UNITT AC2026
Avatar for 石井力重 ishiirikie
0
410
Lambda MicroVMsは常駐サーバーの代わりに なるか？ Kiro Crew を動かして検証してみた / Kiro Crew on Lambda MicroVMs
Avatar for Kazuki Adachi k_adachi_01
2
340
AWS DevOps Agent スキルをつかいこなそう / Master AWS DevOps Agent Skills
Avatar for Masanori Yamaguchi kinunori
2
630
TiDBファミリーにDWHが新登場！！ TiDB最新情報 / TiDB update 202609
Avatar for YoshiakiYamasaki yoshiakiyamasaki
0
150
The kernel report
Avatar for Kernel Recipes ennael
PRO
1
130

Featured

See All Featured
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
560
B2B Lead Gen: Tactics, Traps & Triumph
Avatar for Sophie Logan marketingsoph
0
250
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
250
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
450
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
56
3.5k
Docker and Python
Avatar for Tania Allard trallard
47
4.2k
What Being in a Rock Band Can Teach Us About Real World SEO
Avatar for Ade Holder 427marketing
0
1.1k
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.8k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.5k
Ten Tips & Tricks for a 🌱 transition
Avatar for Manu Carrasco Molina stuffmc
1
240
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
390
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.5k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com