Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
ランサムウェア対策としてのpnpm導入のススメ
Avatar for Satoru Ishikawa ishikawa_satoru
0
150
小さく始めるBCP ― 多プロダクト環境で始める最初の一歩
Avatar for kekke-n kekke_n
1
440
Webhook best practices for rock solid and resilient deployments
Avatar for Guillaume Laforge glaforge
1
290
Amazon Bedrock Knowledge Basesチャンキング解説！
Avatar for 野口碧生 aoinoguchi
0
150
【Ubie】AIを活用した広告アセット「爆速」生成事例 | AI_Ops_Community_Vol.2
Avatar for Yoshiki yoshiki_0316
1
100
Context Engineeringが企業で不可欠になる理由
Avatar for Hirosato Gamo hirosatogamo
PRO
3
610
AzureでのIaC - Bicep? Terraform? それ早く言ってよ会議
Avatar for Toru Makabe torumakabe
1
570
フルカイテン株式会社 エンジニア向け採用資料
Avatar for FULL KAITEN fullkaiten
0
10k
茨城の思い出を振り返る ～CDKのセキュリティを添えて～ / 20260201 Mitsutoshi Matsuo
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
320
Kiro IDEのドキュメントを全部読んだので地味だけどちょっと嬉しい機能を紹介する
Avatar for khmoryz khmoryz
0
200
15 years with Rails and DDD (AI Edition)
Avatar for Andrzej Krzywda andrzejkrzywda
0
200
Bedrock PolicyでAmazon Bedrock Guardrails利用を強制してみた
Avatar for Yudai Jinno yuu551
0
240

Featured

See All Featured
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
160
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
エンジニアに許された特別な時間の終わり
Avatar for watany watany
106
230k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.3k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
76
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8.7k
The agentic SEO stack - context over prompts
Avatar for schlessera schlessera
0
640
Exploring the relationship between traditional SERPs and Gen AI search
Avatar for Ray Grieselhuber raygrieselhuber
PRO
2
3.6k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
80
6.2k
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
120
Claude Code どこまでも/ Claude Code Everywhere
Avatar for nwiizo nwiizo
61
52k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com