Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
420
0

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
220
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
人材育成分科会.pdf
Avatar for _awache _awache
2
160
10倍の生産性を実現するAI駆動並列エージェントのすべて
Avatar for 熊井悠 kumaiu
5
1.4k
【NRUG vol.18】KubernetesにおけるNew Relicデータ取得量削減の考え方
Avatar for NRUG member nrug_member
0
110
ACE-Step-1.5で見る 音楽生成AIのしくみと“破綻だけ直す”Retake機能の開発【zennfes spring 2026 登壇資料】
Avatar for asap personabb
1
260
How Timee Delivers Day 1 Production Ready LLM Features
Avatar for tomoyks tomoyks
0
190
気づかぬうちにセキュリティ負債を生むAPIキー運用
Avatar for Michitaka Sugawara sgwrmctk
0
120
小さくはじめるSLI/SLO ～育てながら組織に定着させる実践知～ / Starting Small with SLI/SLOs: Building Adoption Through Continuous Growth
Avatar for Narimichi Takamura nari_ex
7
1.9k
中期計画、2回作ってみた　～業務委託と正社員、両方の視点から～
Avatar for 株式会社出前館 demaecan
1
730
自宅LLMの話
Avatar for Kazuto Kusama jacopen
1
510
200個のGitHubリポジトリを横断調査したかった
Avatar for Issei.Komori icck
0
120
Claude Code×Terraform IaC テンプレート駆動開発
Avatar for Itou Hideki itouhi
1
510
Building applications in the Gemini API family.
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
3.2k

Featured

See All Featured
First, design no harm
Avatar for Per Axbom axbom
PRO
2
1.2k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
408
66k
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.1k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon Słowik szymonslowik
1
1.1k
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
2.1k
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
310
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
220
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
28
3.5k
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
390
Side Projects
Avatar for Sacha Greif sachag
455
43k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com