Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
[OCI Technical Deep Dive] OCIで生成AIを活用するためのソリューション解説(2025年8月5日開催)
Avatar for oracle4engineer oracle4engineer
PRO
0
110
ファッションコーディネートアプリ「WEAR」における、Vertex AI Vector Searchを利用したレコメンド機能の開発・運用で得られたノウハウの紹介
Avatar for ZOZO Developers zozotech
PRO
0
550
Oracle Exadata Database Service on Cloud@Customer X11M (ExaDB-C@C) サービス概要
Avatar for oracle4engineer oracle4engineer
PRO
2
6.3k
20250807 Applied Engineer Open House
Avatar for Sakana AI sakana_ai
PRO
2
530
九州の人に知ってもらいたいGISスポット / gis spot in kyushu 2025
Avatar for sakaik sakaik
0
180
薬屋のひとりごとにみるトラブルシューティング
Avatar for tomokusaba tomokusaba
0
380
Intro to Software Startups: Spring 2025
Avatar for Arnab Nandi arnabdotorg
0
270
マルチプロダクト×マルチテナントを支えるモジュラモノリスを中心としたアソビューのアーキテクチャ
Avatar for disc99 disc99
1
630
Delegate authentication and a lot more to Keycloak with OpenID Connect
Avatar for Alexander Schwartz ahus1
0
230
AIは変更差分からユニットテスト_結合テスト_システムテストでテストすべきことが出せるのか？
Avatar for Matsu mineo_matsuya
1
660
PL/pgSQLの基本と使い所
Avatar for まぐろ tameguro
2
230
Findy Freelance 利用シーン別AI活用例
Avatar for nesskazu ness
0
660

Featured

See All Featured
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
160
23k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
333
22k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
329
21k
Building Adaptive Systems
Avatar for Chris Keathley keathley
43
2.7k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
40
2k
Visualization
Avatar for Eitan Lees eitanlees
146
16k
Fireside Chat
Avatar for paigeccino paigeccino
39
3.6k
Done Done
Avatar for chrislema chrislema
185
16k
Navigating Team Friction
Avatar for Lara Hogan lara
188
15k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
2
550
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
251
21k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com