$30 off During Our Annual Pro Sale. View Details »

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
意外とあった SQL Server 関連アップデート + Database Savings Plans
Avatar for Takuya Shibata stknohg
PRO
0
300
研究開発×プロダクトマネジメントへの挑戦 / ly_mlpm_meetup
Avatar for Sansan R&D sansan_randd
0
100
A Compass of Thought: Guiding the Future of Test Automation ( #jassttokai25 , #jassttokai )
Avatar for teyamagu teyamagu
PRO
1
250
計算機科学をRubyと歩む 〜DFA型正規表現エンジンをつくる～
Avatar for ydah ydah
3
210
因果AIへの招待
Avatar for Shohei SHIMIZU sshimizu2006
0
940
チーリンについて
Avatar for ぐっち hirotomotaguchi
5
1.6k
Playwright x GitHub Actionsで実現する「レビューしやすい」E2Eテストレポート
Avatar for kinosuke01 kinosuke01
0
500
GitHub Copilotを使いこなす 実例に学ぶAIコーディング活用術
Avatar for 74th(Atsushi Morimoto) 74th
3
1.9k
AI活用によるPRレビュー改善の歩み ― 社内全体に広がる学びと実践
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
1
190
MapKitとオープンデータで実現する地図情報の拡張と可視化
Avatar for ZOZO Developers zozotech
PRO
1
130
今年のデータ・ML系アップデートと気になるアプデのご紹介
Avatar for Nayuta S. nayuts
1
220
非CUDAの悲哀 〜Claude Code と挑んだ image to 3D “Hunyuan3D”を EVO-X2(Ryzen AI Max+395)で動作させるチャレンジ〜
Avatar for hawky the miscellaneous hawkymisc
1
160

Featured

See All Featured
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
180
10k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
22k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
8
1.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.2k
Done Done
Avatar for chrislema chrislema
186
16k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
500
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
7k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
39k
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
140k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com