Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
安いGPUレンタルサービスについて
Avatar for Aratako aratako
2
2.7k
コミューンのデータ分析AIエージェント「Community Sage」の紹介
Avatar for Yusuke Fukasawa fufufukakaka
0
460
AWS CLIの新しい認証情報設定方法aws loginコマンドの実態
Avatar for wkm2 wkm2
6
640
ChatGPTで論⽂は読めるのか
Avatar for Spatial AI Network spatial_ai_network
0
930
大企業でもできる！ボトムアップで拡大させるプラットフォームの作り方
Avatar for ファインディ株式会社（イベント資料アップ用） findy_eventslides
1
640
AIと二人三脚で育てた、個人開発アプリグロース術
Avatar for ZOZO Developers zozotech
PRO
1
690
EM歴1年10ヶ月のぼくがぶち当たった苦悩とこれからへ向けて
Avatar for maaaato maaaato
0
270
Karate+Database RiderによるAPI自動テスト導入工数をCline＋GitLab MCPを使って2割削減を目指す！ / 20251206 Kazuki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
600
AWS Bedrock AgentCoreで作る 1on1支援AIエージェント 〜Memory × Evaluationsによる実践開発〜
Avatar for Yusuke Shimizu yusukeshimizu
6
380
多様なデジタルアイデンティティを攻撃からどうやって守るのか / 20251212
Avatar for Ayokura ayokura
0
360
re:Invent 2025 ~何をする者であり、どこへいくのか~
Avatar for tetutetu214 tetutetu214
0
180
A Compass of Thought: Guiding the Future of Test Automation ( #jassttokai25 , #jassttokai )
Avatar for teyamagu teyamagu
PRO
1
250

Featured

See All Featured
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
22k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.1k
Navigating Team Friction
Avatar for Lara Hogan lara
191
16k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.4k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1371
200k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.2k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
6.8k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.6k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com