$30 off During Our Annual Pro Sale. View Details »

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
400

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
バグハンター視点によるサプライチェーンの脆弱性
Avatar for morioka12 scgajge12
2
510
Docker, Infraestructuras seguras y Hardening
Avatar for José Juan Sánchez Hernández josejuansanchez
0
150
Digitization部 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
6.1k
Symfony AI in Action
Avatar for Christopher Hertel el_stoffel
2
370
履歴テーブル、今回はこう作りました 〜 Delegated Types編 〜 / How We Built Our History Table This Time — With Delegated Types
Avatar for moznion moznion
16
9.5k
Databricksによるエージェント構築
Avatar for Takaaki Yayoi taka_aki
1
120
プロダクトマネージャーが押さえておくべき、ソフトウェア資産とAIエージェント投資効果 / pmconf2025
Avatar for Masato Ishigaki / 石垣雅人 i35_267
2
360
セキュリティAIエージェントの現在と未来 / PSS #2 Takumi Session
Avatar for GMO Flatt Security flatt_security
3
1.4k
Master Dataグループ紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
4k
その設計、 本当に価値を生んでますか？
Avatar for akshimo shimomura
3
190
pmconf2025 - データを活用し「価値」へ繋げる
Avatar for Emy - Emiko Yoshihara glorypulse
0
460
技術以外の世界に『越境』しエンジニアとして進化を遂げる 〜Kotlinへの愛とDevHRとしての挑戦を添えて〜
Avatar for subroh_0508 subroh0508
1
160

Featured

See All Featured
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
180
10k
KATA
Avatar for Megan Lloyd mclloyd
PRO
32
15k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
6.8k
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
60
9.6k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
73
11k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
1.8k
What’s in a name? Adding method to the madness
Avatar for Product Marketing Alliance productmarketing
PRO
24
3.8k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
25
1.6k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
80
6.1k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
48
9.8k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
274
41k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
791
250k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com