Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
許しとアジャイル
Avatar for Jun Nakajima jnuank
1
120
リーダーになったら未来を語れるようになろう/Speak the Future
Avatar for Genki Sano sanogemaru
0
280
コンテキストエンジニアリングとは? 考え方と応用方法
Avatar for ファインディ株式会社（イベント資料アップ用） findy_eventslides
4
900
フルカイテン株式会社 エンジニア向け採用資料
Avatar for FULL KAITEN fullkaiten
0
9k
Pure Goで体験するWasmの未来
Avatar for asuka askua
1
180
非エンジニアのあなたもできる＆もうやってる！コンテキストエンジニアリング
Avatar for ファインディ株式会社（イベント資料アップ用） findy_eventslides
3
910
AI駆動開発を推進するためにサービス開発チームで 取り組んでいること
Avatar for おーしろ noayaoshiro
0
170
KMP の Swift export
Avatar for Koki Hirokawa kokihirokawa
0
330
SREとソフトウェア開発者の合同チームはどのようにS3のコストを削減したか？
Avatar for Masahiro Yoshizawa muziyoshiz
1
100
about #74462 go/token#FileSet
Avatar for tom twinkle tomtwinkle
1
290
Optuna DashboardにおけるPLaMo2連携機能の紹介 / PFN LLM セミナー
Avatar for Preferred Networks pfn
PRO
1
880
空間を設計する力を考える / 20251004 Naoki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
3
330

Featured

See All Featured
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
610
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
333
22k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
43
7.7k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
36
2.5k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
209
24k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
49
14k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
27
2k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
180
9.9k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com