Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
390

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
190
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
11k

Other Decks in Technology

See All in Technology
エラーとアクセシビリティ
Avatar for Tajima Sachiko schktjm
1
1.2k
MCPで変わる Amebaデザインシステム「Spindle」の開発
Avatar for Ameba Spindle spindle
PRO
3
3.2k
OCI Oracle Database Services新機能アップデート(2025/06-2025/08)
Avatar for oracle4engineer oracle4engineer
PRO
0
110
「どこから読む？」コードとカルチャーに最速で馴染むための実践ガイド
Avatar for ZOZO Developers zozotech
PRO
0
290
サラリーマンの小遣いで作るtoCサービス - Cloudflare Workersでスケールする開発戦略
Avatar for shinaps shinaps
2
420
5分でカオスエンジニアリングを分かった気になろう
Avatar for Aja9tochin pandayumi
0
230
「Linux」という言葉が指すもの
Avatar for Satoru Takeuchi sat
PRO
4
120
COVESA VSSによる車両データモデルの標準化とAWS IoT FleetWiseの活用
Avatar for Yuto Osawa osawa
1
270
未経験者・初心者に贈る！40分でわかるAndroidアプリ開発の今と大事なポイント
Avatar for operandoOS operando
5
370
バイブスに「型」を！Kent Beckに学ぶ、AI時代のテスト駆動開発
Avatar for amixedcolor amixedcolor
2
540
AWSで始める実践Dagster入門
Avatar for キタガワ kitagawaz
1
610
Automating Web Accessibility Testing with AI Agents
Avatar for South maminami373
0
1.2k

Featured

See All Featured
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.9k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
15k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
224
9.9k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
34
6k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
3k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
90
590k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
173
14k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.4k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
26
1.9k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
131
19k

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com