Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Ingestion & Processing at Flipkart

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Siddhartha Reddy Siddhartha Reddy
May 15, 2015
Technology
0
410

Streaming Ingestion & Processing at Flipkart

Presented at the Bangalore Hadoop Meetup held on 15th May 2015.

Avatar for Siddhartha Reddy

Siddhartha Reddy

May 15, 2015
Tweet

More Decks by Siddhartha Reddy

See All by Siddhartha Reddy
Future Patterns in Data Ecosystem
Avatar for Siddhartha Reddy sids
1
200
CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA
Avatar for Siddhartha Reddy sids
6
12k

Other Decks in Technology

See All in Technology
AzureでのIaC - Bicep? Terraform? それ早く言ってよ会議
Avatar for Toru Makabe torumakabe
1
580
レガシー共有バッチ基盤への挑戦 - SREドリブンなリアーキテクチャリングの取り組み
Avatar for Konishi tatsuhiro tatsukoni
0
220
学生・新卒・ジュニアから目指すSRE
Avatar for Hiroya Onoe hiroyaonoe
2
630
OWASP Top 10:2025 リリースと 少しの日本語化にまつわる裏話
Avatar for Riotaro OKADA okdt
PRO
3
810
広告の効果検証を題材にした因果推論の精度検証について
Avatar for ZOZO Developers zozotech
PRO
0
190
顧客との商談議事録をみんなで読んで顧客解像度を上げよう
Avatar for shibayu36 shibayu36
0
260
顧客の言葉を、そのまま信じない勇気
Avatar for yamatai12 yamatai1212
1
360
Introduction to Sansan, inc / Sansan Global Development Center, Inc.
Avatar for Sansan, Inc. sansan33
PRO
0
3k
SRE Enabling戦記 - 急成長する組織にSREを浸透させる戦いの歴史
Avatar for Masaki Ishigaki markie1009
0
130
コスト削減から「セキュリティと利便性」を担うプラットフォームへ
Avatar for SansanTech sansantech
PRO
3
1.5k
制約が導く迷わない設計 〜 信頼性と運用性を両立するマイナンバー管理システムの実践 〜
Avatar for ないとー bwkw
3
960
Kiro IDEのドキュメントを全部読んだので地味だけどちょっと嬉しい機能を紹介する
Avatar for khmoryz khmoryz
0
200

Featured

See All Featured
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
410
AI: The stuff that nobody shows you
Avatar for John Nunemaker jnunemaker
PRO
2
260
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
6k
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
51
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
2
250
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
Marketing Yourself as an Engineer | Alaka | Gurzu
Avatar for Gurzu gurzu
0
130
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
110
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
390
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
Avatar for Tech SEO Connect techseoconnect
PRO
0
62
Stewardship and Sustainability of Urban and Community Forests
Avatar for Eric Wiseman pwiseman
0
110

Transcript

  1. Streaming Ingestion & Processing at Flipkart Siddhartha Reddy @sids

  2. Flipkart Data Platform (an oversimplified view)

  3. Streaming Ingestion

  4. Choices • push, not pull • schemas & validations

  5. Streaming Ingestion v1.0

  6. None

  7. • Push 㱺 accountability (with source teams) • good call!

    • Schemas 㱺 contracts for consumers • can make assumptions that are assured to be true • Insufficient tooling 㱺 too many “ingestion frameworks” • adopt some frameworks & offer as tools! • Synchronous error handling 㱺 complexity • accept all data

  8. Streaming Ingestion v2.0

  9. Stream Processing

  10. An Example

  11. Streaming Joins: Example It works! But… how do we deal

    with lookup failures?

  12. Streaming Joins: Handling Failures

  13. None
  14. None

  15. Streaming Joins: Bootstrapping With a little help from MR friends

  16. Streaming Joins: But… The example that doesn’t really work correctly

  17. Streaming Joins

  18. In summary • Streaming Ingestion: push, schemas & validation, HTTP

    service, local daemon, change data capture • Streaming Joins: indexing, lookup tables, map-joins, retry queue, batch re-driver sid@flipkart.com