Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
86
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
590
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
TestingOsaka6_Ozono
Avatar for O3(ozono) o3
0
180
Context is King？ 〜Verifiability時代とコンテキスト設計 / Beyond "Context is King"
Avatar for r-kagaya rkaga
10
1.4k
안드로이드 9년차 개발자, 프론트엔드 주니어로 커리어 리셋하기
Avatar for Seungmin 마량 maryang
1
140
GISエンジニアから見たLINKSデータ
Avatar for nokonoko1203 nokonoko1203
0
180
新卒エンジニアのプルリクエスト with AI駆動
Avatar for Yuta Fukunaga fukunaga2025
0
230
Navigating Dependency Injection with Metro
Avatar for HyunWoo Lee l2hyunwoo
1
180
TerraformとStrands AgentsでAmazon Bedrock AgentCoreのSSO認証付きエージェントを量産しよう！
Avatar for neruneruo neruneruo
4
1.8k
Grafana：建立系統全知視角的捷徑
Avatar for Blueswen blueswen
0
220
Claude Codeの「Compacting Conversation」を体感50%減！ CLAUDE.md + 8 Skills で挑むコンテキスト管理術
Avatar for Kazuki Murahama kmurahama
1
640
Rubyで鍛える仕組み化プロヂュース力
Avatar for muryoimpl muryoimpl
0
160
Basic Architectures
Avatar for Denys Poltorak denyspoltorak
0
120
Python札幌 LT資料
Avatar for t3tra t3tra
7
1.1k

Featured

See All Featured
Scaling GitHub
Avatar for Zach Holman holman
464
140k
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
0
89
Pawsitive SEO: Lessons from My Dog (and Many Mistakes) on Thriving as a Consultant in the Age of AI
Avatar for David Carrasco davidcarrasco
0
37
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
24k
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
0
2.2k
From π to Pie charts
Avatar for Rasagy Sharma rasagy
0
91
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
269
13k
Winning Ecommerce Organic Search in an AI Era - #searchnstuff2025
Avatar for Aleyda Solis aleyda
0
1.8k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
304
21k
More Than Pixels: Becoming A User Experience Designer
Avatar for Michelle Schulp Hunt marktimemedia
2
260
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
54
7.9k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!