Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
330
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
540
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
Goで作る、開発・CI環境
sin392
0
230
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
260
PipeCDのプラグイン化で目指すところ
warashi
1
280
20250704_教育事業におけるアジャイルなデータ基盤構築
hanon52_
5
790
RailsGirls IZUMO スポンサーLT
16bitidol
0
180
AIプログラマーDevinは PHPerの夢を見るか?
shinyasaita
1
220
Porting a visionOS App to Android XR
akkeylab
0
460
MDN Web Docs に日本語翻訳でコントリビュートしたくなる
ohmori_yusuke
1
120
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
2.1k
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
sutochin26
1
5.8k
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
170
AI時代のソフトウェア開発を考える(2025/07版) / Agentic Software Engineering Findy 2025-07 Edition
twada
PRO
87
29k
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
43
7.6k
RailsConf 2023
tenderlove
30
1.1k
Documentation Writing (for coders)
carmenintech
72
4.9k
Done Done
chrislema
184
16k
GitHub's CSS Performance
jonrohan
1031
460k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
181
54k
For a Future-Friendly Web
brad_frost
179
9.8k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Designing for Performance
lara
610
69k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
510
Producing Creativity
orderedlist
PRO
346
40k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!