Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
330
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
550
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
なぜ今、Terraformの本を書いたのか? - 著者陣に聞く!『Terraformではじめる実践IaC』登壇資料
fufuhu
4
620
「リーダーは意思決定する人」って本当?~ 学びを現場で活かす、リーダー4ヶ月目の試行錯誤 ~
marina1017
0
220
State of CSS 2025
benjaminkott
1
110
Comparing decimals in Swift Testing
417_72ki
0
170
ゲームの物理
fadis
5
1.2k
プロダクトという一杯を作る - プロダクトチームが味の責任を持つまでの煮込み奮闘記
hiliteeternal
0
460
CEDEC 2025 『ゲームにおけるリアルタイム通信への QUIC導入事例の紹介』
segadevtech
3
890
Nuances on Kubernetes - RubyConf Taiwan 2025
envek
0
170
コーディングは技術者(エンジニア)の嗜みでして / Learning the System Development Mindset from Rock Lady
mackey0225
2
490
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
1
280
TROCCO×dbtで実現する人にもAIにもやさしいデータ基盤
nealle
0
240
レガシープロジェクトで最大限AIの恩恵を受けられるようClaude Codeを利用する
tk1351
2
140
Featured
See All Featured
What's in a price? How to price your products and services
michaelherold
246
12k
Automating Front-end Workflow
addyosmani
1370
200k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Embracing the Ebb and Flow
colly
86
4.8k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
How to train your dragon (web standard)
notwaldorf
96
6.2k
Why Our Code Smells
bkeepers
PRO
338
57k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.6k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
283
13k
The Cost Of JavaScript in 2023
addyosmani
53
8.8k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!