Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
330
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
vananth22
0
85
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
570
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
vananth22
0
410
The_journey_towards_Pinot.pdf
vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.5k
Other Decks in Programming
See All in Programming
CSC509 Lecture 08
javiergs
PRO
0
260
釣り地図SNSにおける有料機能の実装
nokonoko1203
0
200
CSC305 Lecture 08
javiergs
PRO
0
280
Catch Up: Go Style Guide Update
andpad
0
260
Blazing Fast UI Development with Compose Hot Reload (droidcon London 2025)
zsmb
0
250
なんでRustの環境構築してないのにRust製のツールが動くの? / Why Do Rust-Based Tools Run Without a Rust Environment?
ssssota
14
46k
マンガアプリViewerの大画面対応を考える
kk__777
0
360
AkarengaLT vol.38
hashimoto_kei
1
130
SwiftDataを使って10万件のデータを読み書きする
akidon0000
0
240
Towards Transactional Buffering of CDC Events @ Flink Forward 2025 Barcelona Spain
hpgrahsl
0
120
社会人になっても趣味開発を続けたい! / traPavilion
mazrean
1
110
Reactive Thinking with Signals and the Resource API
manfredsteyer
PRO
0
120
Featured
See All Featured
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Practical Orchestrator
shlominoach
190
11k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
36
6.1k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
116
20k
Agile that works and the tools we love
rasmusluckow
331
21k
YesSQL, Process and Tooling at Scale
rocio
173
15k
What's in a price? How to price your products and services
michaelherold
246
12k
A Modern Web Designer's Workflow
chriscoyier
697
190k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
We Have a Design System, Now What?
morganepeng
53
7.8k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!