Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
330
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
560
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
vananth22
0
400
The_journey_towards_Pinot.pdf
vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.5k
Other Decks in Programming
See All in Programming
ててべんす独演会〜Flowの全てを語ります〜
tbsten
1
220
CSC305 Lecture 02
javiergs
PRO
1
260
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
ioquatix
3
910
2分台で1500examples完走!爆速CIを支える環境構築術 - Kaigi on Rails 2025
falcon8823
3
3.4k
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
5
1.5k
あなたの知らない「動画広告」の世界 - iOSDC Japan 2025
ukitaka
0
420
GitHub Actions × AWS OIDC連携の仕組みと経緯を理解する
ota1022
0
240
The Flutter Journey of Building a Live Streaming App — With a Side of Performance Tuning
u503
1
100
複雑化したリポジトリをなんとかした話 pipenvからuvによるモノレポ構成への移行
satoshi256kbyte
1
850
株式会社 Sun terras カンパニーデック
sunterras
0
250
Django Ninja による API 開発効率化とリプレースの実践
kashewnuts
0
1.1k
開発生産性を上げるための生成AI活用術
starfish719
1
200
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
Automating Front-end Workflow
addyosmani
1371
200k
Designing for Performance
lara
610
69k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.7k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
31
9.7k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Visualization
eitanlees
148
16k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
Mobile First: as difficult as doing things right
swwweet
224
10k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!