Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
320
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
97
Data Catalogs - Rebuild the Broken Promise
vananth22
0
82
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
520
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
380
The_journey_towards_Pinot.pdf
vananth22
0
210
Reliable_Event_Pipeline___scale.pdf
vananth22
0
180
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.4k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
AtCoder Heuristic First-step Vol.1 講義スライド(山登り法・焼きなまし法編)
takumi152
4
1k
アプリを起動せずにアプリを開発して品質と生産性を上げる
ishkawa
0
2.3k
CRE Meetup!ユーザー信頼性を支えるエンジニアリング実践例の発表資料です
tmnb
0
580
PHPのガベージコレクションを深掘りしよう
rinchoku
0
260
Kubernetesで実現できるPlatform Engineering の現在地
nwiizo
3
1.8k
趣味全開のAITuber開発
kokushin
0
180
SLI/SLOの設定を進めるその前に アラート品質の改善に取り組んだ話
tanden
2
790
海外のアプリで見かけたかっこいいTransitionを真似てみる
shogotakasaki
1
150
技術選定を未来に繋いで活用していく
sakito
3
100
ノーコードツールの裏側につきまとう「20分岐」との戦い
oguemon
0
200
ReactFlow への移行で実現するユーザー体験と開発体験の向上
j9141997
0
150
DomainException と Result 型で作る型安全なエラーハンドリング
karszawa
0
850
Featured
See All Featured
Agile that works and the tools we love
rasmusluckow
328
21k
Adopting Sorbet at Scale
ufuk
75
9.3k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.7k
Building Applications with DynamoDB
mza
94
6.3k
Designing for humans not robots
tammielis
251
25k
4 Signs Your Business is Dying
shpigford
183
22k
A Tale of Four Properties
chriscoyier
158
23k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
21k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.6k
What's in a price? How to price your products and services
michaelherold
245
12k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
22
2.6k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!