Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
330
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
560
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
400
The_journey_towards_Pinot.pdf
vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
そのAPI、誰のため? Androidライブラリ設計における利用者目線の実践テクニック
mkeeda
2
240
Vue・React マルチプロダクト開発を支える Vite
andpad
0
110
How Android Uses Data Structures Behind The Scenes
l2hyunwoo
0
360
Compose Multiplatform × AI で作る、次世代アプリ開発支援ツールの設計と実装
thagikura
0
120
250830 IaCの選定~AWS SAMのLambdaをECSに乗り換えたときの備忘録~
east_takumi
0
380
AI時代のUIはどこへ行く?
yusukebe
16
8.7k
AIコーディングAgentとの向き合い方
eycjur
0
260
Putting The Genie in the Bottle - A Crash Course on running LLMs on Android
iurysza
0
140
アセットのコンパイルについて
ojun9
0
120
FindyにおけるTakumi活用と脆弱性管理のこれから
rvirus0817
0
480
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
280
CJK and Unicode From a PHP Committer
youkidearitai
PRO
0
110
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Scaling GitHub
holman
463
140k
How STYLIGHT went responsive
nonsquared
100
5.8k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
Documentation Writing (for coders)
carmenintech
74
5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Why Our Code Smells
bkeepers
PRO
339
57k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
Rails Girls Zürich Keynote
gr2m
95
14k
Art, The Web, and Tiny UX
lynnandtonic
302
21k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Building an army of robots
kneath
306
46k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!