Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Programming
1
330
Search Infrastructure using Lambda Architecture
Search
Ananth Packkildurai
May 31, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
560
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
400
The_journey_towards_Pinot.pdf
vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
Ruby×iOSアプリ開発 ~共に歩んだエコシステムの物語~
temoki
0
350
OSS開発者という働き方
andpad
5
1.7k
Laravel Boost 超入門
fire_arlo
3
220
為你自己學 Python - 冷知識篇
eddie
1
350
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
410
Navigating Dependency Injection with Metro
zacsweers
3
2.5k
より安全で効率的な Go コードへ: Protocol Buffers Opaque API の導入
shwatanap
2
730
AI時代のUIはどこへ行く?
yusukebe
18
9k
Reading Rails 1.0 Source Code
okuramasafumi
0
250
Oracle Database Technology Night 92 Database Connection control FAN-AC
oracle4engineer
PRO
1
470
スケールする組織の実現に向けた インナーソース育成術 - ISGT2025
teamlab
PRO
1
160
テストコードはもう書かない:JetBrains AI Assistantに委ねる非同期処理のテスト自動設計・生成
makun
0
520
Featured
See All Featured
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.7k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
4 Signs Your Business is Dying
shpigford
184
22k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
810
Raft: Consensus for Rubyists
vanstee
140
7.1k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
Docker and Python
trallard
46
3.6k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.9k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
113
20k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
Transcript
Search infrastructure using Lambda Architecture Ananth Packkildurai
Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC
Design patterns
Search @ Slack
http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e
Lambda Architecture
Stream processing trade-offs Pick 2 Speed Accuracy Volume
Lambda Architecture http://lambda-architecture.net/
Search Infrastructure overview
Search infrastructure overview
! Online sharding by Collections ! Offline sharding by CompositeId
Router Sharding strategy
mutating Index & Delta Replication • What happens if someone
edits the message that indexed in batch mode? • A snapshot occurs at time T1, the index built, and the core comes online at T2.Between T1 & T2, How to handle the update?
FCC Pattern
Fork Pattern
Contract
Clone
Any Questions? Thanks!