Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
540
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
390
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
すべてのコンテキストを、 ユーザー価値に変える
Avatar for Michiru Kato applism118
3
1.3k
Modern Angular with Signals and Signal Store: New Rules for Your Architecture @enterJS Advanced Angular Day 2025
Avatar for Manfred Steyer manfredsteyer
PRO
0
220
Goで作る、開発・CI環境
Avatar for Shinpei Mine sin392
0
230
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
Avatar for Shohei Okada okashoi
1
260
PipeCDのプラグイン化で目指すところ
Avatar for Warashi warashi
1
280
Flutterで備える！Accessibility Nutrition Labels完全ガイド
Avatar for 野瀬田　裕樹 yuukiw00w
0
160
The Modern View Layer Rails Deserves: A Vision For 2025 And Beyond @ RailsConf 2025, Philadelphia, PA
Avatar for Marco Roth marcoroth
1
170
0626 Findy Product Manager LT Night_高田スライド_speaker deck用
Avatar for Mana Takada mana_takada
0
170
Porting a visionOS App to Android XR
Avatar for Akio Itaya akkeylab
0
470
AI時代の『改訂新版 良いコード／悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
Avatar for MinoDriven minodriven
14
5k
Quand Symfony, ApiPlatform, OpenAI et LangChain s'allient pour exploiter vos PDF : de la théorie à la production…
Avatar for Ahmed EBEN HASSINE ahmedbhs123
0
190
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
Avatar for Yusuke Suto sutochin26
1
5.8k

Featured

See All Featured
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
329
21k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
234
140k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
35
2.4k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.8k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k
Fireside Chat
Avatar for paigeccino paigeccino
37
3.5k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
231
18k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1031
460k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
140
7k
Code Review Best Practice
Avatar for Trisha Gee trishagee
69
18k
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
160
23k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
71
11k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!