Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
540
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
390
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
Goで作る、開発・CI環境
Avatar for Shinpei Mine sin392
0
230
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
Avatar for hideki kinjyo o0h
PRO
1
260
PipeCDのプラグイン化で目指すところ
Avatar for Warashi warashi
1
280
20250704_教育事業におけるアジャイルなデータ基盤構築
Avatar for Uchide Hiroki(ucchi-) hanon52_
5
790
RailsGirls IZUMO スポンサーLT
Avatar for 16bit_idol 16bitidol
0
180
AIプログラマーDevinは PHPerの夢を見るか？
Avatar for Shinya Saita shinyasaita
1
220
Porting a visionOS App to Android XR
Avatar for Akio Itaya akkeylab
0
460
MDN Web Docs に日本語翻訳でコントリビュートしたくなる
Avatar for uutan1108 ohmori_yusuke
1
120
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
Avatar for panda_program panda_program
6
2.1k
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
Avatar for Yusuke Suto sutochin26
1
5.8k
Azure AI Foundryではじめてのマルチエージェントワークフロー
Avatar for 瀬尾佳隆 seosoft
0
170
AI時代のソフトウェア開発を考える（2025/07版） / Agentic Software Engineering Findy 2025-07 Edition
Avatar for Takuto Wada twada
PRO
87
29k

Featured

See All Featured
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
43
7.6k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.1k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
72
4.9k
Done Done
Avatar for chrislema chrislema
184
16k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1031
460k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
181
54k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
179
9.8k
What’s in a name? Adding method to the madness
Avatar for Product Marketing Alliance productmarketing
PRO
23
3.5k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
331
22k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
7
510
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!