Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
86
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
580
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
AIコーディングエージェント（skywork）
Avatar for kondai kondai24
0
170
251126 TestState APIってなんだっけ?Step Functionsテストどう変わる?
Avatar for Takumi Abe east_takumi
0
320
ローターアクトEクラブ アメリカンナイト：川端 柚菜 氏（Japan O.K. ローターアクトEクラブ 会長）:2720 Japan O.K. ロータリーEクラブ2025年12月1日卓話
Avatar for 2720 Japan O.K. ロータリーEクラブ 2720japanoke
0
730
C-Shared Buildで突破するAI Agent バックテストの壁
Avatar for po3rin po3rin
0
390
堅牢なフロントエンドテスト基盤を構築するために行った取り組み
Avatar for Shogo Fukami shogo4131
8
2.3k
生成AIを利用するだけでなく、投資できる組織へ
Avatar for pospome pospome
2
330
TypeScriptで設計する 堅牢さとUXを両立した非同期ワークフローの実現
Avatar for Matsumoto Moeka moeka__c
6
3k
手が足りない！兼業データエンジニアに必要だったアーキテクチャと立ち回り
Avatar for zinkosuke zinkosuke
0
680
Rubyで鍛える仕組み化プロヂュース力
Avatar for muryoimpl muryoimpl
0
120
Findy AI+の開発、運用におけるMCP活用事例
Avatar for starfish719 starfish719
0
260
TestingOsaka6_Ozono
Avatar for O3(ozono) o3
0
150
認証・認可の基本を学ぼう後編
Avatar for kaza kouyuume
0
190

Featured

See All Featured
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
35
3.3k
Docker and Python
Avatar for Tania Allard trallard
47
3.7k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.3k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
5.7k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.1k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
48
9.8k
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
54k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.7k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!