Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
350

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
140
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
89
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
600
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
430
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
GoのDB アクセスにおける 「型安全」と「柔軟性」の両立 - Bob という選択肢
Avatar for tak tak848
0
270
Claude Codeセッション現状確認 2026福岡 / fukuoka-aicoding-00-beacon
Avatar for monochromegane monochromegane
4
450
Kubernetesでセルフホストが簡単なNewSQLを求めて / Seeking a NewSQL Database That's Simple to Self-Host on Kubernetes
Avatar for nnaka2992 nnaka2992
0
170
Redox OS でのネームスペース管理と chroot の実現
Avatar for isan isanethen
0
390
今からFlash開発できるわけないじゃん、ムリムリ! (※ムリじゃなかった!?)
Avatar for Sora Arakawa arkw
0
130
Ruby and LLM Ecosystem 2nd
Avatar for Koichi ITO koic
1
1.2k
CSC307 Lecture 15
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
260
nuget-server - あなたが必要だったNuGetサーバー
Avatar for Kouji Matsui kekyo
PRO
0
370
Java 21/25 Virtual Threads 소개
Avatar for Sunghyouk Bae (Debop) debop
0
220
エンジニアの「手元の自動化」を加速するn8n 2026.02.27
Avatar for 西尾義英 symy2co
0
170
ロボットのための工場に灯りは要らない
Avatar for watany watany
11
3.1k
PHP 7.4でもOpenTelemetryゼロコード計装がしたい！ / PHPerKaigi 2026
Avatar for Arthur arthur1
1
350

Featured

See All Featured
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
280
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
82
6.2k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
21k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
75
11k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.8k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
260
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
200
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
210
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
287
14k
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!