Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
350
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
140
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
90
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
600
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
430
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
AI時代のPhpStorm最新事情 #phpcon_odawara
Avatar for yusuke yusuke
0
140
AI活用のコスパを最大化する方法
Avatar for Ochtum ochtum
0
380
Codex CLIのSubagentsによる並列API実装 / Parallel API Implementation with Codex CLI Subagents
Avatar for 冨澤宝斗 takatty
2
870
Kubernetes上でAgentを動かすための最新動向と押さえるべき概念まとめ
Avatar for Sota Makino sotamaki0421
3
450
AWS re：Invent 2025の少し振り返り ＋ DevOps AgentとBacklogを連携させてみた
Avatar for Satoshi Kaneyasu satoshi256kbyte
2
150
AI時代の脳疲弊と向き合う ~言語学としてのPHP~
Avatar for Sakurai sakuraikotone
1
1.8k
Rethinking API Platform Filters
Avatar for Vincent Amstoutz vinceamstoutz
0
11k
3分でわかるatama plusのQA/about atama plus QA
Avatar for atama plus atamaplus
0
130
forteeの改修から振り返るPHPerKaigi 2026
Avatar for muno92 muno92
PRO
3
250
Mastering Event Sourcing: Your Parents Holidayed in Yugoslavia
Avatar for Marek Matulka super_marek
0
150
How Swift's Type System Guides AI Agents
Avatar for Yuta Koshizawa koher
0
190
実践ハーネスエンジニアリング #MOSHTech
Avatar for Takuma Kajikawa kajitack
7
6.1k

Featured

See All Featured
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
33k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
5.1k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8k
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
260
Test your architecture with Archunit
Avatar for Yoan thirion
1
2.2k
The B2B funnel & how to create a winning content strategy
Avatar for Katarina Dahlin katarinadahlin
PRO
1
330
New Earth Scene 8
Avatar for Sydney Stone popppiees
3
2k
AI Search:
Where Are We & What Can We Do About It?
Avatar for Aleyda Solis aleyda
0
7.3k
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.2k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
400
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
25k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!