Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
350

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
140
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
89
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
600
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
430
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
What Spring Developers Should Know About Jakarta EE
Avatar for ivargrimstad ivargrimstad
0
660
Redox OS でのネームスペース管理と chroot の実現
Avatar for isan isanethen
0
450
車輪の再発明をしよう！PHP で実装して学ぶ、Web サーバーの仕組みと HTTP の正体
Avatar for H1R0 h1r0
2
400
[PHPerKaigi 2026]PHPerKaigi2025の企画CodeGolfが最高すぎて社内で内製して半年運営して得た内製と運営の知見
Avatar for Z.O.E. ikezoemakoto
0
290
AWS×クラウドネイティブソフトウェア設計 / AWS x Cloud-Native Software Design
Avatar for nrs nrslib
16
3.4k
20260315 AWSなんもわからん🥲
Avatar for Chiaki Okamoto chiilog
2
170
Mastering Event Sourcing: Your Parents Holidayed in Yugoslavia
Avatar for Marek Matulka super_marek
0
110
野球解説AI Agentを開発してみた - 2026/02/27 LayerX社内LT会資料
Avatar for Shinichi Nakagawa shinyorke
PRO
0
370
Windows on Ryzen and I
Avatar for 瀬尾佳隆 seosoft
0
400
RailsのValidatesをSwift Macrosで再現してみた
Avatar for Takuma Shimizu hokuron
0
130
PHP 7.4でもOpenTelemetryゼロコード計装がしたい！ / PHPerKaigi 2026
Avatar for Arthur arthur1
1
410
AI 開発合宿を通して得た学び
Avatar for ニフティ株式会社 niftycorp
PRO
0
170

Featured

See All Featured
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.4k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
0
140
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
130
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
2
290
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
2
1k
Paper Plane
Avatar for Katie Cordina Lindsey katiecoart
PRO
0
48k
Stewardship and Sustainability of Urban and Community Forests
Avatar for Eric Wiseman pwiseman
0
160
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
3
300
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
10
1.1k
Future Trends and Review - Lecture 12 - Web Technologies (1019888BNR)
Avatar for Beat Signer signer
PRO
0
3.3k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
7k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!