Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
88
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
590
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
AI Agent の開発と運用を支える Durable Execution #AgentsInProd
Avatar for izumin5210 izumin5210
7
2.3k
AIによる高速開発をどう制御するか？ ガードレール設置で開発速度と品質を両立させたチームの事例
Avatar for tonkotsuboy_com tonkotsuboy_com
7
2k
CSC307 Lecture 04
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
660
OCaml 5でモダンな並列プログラミングを Enjoyしよう!
Avatar for Haochen Kotoi-Xie haochenx
0
140
SourceGeneratorのススメ
Avatar for tkym htkym
0
190
20260127_試行錯誤の結晶を1冊に。著者が解説 先輩データサイエンティストからの指南書 / author's_commentary_ds_instructions_guide
Avatar for nash_efp nash_efp
0
930
なぜSQLはAIぽく見えるのか/why does SQL look AI like
Avatar for florets1 florets1
0
450
Kotlin Multiplatform Meetup - Compose Multiplatform 외부 의존성 아키텍처 설계부터 운영까지
Avatar for Suhyeon Kim wisemuji
0
190
今から始めるClaude Code超入門
Avatar for 448jp | OKI Yoshiya 448jp
8
8.5k
登壇資料を作る時に意識していること #登壇資料_findy
Avatar for konifar konifar
4
990
CSC307 Lecture 07
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
550
AIフル活用時代だからこそ学んでおきたい働き方の心得
Avatar for shinoyu shinoyu
0
130

Featured

See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.2k
Marketing Yourself as an Engineer | Alaka | Gurzu
Avatar for Gurzu gurzu
0
130
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.4k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
432
66k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
Fireside Chat
Avatar for paigeccino paigeccino
41
3.8k
The SEO Collaboration Effect
Avatar for Kristina Bergwall kristinabergwall1
0
350
The Limits of Empathy - UXLibs8
Avatar for Cassini Nazir cassininazir
1
210
Digital Ethics as a Driver of Design Innovation
Avatar for Per Axbom axbom
PRO
1
170
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
100
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
160
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
80
6.2k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!