Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
83
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
530
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
390
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
レガシーシステムの機能調査・開発におけるAI利活用
Avatar for takuya ohtonari takuya_ohtonari
0
610
Blazing Fast UI Development with Compose Hot Reload (droidcon New York 2025)
Avatar for Márton Braun zsmb
1
120
『自分のデータだけ見せたい！』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
Avatar for AkitoTsukahara akitotsukahara
1
360
明示と暗黙 ー PHPとGoの インターフェイスの違いを知る
Avatar for shimabox shimabox
2
170
GoのGenericsによるslice操作との付き合い方
Avatar for syumai syumai
2
680
Kotlin エンジニアへ送る：Swift 案件に参加させられる日に備えて～似てるけど色々違う Swift の仕様 / from Kotlin to Swift
Avatar for Elvis Shi lovee
1
250
無関心の谷
Avatar for kanayannet kanayannet
0
180
Julia という言語について （FP in Julia « SIDE: F »） for 関数型まつり2025
Avatar for GOTOH Shunsuke antimon2
3
970
関数型まつり2025登壇資料「関数プログラミングと再帰」
Avatar for Taison Tsukada taisontsukada
2
840
KotlinConf 2025 現地で感じたServer-Side Kotlin
Avatar for Takehata Naoto n_takehata
1
220
C++20 射影変換
Avatar for Akira Takahashi faithandbrave
0
500
git worktree × Claude Code × MCP ～生成AI時代の並列開発フロー～
Avatar for hisuzuya hisuzuya
0
220

Featured

See All Featured
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
271
27k
The Language of Interfaces
Avatar for Des Traynor destraynor
158
25k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
281
13k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
71
4.9k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
7
700
Building an army of robots
Avatar for Kyle Neath kneath
306
45k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.6k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
4
200
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
36
2.8k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.7k
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
54
13k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!