Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
88
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
590
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
Implementation Patterns
Avatar for Denys Poltorak denyspoltorak
0
280
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
490
登壇資料を作る時に意識していること #登壇資料_findy
Avatar for konifar konifar
4
930
AgentCoreとHuman in the Loop
Avatar for Har1101 har1101
5
220
今こそ知るべき耐量子計算機暗号(PQC)入門 / PQC: What You Need to Know Now
Avatar for mackey0225 mackey0225
3
370
Amazon Bedrockを活用したRAGの品質管理パイプライン構築
Avatar for とすり tosuri13
4
240
humanlayerのブログから学ぶ、良いCLAUDE.mdの書き方
Avatar for Yuto tsukamoto1783
0
180
プロダクトオーナーから見たSOC2 _SOC2ゆるミートアップ#2
Avatar for Kenta Suzuki kekekenta
0
200
IFSによる形状設計／デモシーンの魅力 @ 慶應大学SFC
Avatar for がむ gam0022
1
300
MDN Web Docs に日本語翻訳でコントリビュート
Avatar for uutan1108 ohmori_yusuke
0
640
コントリビューターによるDenoのすゝめ / Deno Recommendations by a Contributor
Avatar for petamoriken / 森建 petamoriken
0
200
生成AIを使ったコードレビューで定性的に品質カバー
Avatar for Chiaki Okamoto chiilog
1
240

Featured

See All Featured
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
120
Everyday Curiosity
Avatar for Cassini Nazir cassininazir
0
130
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.6k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
820
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
110
Designing for Performance
Avatar for Lara Hogan lara
610
70k
First, design no harm
Avatar for Per Axbom axbom
PRO
2
1.1k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
34
2.6k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1032
470k
Fireside Chat
Avatar for paigeccino paigeccino
41
3.8k
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
Avatar for Tech SEO Connect techseoconnect
PRO
0
55

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!