Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
350
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
140
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
90
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
610
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
430
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
TiDBのアーキテクチャから学ぶ分散システム入門 〜MySQL互換のNewSQLは何を解決するのか〜 / tidb-architecture-study
Avatar for DPon dznbk
1
180
アーキテクチャモダナイゼーションとは何か
Avatar for nwiizo nwiizo
19
5.3k
ドメインイベントでビジネスロジックを解きほぐす #phpcon_odawara
Avatar for Takuma Kajikawa kajitack
3
790
一度始めたらやめられない開発効率向上術 / Findy あなたのdotfilesを教えて！
Avatar for Takashi Kokubun k0kubun
4
3k
SkillがSkillを生む：QA観点出しを自動化した
Avatar for sontixyou sontixyou
6
3.4k
CursorとClaudeCodeとCodexとOpenCodeを実際に比較してみた
Avatar for Terisuke terisuke
1
480
クラウドネイティブなエンジニアに向ける Raycastの魅力と実際の活用事例
Avatar for Nealle nealle
2
210
Don't Prompt Harder, Structure Better
Avatar for Yusuke Kita kitasuke
0
780
検索設計から
推論設計への重心移動と
Recall-First Retrieval
Avatar for po3rin po3rin
2
860
Claude Code × Gemini × Ebitengine ゲーム制作素人WebエンジニアがGoでゲームを作った話
Avatar for webzawa webzawa
0
150
From Formal Specification to Property Based Test
Avatar for Masato Ohba ohbarye
0
120
煩雑なSkills管理をSoC（関心の分離）により解決する――関心を分離し、プロンプトを部品として育てるためのOSSを作った話 / Solving Complex Skills Management Through SoC (Separation of Concerns)
Avatar for nrs nrslib
4
980

Featured

See All Featured
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
180
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
710
Lessons Learnt from Crawling 1000+ Websites
Avatar for Charles Meaden charlesmeaden
PRO
1
1.2k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
9
1.9k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
3.3k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
1
160
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
160
First, design no harm
Avatar for Per Axbom axbom
PRO
2
1.2k
Visual Storytelling: How to be a Superhuman Communicator
Avatar for David Neal reverentgeek
2
510
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
420
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
2.6k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!