Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
88
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
590
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
MDN Web Docs に日本語翻訳でコントリビュート
Avatar for uutan1108 ohmori_yusuke
0
630
AI前提で考えるiOSアプリのモダナイズ設計
Avatar for 野瀬田　裕樹 yuukiw00w
0
220
副作用をどこに置くか問題：オブジェクト指向で整理する設計判断ツリー
Avatar for Koya Masuda koxya
1
570
Fluid Templating in TYPO3 14
Avatar for Simon Praetorius s2b
0
120
humanlayerのブログから学ぶ、良いCLAUDE.mdの書き方
Avatar for Yuto tsukamoto1783
0
170
ELYZA_Findy AI Engineering Summit登壇資料_AIコーディング時代に「ちゃんと」やること_toB LLMプロダクト開発舞台裏_20251216
Avatar for 株式会社ELYZA elyza
2
1.4k
余白を設計しフロントエンド開発を 加速させる
Avatar for karacoro / からころ tsukuha
7
2.1k
それ、本当に安全？ ファイルアップロードで見落としがちなセキュリティリスクと対策
Avatar for ikechi penpeen
7
2.4k
AI Schema Enrichment for your Oracle AI Database
Avatar for thatjeffsmith thatjeffsmith
0
200
AIで開発はどれくらい加速したのか？AIエージェントによるコード生成を、現場の評価と研究開発の評価の両面からdeep diveしてみる
Avatar for どすこい daisuketakeda
1
940
AIエージェントの設計で注意するべきポイント6選
Avatar for Har1101 har1101
7
3.3k
AI Agent Tool のためのバックエンドアーキテクチャを考える #encraft
Avatar for izumin5210 izumin5210
6
1.8k

Featured

See All Featured
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
182
10k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
273
21k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
16k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
2.7k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.4k
BBQ
Avatar for Matthew Crist matthewcrist
89
10k
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
160
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
9.5k
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
130
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
1
480
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
280
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
0
51

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!