Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
350
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
140
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
92
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
620
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
440
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
250
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.6k

Other Decks in Programming

See All in Programming
代数的データ型って何が嬉しいの？ #frontend_phpcon_do
Avatar for Takuma Kajikawa kajitack
8
3.1k
柔軟なPDFレイアウトエディタを支える型システム設計 — Discriminated UnionとConditional Typeの実践
Avatar for minako-ph minako__ph
4
1.3k
自動レビューエンジンの実装と運用 ~レビューのない世界へ~
Avatar for Kanato kurukuru1999
2
310
今さら聞けないCancellationToken
Avatar for tkym htkym
0
210
密結合なバックエンドから TypeScript のコードを生成する
Avatar for kemuridama kemuridama
1
680
Inspired By RubyKaigi (EN)
Avatar for Atsushi Kawamura (atzz/a2c) atzzcokek
0
500
不変条件と整合性境界—ビジネスが決める設計判断と実現パターン / Invariants and Consistency Boundaries
Avatar for nrs nrslib
13
3.3k
さぁV100、メモリをお食べ・・・
Avatar for nilpe nilpe
0
130
LLM本来の能力を解き放つサンドボックス技術とAI民主化への適用
Avatar for Yuku Kotani yukukotani
3
2k
サーバーレスで作る、動画データ管理基盤
Avatar for Keigo Ibaraki oyasumipants
0
350
SPMマルチモジュールで テストカバレッジを取得する技法
Avatar for yosshi yosshi4486
0
140
PHPで使える日時の表現と、その知り方 #frontend_phpcon_do
Avatar for hideki kinjyo o0h
PRO
0
180

Featured

See All Featured
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
27k
The B2B funnel & how to create a winning content strategy
Avatar for Katarina Dahlin katarinadahlin
PRO
1
380
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
2
320
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
Avatar for Tech SEO Connect techseoconnect
PRO
0
170
Between Models and Reality
Avatar for mayunak mayunak
4
320
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
6k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
34
9.3k
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
180
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
302
52k
B2B Lead Gen: Tactics, Traps & Triumph
Avatar for Sophie Logan marketingsoph
0
130
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
28
3.5k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!