Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
83
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
530
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
390
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
190
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
RubyKaigiで得られる10の価値 〜Ruby話を聞くことだけが RubyKaigiじゃない〜
Avatar for tomohiko.kuzuba tomohiko9090
0
110
PT AI без купюр
Avatar for Vladimir Kochetkov v0lka
0
200
DevTalks 25 - Create your own AI-infused Java apps with ease
Avatar for Kevin Dubois kdubois
2
130
抽象データ型について学んだ
Avatar for ryounasso ryounasso
0
210
バランスを見極めよう！実装の意味を明示するための型定義 TSKaigi 2025 Day2 (5/24)
Avatar for whatasoda whatasoda
2
780
Cursor Meetup Tokyo ゲノミクスとCursor： 進化と制約のあいだ
Avatar for Masaru Koido koido
1
430
Feature Flag 自動お掃除のための TypeScript プログラム変換
Avatar for azarashi azrsh
PRO
4
630
Devinで実践する！AIエージェントと協働する開発組織の作り方
Avatar for Masahiro Nishimi masahiro_nishimi
6
2.6k
Practical Domain-Driven Design - Workshop at NDC 2025
Avatar for Mufrid Krilic mufrid
0
130
漸進。
Avatar for TOMIKAWA Sotaro ssssota
0
1.2k
バリデーションライブラリ徹底比較
Avatar for Yuta Tanaka nayuta999999
1
460
從零到一：搭建你的第一個 Observability 平台
Avatar for Blueswen blueswen
0
230

Featured

See All Featured
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
307
110k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
76
5.8k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
6.6k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
430
65k
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
56
9.4k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
15
890
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.6k
It's Worth the Effort
Avatar for 3n 3n
184
28k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
35
6.7k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
331
21k
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
28
3.8k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!