Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
550
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
390
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
なぜ今、Terraformの本を書いたのか？ - 著者陣に聞く！『Terraformではじめる実践IaC』登壇資料
Avatar for Ryoma Fujiwara fufuhu
4
620
「リーダーは意思決定する人」って本当？～ 学びを現場で活かす、リーダー4ヶ月目の試行錯誤 ～
Avatar for Marina Nakagawa marina1017
0
220
State of CSS 2025
Avatar for Benjamin Kott benjaminkott
1
110
Comparing decimals in Swift Testing
Avatar for 417.72KI 417_72ki
0
170
ゲームの物理
Avatar for Fadis fadis
5
1.2k
プロダクトという一杯を作る - プロダクトチームが味の責任を持つまでの煮込み奮闘記
Avatar for 幡ヶ谷亭直吉 hiliteeternal
0
460
CEDEC 2025 『ゲームにおけるリアルタイム通信への QUIC導入事例の紹介』
Avatar for SEGADevTech segadevtech
3
890
Nuances on Kubernetes - RubyConf Taiwan 2025
Avatar for Andrey Novikov envek
0
170
コーディングは技術者(エンジニア)の嗜みでして / Learning the System Development Mindset from Rock Lady
Avatar for mackey0225 mackey0225
2
490
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
Avatar for sushi-cat sushi0120
1
280
TROCCO×dbtで実現する人にもAIにもやさしいデータ基盤
Avatar for Nealle nealle
0
240
レガシープロジェクトで最大限AIの恩恵を受けられるようClaude Codeを利用する
Avatar for tk1351 tk1351
2
140

Featured

See All Featured
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
332
24k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
86
4.8k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
96
6.2k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
338
57k
What’s in a name? Adding method to the madness
Avatar for Product Marketing Alliance productmarketing
PRO
23
3.6k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
56
5.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
283
13k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
53
8.8k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
77
5.9k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!