Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
85
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
570
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
410
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
CSC509 Lecture 08
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
260
釣り地図SNSにおける有料機能の実装
Avatar for nokonoko1203 nokonoko1203
0
200
CSC305 Lecture 08
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
280
Catch Up: Go Style Guide Update
Avatar for ANDPAD inc andpad
0
260
Blazing Fast UI Development with Compose Hot Reload (droidcon London 2025)
Avatar for Márton Braun zsmb
0
250
なんでRustの環境構築してないのにRust製のツールが動くの？ / Why Do Rust-Based Tools Run Without a Rust Environment?
Avatar for TOMIKAWA Sotaro ssssota
14
46k
マンガアプリViewerの大画面対応を考える
Avatar for kk__777 kk__777
0
360
AkarengaLT vol.38
Avatar for hashimoto-kei hashimoto_kei
1
130
SwiftDataを使って10万件のデータを読み書きする
Avatar for akidon0000 akidon0000
0
240
Towards Transactional Buffering of CDC Events @ Flink Forward 2025 Barcelona Spain
Avatar for Hans-Peter Grahsl hpgrahsl
0
120
社会人になっても趣味開発を続けたい！ / traPavilion
Avatar for mazrean mazrean
1
110
Reactive Thinking with Signals and the Resource API
Avatar for Manfred Steyer manfredsteyer
PRO
0
120

Featured

See All Featured
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
53k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
190
11k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
285
14k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
36
6.1k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
23
1.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
116
20k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
21k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
173
15k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
697
190k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
35
3.2k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.8k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!