Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
320

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
100
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
83
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
520
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
380
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
190
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
プロフェッショナルとしての成長「問題の深掘り」が導く真のスキルアップ / issue-analysis-and-skill-up
Avatar for MinoDriven minodriven
8
1.9k
Ruby on Railroad: The Power of Visualizing CFG
Avatar for ydah ydah
0
310
Vibe Coding の話をしよう
Avatar for schroneko schroneko
14
3.8k
2025-04-25 GitHub Copilot Agent ライブデモ（スクリプト）
Avatar for GOAMI Takaaki goataka
0
110
fieldalignmentから見るGoの構造体
Avatar for kuro kuro_kurorrr
0
130
The Missing Link in Angular’s Signal Story: Resource API and httpResource
Avatar for Manfred Steyer manfredsteyer
PRO
0
140
今話題のMCPサーバーをFastAPIでサッと作ってみた
Avatar for Yuuki Shimizu yuukis
0
130
API for docs
Avatar for Soutaro Matsumoto soutaro
4
1.7k
On-the-fly Suggestions of Rewriting Method Deprecations
Avatar for Masato Ohba ohbarye
3
5.2k
監視 やばい
Avatar for syossan27 syossan27
12
10k
オープンソースコントリビュート入門
Avatar for katsuma _katsuma
0
130
Making TCPSocket.new "Happy"!
Avatar for Misaki Shioi(塩井美咲/しおい) coe401_
1
3.4k

Featured

See All Featured
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
52
2.5k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
29
9.5k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
230
18k
Building an army of robots
Avatar for Kyle Neath kneath
305
45k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
6
540
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
26k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
23
2.7k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.7k
The Language of Interfaces
Avatar for Des Traynor destraynor
158
25k
Visualization
Avatar for Eitan Lees eitanlees
146
16k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
45
7.2k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!