Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
560
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
400
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
ててべんす独演会〜Flowの全てを語ります〜
Avatar for てべすてん tbsten
1
220
CSC305 Lecture 02
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
260
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
Avatar for Samuel Williams ioquatix
3
910
2分台で1500examples完走！爆速CIを支える環境構築術 - Kaigi on Rails 2025
Avatar for Hayato OKUMOTO falcon8823
3
3.4k
Pythonスレッドとは結局何なのか？ CPython実装から見るNoGIL時代の変化
Avatar for curekoshimizu curekoshimizu
5
1.5k
あなたの知らない「動画広告」の世界 - iOSDC Japan 2025
Avatar for ukitaka ukitaka
0
420
GitHub Actions × AWS OIDC連携の仕組みと経緯を理解する
Avatar for Itaru Ota ota1022
0
240
The Flutter Journey of Building a Live Streaming App — With a Side of Performance Tuning
Avatar for Sugiy u503
1
100
複雑化したリポジトリをなんとかした話　pipenvからuvによるモノレポ構成への移行
Avatar for Satoshi Kaneyasu satoshi256kbyte
1
850
株式会社 Sun terras カンパニーデック
Avatar for Sun terras.Inc sunterras
0
250
Django Ninja による API 開発効率化とリプレースの実践
Avatar for Kashun Yoshida kashewnuts
0
1.1k
開発生産性を上げるための生成AI活用術
Avatar for starfish719 starfish719
1
200

Featured

See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
46
7.6k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
114
20k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1371
200k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
411
22k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
64
7.9k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
43
7.7k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
9.7k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
333
22k
Visualization
Avatar for Eitan Lees eitanlees
148
16k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
1.6k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
224
10k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!