Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
350
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
140
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
90
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
610
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
440
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
「OSSがあるなら自作するな」は AI時代も正しいか ── Build vs Adopt の新しい判断基準
Avatar for Kumo Ishikawa kumorn5s
7
2.7k
PHPでバイナリをパースして理解するASN.1
Avatar for muno92 muno92
PRO
0
460
書籍「ユーザーストーリーマッピング」が私のバイブル
Avatar for asumikam asumikam
4
490
Lightning-Fast Method Calls with Ruby 4.1 ZJIT / RubyKaigi 2026
Avatar for Takashi Kokubun k0kubun
3
3k
運転動画を検索可能にする〜Cosmos-Embed1とDatabricks Vector Searchで〜/cosmos-embed1-databricks-vector-search
Avatar for 開発室Graph studio_graph
3
920
PHPer、Cloudflare に引っ越す
Avatar for Suguru Oki / スー suguruooki
2
200
Terraform言語の静的解析 / static analysis of Terraform language
Avatar for Kazuma Watanabe wata727
1
150
Liberating Ruby's Parser from Lexer Hacks
Avatar for ydah ydah
2
2.7k
How We Practice Exploratory Testing in Iterative Development( #scrumniigata ) / 反復開発の中で、探索的テストをどう実施しているか
Avatar for teyamagu teyamagu
PRO
3
820
ローカルLLMでどこまでコードが書けるか / How much code can be written on a local LLM
Avatar for Naoki Kishida kishida
2
360
KMP × Kotlin 2.3 - How Android Got Slower While iOS Builds Improved by 47%
Avatar for Ryosuke Shimizu (RIO) rio432
0
190
Cloudflare で始める Data Platform
Avatar for Takumi Abe ta93abe
0
100

Featured

See All Featured
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
370
Public Speaking Without Barfing On Your Shoes - THAT 2023
Avatar for David Neal reverentgeek
1
390
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
150
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
1
740
Abbi's Birthday
Avatar for Violet Bock coloredviolet
2
7.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.4k
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
3
350
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon Słowik szymonslowik
1
1.1k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
1.9k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.7k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
0
280

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!