Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
540
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
390
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
AWS Summit Japan 2024と2025の比較／はじめてのKiro、今あなたは岐路に立つ
Avatar for Satoshi Kaneyasu satoshi256kbyte
0
120
High-Level Programming Languages in AI Era -Human Thought and Mind-
Avatar for Hayato Ishida hayat01sh1da
PRO
0
880
PHP 8.4の新機能「プロパティフック」から学ぶオブジェクト指向設計とリスコフの置換原則
Avatar for 武田 憲太郎 kentaroutakeda
2
1k
バイブコーディング超えてバイブデプロイ〜CloudflareMCPで実現する、未来のアプリケーションデリバリー〜
Avatar for azukiazusa azukiazusa1
0
330
システム成長を止めない！本番無停止テーブル移行の全貌
Avatar for さかうぇ sakawe_ee
1
360
『自分のデータだけ見せたい！』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
Avatar for AkitoTsukahara akitotsukahara
2
660
ニーリーにおけるプロダクトエンジニア
Avatar for Nealle nealle
0
950
#QiitaBash MCPのセキュリティ
Avatar for Tommy(sigma) ryosukedtomita
1
1.5k
Startups on Rails in Past, Present and Future–Irina Nazarova, RailsConf 2025
Avatar for Irina Nazarova irinanazarova
0
240
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
Avatar for kesoji kesoji
0
300
はじめてのWeb API体験 ー 飲食店検索アプリを作ろうー
Avatar for Aki Nakahara akinko_0915
0
140
Quand Symfony, ApiPlatform, OpenAI et LangChain s'allient pour exploiter vos PDF : de la théorie à la production…
Avatar for Ahmed EBEN HASSINE ahmedbhs123
0
220

Featured

See All Featured
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
328
39k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
278
23k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.4k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
138
34k
Gamification - CAS2011
Avatar for David Bonilla davidbonilla
81
5.4k
Writing Fast Ruby
Avatar for Erik Berlin sferik
628
62k
Code Review Best Practice
Avatar for Trisha Gee trishagee
69
19k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
346
40k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
63
7.8k
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
271
27k
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
28
3.9k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!