Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
360
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
150
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
93
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
630
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
440
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
250
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.6k

Other Decks in Programming

See All in Programming
アルゴリズムは何を圧縮しているのか ─ Haskell から育った「圧縮代数」というメンタルモデル
Avatar for Naoya Ito naoya
16
3.6k
Terraform標準の組織で AWS CDKをどう使うか
Avatar for mu7889yoon / Yuta Nakamura mu7889yoon
1
390
【やさしく解説 設計編 #1】「ドメイン駆動」と「実装駆動」ってなに？ 〜設計の考え方を、たとえ話で学ぼう〜
Avatar for HideyukiKitao panda728
PRO
1
140
PHPだって関数型したい 〜できること、できないこと〜 / fp-in-php
Avatar for jsoizo jsoizo
1
250
yield再入門 #phpcon
Avatar for hideki kinjyo o0h
PRO
0
800
Prismを使った型安全な暗号化_関数型まつり2026
Avatar for _fhhmm _fhhmm
0
150
ソフトウェア設計に溶けるインフラ ― AWS CDK のインフラ認識論
Avatar for Kenji Kono konokenj
3
670
Apache Hive: Toward a Cloud Native Lakehouse
Avatar for okumin okumin
0
160
PHP に部分適用が来るぞ！……ところで何それ？おいしいの？ #phpcon / phpcon-2026
Avatar for shogogg shogogg
0
400
20260722_microCMSで考える、AI時代のコンテンツ運用設計
Avatar for Yoshihisa Kaino yosh1
0
130
광주소프트웨어마이스터고등학교 DevFest 특강 - 바이브 코딩 시대에서 주니어 개발자로 살아남는 방법
Avatar for Chris Ohk utilforever
1
150
テーブルをDELETEした
Avatar for yuzneri yuzneri
0
110

Featured

See All Featured
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
52k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
560
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
1
1.6k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
660
Public Speaking Without Barfing On Your Shoes - THAT 2023
Avatar for David Neal reverentgeek
1
460
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
10k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
56k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
201
75k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
340
Believing is Seeing
Avatar for Spiro Bolos oripsolob
1
170
A Soul's Torment
Avatar for Nat Ma seathinner
6
3.1k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!