Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
88
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
590
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
今、アーキテクトとして 品質保証にどう関わるか
Avatar for Nealle nealle
0
200
API Platformを活用したPHPによる本格的なWeb API開発 / api-platform-book-intro
Avatar for Takashi Kanemoto ttskch
1
110
PostgreSQL を使った快適な go test 環境を求めて
Avatar for Kotaro Otaka otakakot
0
370
要求定義・仕様記述・設計・検証の手引き - 理論から学ぶ明確で統一された成果物定義
Avatar for Kuniwak orgachem
PRO
1
500
AI活用のコスパを最大化する方法
Avatar for Ochtum ochtum
0
120
朝日新聞のデジタル版を支えるGoバックエンド ー価値ある情報をいち早く確実にお届けするために
Avatar for JunkiIshida junkiishida
1
290
メタプログラミングで実現する「コードを仕様にする」仕組み/nikkei-tech-talk43
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
0
140
grapheme_strrev関数が採択されました（あと雑感）
Avatar for てきめん tekimen youkidearitai
PRO
1
190
Python’s True Superpower
Avatar for Hynek Schlawack hynek
0
190
モジュラモノリスにおける境界をGoのinternalパッケージで守る
Avatar for magavel magavel
0
3.3k
AHC061解説
Avatar for Shun_PI shun_pi
0
280
DevinとClaude Code、SREの現場で使い倒してみた件
Avatar for karia/Y.Hisamatsu karia
1
810

Featured

See All Featured
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
5.8k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
Everyday Curiosity
Avatar for Cassini Nazir cassininazir
0
150
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
7.1k
Information Architects: The Missing Link in Design Systems
Avatar for Michelle Chin soysaucechin
0
810
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
2.3k
Chasing Engaging Ingredients in Design
Avatar for Sebastian Deterding codingconduct
0
130
Fireside Chat
Avatar for paigeccino paigeccino
41
3.8k
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
620
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
356
21k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.6k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.4k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!