Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
340

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
130
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
88
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
590
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
420
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
240
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.5k

Other Decks in Programming

See All in Programming
CSC307 Lecture 05
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
490
生成AIを使ったコードレビューで定性的に品質カバー
Avatar for Chiaki Okamoto chiilog
1
240
Vibe codingでおすすめの言語と開発手法
Avatar for uyuki uyuki234
0
220
Kotlin Multiplatform Meetup - Compose Multiplatform 외부 의존성 아키텍처 설계부터 운영까지
Avatar for Suhyeon Kim wisemuji
0
190
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
500
高速開発のためのコード整理術
Avatar for sutetotanuki sutetotanuki
1
390
15年続くIoTサービスの SREエンジニアが挑む 分散トレーシング導入
Avatar for Melonps melonps
2
170
なるべく楽してバックエンドに型をつけたい！(楽とは言ってない)
Avatar for HIBIKI CUBE hibiki_cube
0
140
【卒業研究】会話ログ分析によるユーザーごとの関心に応じた話題提案手法
Avatar for MomokoShirakawa momok47
0
190
Unicodeどうしてる？ PHPから見たUnicode対応と他言語での対応についてのお伺い
Avatar for てきめん tekimen youkidearitai
PRO
1
1.1k
0→1 フロントエンド開発 Tips🚀 #レバテックMeetup
Avatar for 弁護士ドットコム bengo4com
0
540
Data-Centric Kaggle
Avatar for ねぼすけAI isax1015
2
760

Featured

See All Featured
Tell your own story through comics
Avatar for letsgokoyo letsgokoyo
1
800
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
520
Heart Work Chapter 1 - Part 1
Avatar for Lake Fama lfama
PRO
5
35k
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
2
210
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
2
930
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
52
5.8k
Designing for Performance
Avatar for Lara Hogan lara
610
70k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
0
220
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.3k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
273
21k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.2k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.6k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!