Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Search Infrastructure using Lambda Architecture

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
370
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
The Knowledge Spine: A Machine-Executable Ontology for Governed Marketing Activation
Avatar for Ananth Packkildurai vananth22
0
120
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
160
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
100
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
660
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
450
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
260
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k

Other Decks in Programming

See All in Programming
AHC070解法紹介
Avatar for eijirou eijirou
0
130
What We Talk About When We Talk About XP
Avatar for seki at druby.org m_seki
2
680
海上で動くGoサーバー: goroutineとchannelでさばく航行データストリーム
Avatar for Atsuki Seo atsuki_seo
0
920
ハーネス設計入門 〜 基礎知識の整理から実務へのステップアップ 〜
Avatar for kinopee kinopeee
16
14k
【高い買い物LT会】初任給で話題の国産フィジカルAIを買った話
Avatar for 赤神青空 akagami
PRO
0
170
iOS 27でニュースアプリはどう変わる！？ 〜日経電子版の新機能対応と、開発事例から〜
Avatar for Kazuki Nakashima lynnswap
3
12k
Vue Fes Japan 2026 タイムテーブル徹底解説
Avatar for 448jp | OKI Yoshiya 448jp
1
510
標準パッケージに uuid が追加された 背景から見る Go らしい意思決定 / go_127_uuid_decision
Avatar for convto convto
5
8.3k
Family mrubyの進捗
Avatar for Katsuhiko Kageyama kishima
1
130
MVNOの申込からeSIM開通までをiOSアプリでつなぐ- 本人確認・MNP・通信事業者基盤をまたぐ実装
Avatar for Sato Takeshi satotakeshi
0
470
JPUG勉強会 OSSデータベースの内部構造を理解しよう(第２回)
Avatar for Atsushi Ogawa oga5
0
270
Augmenting AI with the Power of Jakarta EE
Avatar for ivargrimstad ivargrimstad
0
260

Featured

See All Featured
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Done Done
Avatar for chrislema chrislema
186
17k
Everyday Curiosity
Avatar for Cassini Nazir cassininazir
0
320
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
15k
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
Avatar for Tomoya Matsuura tomoyanonymous
3
1.3k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
530
Lessons Learnt from Crawling 1000+ Websites
Avatar for Charles Meaden charlesmeaden
PRO
1
1.6k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
Avatar for Helen Beal helenjbeal
1
340
sira's awesome portfolio website redesign presentation
Avatar for ElsiraPls elsirapls
0
420
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
1
370
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
500

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!