Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
370
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
The Knowledge Spine: A Machine-Executable Ontology for Governed Marketing Activation
Avatar for Ananth Packkildurai vananth22
0
130
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
170
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
110
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
680
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
460
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
270
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k

Other Decks in Programming

See All in Programming
ソニーのクラウド共通基盤の変遷と AI時代の開発スタイルに合わせた進化 / The Journey of Sony’s Common Cloud Platform and Its Evolution for AI-Native Development
Avatar for KenjiYoneyama kenjiyoneyama
0
250
パスワード再設定メールのURLについて知っておこう
Avatar for uyuki uyuki234
0
140
Verilogで学ぶCPU自作入門
Avatar for uyuki uyuki234
8
4.1k
Everything will be SERVERLESS — 信じて運用した10年の経験値 / Everything Will be Serverless — Lessons Learned from 10 Years of Operational Experience
Avatar for shiro seike seike460
PRO
1
1k
R8 の設定見直しでアプリサイズ削減
Avatar for ANDPAD inc andpad
0
210
コーディング自動化の次へ。人の介入をゼロにするファクトリーとサポートAIによる実践
Avatar for Seiji Takahashi timakin
2
1.6k
モデルのリファクタリングが難しいと思ったら、そもそも複雑だったのはビジネス仕様だった ? / is-the-business-domain-the-real-complexity
Avatar for hatsu hatsu38
0
470
最近”作らせてる” ローカルAIアプリの話
Avatar for ITO Akihiro akit37
0
460
When benchmarks go bad - what I learned from measuring performance wrong
Avatar for Holly Cummins hollycummins
0
210
APNsからLive Activityを開始する話
Avatar for Ryotaro Onoue yumnumm
0
170
IBM Bob でAI駆動開発『QC-Playbook』
Avatar for 1kano ichirokano2
0
110
HTML-in-Canvasで作る未来のWeb UI表現
Avatar for IKEDA Yasunobu clockmaker
5
4.8k

Featured

See All Featured
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
220
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
750
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
450
Intergalactic Javascript Robots from Outer Space
Avatar for Vicent Martí tanoku
273
27k
The SEO identity crisis: Don't let AI make you average
Avatar for Varn varn
0
590
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
41
2.8k
Ruling the World: When Life Gets Gamed
Avatar for Sebastian Deterding codingconduct
0
400
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
560
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
2
870
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
530
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
850
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
10k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!