Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
360
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
150
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
94
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
640
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
440
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
250
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.6k

Other Decks in Programming

See All in Programming
自動化したのに回らない テスト運用の壁 ―AI時代の品質責任と生産性
Avatar for Masahiko Funaki(舟木 将彦) mfunaki
0
260
php-fpmのプロセスが枯渇した日-調査・対処・そして本当にやるべきだったこと-
Avatar for にわ shibuchaaaan
0
300
仕様駆動開発の消費期限
Avatar for watany watany
20
8.5k
Building a Meta Ray-Ban display app
Avatar for Akio Itaya akkeylab
0
160
属人化した知識を、 AIが辿れる地図にする
Avatar for PKSHA Technology（パークシャテクノロジー） pkshadeck
PRO
1
180
わからない話を追いかけたら、プログラミング言語を作る側にいた
Avatar for ydah ydah
3
520
PHP初心者セッション2026　〜生成AIでは見えない裏側を知る：今だからLAMPを通して仕組みを学ぶ〜
Avatar for Hideo Kashioka kashioka
0
940
ドリフトを絶対に許さない（？）CDK運用 / CDK Ops with Zero Tolerance for Drifts (?)
Avatar for アキキー | Akihisa Ikeda akihisaikeda
1
200
freeeにおけるEvalsの実践例の紹介
Avatar for freee freee
PRO
0
120
AI時代に設計が 最大の生産性レバーになる 意図駆動開発とデータを消さない設計｜Don't Delete Your Data or Your Intent — Design as the Deepest Lever in the AI Era
Avatar for Tomohisa Takaoka tomohisa
1
980
<title><a id="</title>君はこのHTMLをパースできるか"></a></title> #雑LT_study
Avatar for pizzacat83 pizzacat83
0
150
komatsuna「分散システムにおけるバグ分析手法」
Avatar for こまつな komatsunaqa
0
260

Featured

See All Featured
Public Speaking Without Barfing On Your Shoes - THAT 2023
Avatar for David Neal reverentgeek
1
530
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
280
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
56k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
12
1.2k
<Decoding/> the Language of Devs - We Love SEO 2024
Avatar for Nikki Halliwell nikkihalliwell
1
290
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
270
SERP Conf. Vienna - Web Accessibility: Optimizing for Inclusivity and SEO
Avatar for Sara Fernández Carmona sarafernandez
2
1.6k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
250
The Anti-SEO Checklist Checklist. Pubcon Cyber Week
Avatar for Ryan Jones ryanjones
0
210
Heart Work Chapter 1 - Part 1
Avatar for Lake Fama lfama
PRO
8
36k
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
30k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!