Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
360
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
150
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
92
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
630
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
440
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
250
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
220
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.6k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.6k

Other Decks in Programming

See All in Programming
スマートグラスで並列バイブコーディング
Avatar for Hayami Shuhei hyshu
0
270
エージェンティックRAGにAWSで入門しよう！
Avatar for Har1101 har1101
9
1.8k
Agentic UI
Avatar for Manfred Steyer manfredsteyer
PRO
0
210
Hunting Vulnerabilities in Symfony with LLMs
Avatar for Vincent Amstoutz vinceamstoutz
0
570
どこまでゆるくて許されるのか
Avatar for tk3fftk tk3fftk
0
290
代数的データ型って何が嬉しいの？ #frontend_phpcon_do
Avatar for Takuma Kajikawa kajitack
8
3.8k
キャリア迷子上等 ─ "ない道"は自分で作ればいい
Avatar for 16bit_idol 16bitidol
3
2.6k
なぜ型を書くのか？ TSKaigi2026で改めて考える #tskaigi_smarthr
Avatar for Takuma Kajikawa kajitack
0
180
正しくソフトウェアを作る、前提を疑うための認知の視点 / doubt-premise
Avatar for MinoDriven minodriven
21
7.1k
TSKaigi Night Talks 2026_TypeScriptでサプライチェーンの整合性を型に閉じ込める
Avatar for ギークプラス ソフトウェア事業部 geekplus_tech
0
420
そのテスト、説明できますか？~LWテスト戦略FW~のご紹介
Avatar for Kei Nakahara nakahara
0
180
「なぜそう決めたのか」を残し続ける仕組み ― Notion AI カスタムエージェント × Slack連携による設計判断の自動記録 - NIKKEI Tech Talk #47
Avatar for ニフティ株式会社 niftycorp
PRO
0
250

Featured

See All Featured
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
1
2.8k
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
55k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.1k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
356
21k
Tips & Tricks on How to Get Your First Job In Tech
Avatar for Honza Javorek honzajavorek
1
550
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
123
22k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.2k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
3
1.1k
Leadership Guide Workshop - DevTernity 2021
Avatar for David Neal reverentgeek
1
320
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
1
360
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
230
23k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!