Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
360
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
150
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
96
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
650
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
450
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
260
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.6k

Other Decks in Programming

See All in Programming
Oxlintはいいぞ（続）
Avatar for Yuji Yamaguchi yug1224
1
530
Jindong: Introducing Declarative Haptics in Compose Multiplatform
Avatar for HyunWoo Lee l2hyunwoo
0
140
AIに既存システムを理解させる技術　～レガシーを見捨てないハーネスエンジニアリング入門～
Avatar for Ochtum ochtum
0
200
How I Won Prize Money at a Hackathon Using Codex and Symphony Alpha
Avatar for yasei no otoko yasei_no_otoko
0
140
Press start. Python's next generation.
Avatar for Carol Willing willingc
PRO
3
300
レビュー履歴をAIに食わせて、 Compose移行を加速するs
Avatar for Yuki Shiho shihochan
0
260
関東Kaggler会_NVIDIA_Nemotron_コンペ_振り返り
Avatar for rick rick_ds
0
800
AI駆動開発にグラフDBを重ねてみた
Avatar for Satoshi Kaneyasu satoshi256kbyte
2
700
高専、大学編入、そして未踏へ〜プロダクト開発とキャリアの歩み - Technical College, University Transfer, and On to “Mitou” / My Journey in Product Development and Career
Avatar for __pkmiya pkmiya
0
130
FastAPI の並行処理モデルを完全に理解する
Avatar for Yuki Ishikawa hoto17296
9
3.6k
源内ハンズオン概要編
Avatar for Hideyuki Nagata hideg
0
240
go-spidermonkeyでAIエージェントのCode Modeを実装する
Avatar for syumai syumai
3
1.5k

Featured

See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.3k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
460
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
1
2.9k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.3k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
35
2.6k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
The agentic SEO stack - context over prompts
Avatar for schlessera schlessera
0
880
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
52k
Scaling GitHub
Avatar for Zach Holman holman
464
140k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
5.7k
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
1
1.9k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!