Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
560
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
400
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
そのAPI、誰のため？ Androidライブラリ設計における利用者目線の実践テクニック
Avatar for mkeeda mkeeda
2
240
Vue・React マルチプロダクト開発を支える Vite
Avatar for ANDPAD inc andpad
0
110
How Android Uses Data Structures Behind The Scenes
Avatar for HyunWoo Lee l2hyunwoo
0
360
Compose Multiplatform × AI で作る、次世代アプリ開発支援ツールの設計と実装
Avatar for Takeshi Hagikura thagikura
0
120
250830 IaCの選定~AWS SAMのLambdaをECSに乗り換えたときの備忘録~
Avatar for Takumi Abe east_takumi
0
380
AI時代のUIはどこへ行く？
Avatar for Yusuke Wada yusukebe
16
8.7k
AIコーディングAgentとの向き合い方
Avatar for kmuto eycjur
0
260
Putting The Genie in the Bottle - A Crash Course on running LLMs on Android
Avatar for Iury Souza iurysza
0
140
アセットのコンパイルについて
Avatar for ojun ojun9
0
120
FindyにおけるTakumi活用と脆弱性管理のこれから
Avatar for adachi.ryo rvirus0817
0
480
go test -json そして testing.T.Attr / Kyoto.go #63
Avatar for utagawa kiki utgwkk
3
280
CJK and Unicode From a PHP Committer
Avatar for てきめん tekimen youkidearitai
PRO
0
110

Featured

See All Featured
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
54
11k
Scaling GitHub
Avatar for Zach Holman holman
463
140k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.8k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.9k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
74
5k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
16k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
339
57k
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
268
13k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
302
21k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
50k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!