Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Search Infrastructure using Lambda Architecture

Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
1
330

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
120
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
560
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
400
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.5k
Streaming data pipelines @ Slack
Avatar for Ananth Packkildurai vananth22
2
2.4k

Other Decks in Programming

See All in Programming
Ruby×iOSアプリ開発 ~共に歩んだエコシステムの物語~
Avatar for Tomoki Kobayashi temoki
0
350
OSS開発者という働き方
Avatar for ANDPAD inc andpad
5
1.7k
Laravel Boost 超入門
Avatar for Arlo fire_arlo
3
220
為你自己學 Python - 冷知識篇
Avatar for 高見龍 eddie
1
350
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
410
Navigating Dependency Injection with Metro
Avatar for Zac Sweers zacsweers
3
2.5k
より安全で効率的な Go コードへ： Protocol Buffers Opaque API の導入
Avatar for shoyan shwatanap
2
730
AI時代のUIはどこへ行く？
Avatar for Yusuke Wada yusukebe
18
9k
Reading Rails 1.0 Source Code
Avatar for Masafumi Okura okuramasafumi
0
250
Oracle Database Technology Night 92 Database Connection control FAN-AC
Avatar for oracle4engineer oracle4engineer
PRO
1
470
スケールする組織の実現に向けた インナーソース育成術 - ISGT2025
Avatar for teamLab teamlab
PRO
1
160
テストコードはもう書かない：JetBrains AI Assistantに委ねる非同期処理のテスト自動設計・生成
Avatar for makun makun
0
520

Featured

See All Featured
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
3k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
15
1.7k
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
59
9.5k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
184
22k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
9
810
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
140
7.1k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
33
8.8k
Docker and Python
Avatar for Tania Allard trallard
46
3.6k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.9k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
113
20k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
15k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!