Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Search Infrastructure using Lambda Architecture

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Ananth Packkildurai Ananth Packkildurai
May 31, 2017
Programming
370
1

Search Infrastructure using Lambda Architecture

Search

Avatar for Ananth Packkildurai

Ananth Packkildurai

May 31, 2017

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
The Knowledge Spine: A Machine-Executable Ontology for Governed Marketing Activation
Avatar for Ananth Packkildurai vananth22
0
140
Data Contracts & Domain Ownership
Avatar for Ananth Packkildurai vananth22
0
170
Data Catalogs - Rebuild the Broken Promise
Avatar for Ananth Packkildurai vananth22
0
110
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
Avatar for Ananth Packkildurai vananth22
0
680
Back To The Future: Emerging Trends in Data Engineering
Avatar for Ananth Packkildurai vananth22
0
1.3k
Murron: A Reliable Monitoring Pipeline
Avatar for Ananth Packkildurai vananth22
0
460
The_journey_towards_Pinot.pdf
Avatar for Ananth Packkildurai vananth22
0
270
Reliable_Event_Pipeline___scale.pdf
Avatar for Ananth Packkildurai vananth22
0
230
Operating Data Pipeline with Airflow @ Slack
Avatar for Ananth Packkildurai vananth22
1
2.7k

Other Decks in Programming

See All in Programming
GraphQL の onError と Nullability のこれから
Avatar for Yosuke Kurami quramy
0
200
Kotlinize Is Not Just Converting to Kotlin
Avatar for Kazuki Chigita chigichan24
1
220
[2026-10-14](C)Re:ゼロから始める異世界生活〜始まりの終わりと終わりの始まり〜
Avatar for tosite tosite
0
310
はじめてのPLATEAU
Avatar for nokonoko1203 nokonoko1203
1
260
AI-Powered Domain-Driven Design
Avatar for Alberto Brandolini ziobrando
0
380
AIがコードを書いてくれる時代、我々は何を確かめるのか
Avatar for Yosuke Furukawa yosuke_furukawa
PRO
13
5.6k
AI時代のFlutter品質設計 — アプリ品質を守るQuality Layers
Avatar for omatty198 omatty198
0
350
Seven New Principles of Software Engineering
Avatar for Yegor Bugayenko yegor256
0
440
[JA] Domain Modeling with Vanilla Rails: 「操作」から再考するRailsの設計論
Avatar for Yuichi Goto yasaichi
4
1.8k
そのデザイントークン、Flutterでも同じ見た目になりますか？
Avatar for nakamizu nakamizu
0
300
No More NoSuchMethodError
Avatar for TATSUNO Yasuhiro exoego
1
120
Kotlinの15年とServer-Side Kotlin
Avatar for Takehata Naoto n_takehata
1
190

Featured

See All Featured
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
22k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
1
600
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
3
630
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
61
10k
Deep Space Network (abreviated)
Avatar for Tony Rice tonyrice
0
400
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
4
1.2k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
1
650
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
6.3k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
6
1.4k
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
510
Abbi's Birthday
Avatar for Violet Bock coloredviolet
4
11k

Transcript

  1. Search infrastructure using Lambda Architecture Ananth Packkildurai

  2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

    Design patterns

  3. Search @ Slack

  4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

  5. Lambda Architecture

  6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

  7. Lambda Architecture http://lambda-architecture.net/

  8. Search Infrastructure overview

  9. Search infrastructure overview

  10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

  11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

  12. FCC Pattern

  13. Fork Pattern

  14. Contract

  15. Clone

  16. Any Questions? Thanks!