Search Infrastructure using Lambda Architecture

Transcript

1. Search infrastructure using Lambda Architecture Ananth Packkildurai

2. Outline 1.Search @ Slack 2.Lambda Architecture 3.Search infrastructure overview 4.FCC

   Design patterns

3. Search @ Slack

4. http://www.mckinsey.com/industries/high-tech/our-insights/the-social-economy https://slack.engineering/search-at-slack-431f8c80619e

5. Lambda Architecture

6. Stream processing trade-offs Pick 2 Speed Accuracy Volume

7. Lambda Architecture http://lambda-architecture.net/

8. Search Infrastructure overview

9. Search infrastructure overview

10. ! Online sharding by Collections ! Offline sharding by CompositeId

    Router Sharding strategy

11. mutating Index & Delta Replication • What happens if someone

    edits the
 message that indexed in 
 batch mode? • A snapshot occurs at time T1, 
 the index built, and the core 
 comes online at T2.Between T1 & T2, How to handle the update?

12. FCC Pattern

13. Fork Pattern

14. Contract

15. Clone

16. Any Questions? Thanks!