Search API in Ruby with ES and complicated factors

Presented in Vietnamese Engineer in Japan: first seminar on 2019/04/13.

Transcript

1. Search API in Ruby with ES and complicated factors Created by: Trung Vu @chariot9 Vietnamese engineer in Japan: Seminar on 2019/04/13
2. Contents INFORMATION TECHNOLOGY 1. What I'm going to build 2. What kind of technologies? 3. Elasticsearch in deep: Distributed Search 4. Architecture of simple search API 5. Filtering and scoring with functions Vietnamese engineer in Japan
3. 1. What I'm going to build INFORMATION TECHNOLOGY As you know, we are searching everywhere Vietnamese engineer in Japan
4. 2. What I'm going to build INFORMATION TECHNOLOGY Want to know how… Vietnamese engineer in Japan
5. 3. What I'm going to build INFORMATION TECHNOLOGY Created a simple search API in 3 weeks Source code: https://github.com/chariot9/cozy-scout Vietnamese engineer in Japan
6. 2. What kind of technologies? INFORMATION TECHNOLOGY With only: • Ruby on rails • Elasticsearch Vietnamese engineer in Japan
7. 3. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Elasticsearch is a real-time distributed and open source full-text search and analytics engine. • Elasticsearch is scalable up to petabytes of structured and unstructured data. • Elasticsearch can be used as a replacement of document stores • Elasticsearch uses denormalization to improve the search performance. • Elasticsearch is one of the popular enterprise search engines, clients: Wikipedia, The Guardian, StackOverflow, GitHub etc. • Elasticsearch is open source and available under the Apache license version 2.0. Vietnamese engineer in Japan
8. 3. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Node: It refers to a single running instance of Elasticsearch. Single physical and virtual server accommodates multiple nodes depending upon the capabilities of their physical resources like RAM, storage and processing power. Vietnamese engineer in Japan
9. 3. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Cluster: It is a collection of one or more nodes. Cluster provides collective indexing and search capabilities across all the nodes for entire data. Vietnamese engineer in Japan
10. 3. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Index: It is a collection of different type of documents and document properties. Index also uses the concept of shards to improve the performance. For example, a set of document contains data of a social networking application. Vietnamese engineer in Japan
11. 3. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Type/Mapping: It is a collection of documents sharing a set of common fields present in the same index. For example, an Index contains data of a social networking application, and then there can be a specific type for user profile data, another type for messaging data and another for comments data. Vietnamese engineer in Japan
12. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Document: It is a collection of fields in a specific manner defined in JSON format. Every document belongs to a type and resides inside an index. Every document is associated with a unique identifier, called the UID. Vietnamese engineer in Japan
13. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Shard: Indexes are horizontally subdivided into shards. This means each shard contains all the properties of document, but contains less number of JSON objects than index. The horizontal separation makes shard an independent node, which can be store in any node. Primary shard is the original horizontal part of an index and then these primary shards are replicated into replica shards. Vietnamese engineer in Japan
14. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Replicas: Elasticsearch allows a user to create replicas of their indexes and shards. Replication not only helps in increasing the availability of data in case of failure, but also improves the performance of searching by carrying out a parallel search operation in these replicas. Vietnamese engineer in Japan
15. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Example: 1 cluster with 3 nodes PC1 PC2 PC3 Vietnamese engineer in Japan
16. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Where Elasticsearch stores data? shard = hash(routing) % number_of_primary_shards Using hash function to know which shark will store data: Vietnamese engineer in Japan
17. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Steps to store data: Vietnamese engineer in Japan
18. 4. Elasticsearch in deep: Distributed Search INFORMATION TECHNOLOGY Steps to get data: Vietnamese engineer in Japan
19. 4. Architecture of simple search API INFORMATION TECHNOLOGY [White board sections] Vietnamese engineer in Japan
20. 5. Filtering and scoring with functions INFORMATION TECHNOLOGY Let’s start with sample data on Elasticsearch: Sample data set contains company information like: • Id, tax code, name in English, name in Vietnamese • Address, phone number, website, number of employee • … Vietnamese engineer in Japan
21. 5. Filtering and scoring with functions INFORMATION TECHNOLOGY In the real search API, we are using many steps to get the correct results, but in my sample search API, I just developed only two most important phases: 1. Filtering: filter to ignore non-match data with input information 2. Scoring: score list of records by score functions Vietnamese engineer in Japan
22. 5. Filtering and scoring with functions INFORMATION TECHNOLOGY 5.1 Filtering With my sample data set, for filtering, I used the sample filter functions: - Ignore deactivated company - Ignore marked as delete company - Matching keyword Vietnamese engineer in Japan
23. 5. Filtering and scoring with functions INFORMATION TECHNOLOGY 5.1 Filtering Vietnamese engineer in Japan
24. 5. Filtering and scoring with functions INFORMATION TECHNOLOGY 5.2 Scoring With my sample data set, for scoring, I used the sample score function: - Attractive: the number of employees Vietnamese engineer in Japan
25. 5. Filtering and scoring with functions INFORMATION TECHNOLOGY 5.1 Scoring Vietnamese engineer in Japan
26. Conclusion: INFORMATION TECHNOLOGY With Elasticsearch, we are able to build a scalable-flexible- accurate search API Vietnamese engineer in Japan
27. Q&A? INFORMATION TECHNOLOGY Vietnamese engineer in Japan