elasticsearch and Tire

Transcript

1. elasticsearch and Tire
2. • Travis Douce • travis@bignerdranch.com • travisdouce@gmail.com
3. Elasticsearch Tire
4. • Do you need elasticsearch? What will we be talking about?
5. • Do you need elasticsearch? • What is elasticsearch What will we be talking about?
6. • Do you need elasticsearch? • What is elasticsearch • What I needed to understand to be productive with Elasticsearch and Tire – Document Oriented – Restful – Searching Basics (Basic queries, Facets) What will we be talking about?
7. Do I need elasticsearch? http://diabetesadvocacycom.blogspot.com/2012/03/i-dont-know.html
8. • All books with “Apple” in the title • All book authors who have written books with “Apple” in title • Searching text for words that sound like “season” • Auto-completing a search box based on previously issued searches while accounting for mis-spellings http://exploringelasticsearch.com/book/an-overview/what-is-elasticsearch.html
9. Solving problems for which relational databases are optimized. – Calculating how many items are left in the inventory – Figuring out the sum of all line-items on all the invoices sent out in a given month – Executing two operations transactionally with rollback support – Creating records that are guaranteed to be unique across multiple given terms, for instance a phone number and extension http://exploringelasticsearch.com/book/an-overview/what-is-elasticsearch.html
10. Who indexes anyway?
11. What is Elasticsearch? • Tool for querying written words • Standalone database server, written in Java, that takes data in and stores it in a sophisticated format optimized for language based searches • Main protocol is implemented with HTTP/JSON
12. Lucene Elasticsearch
13. http://www.elasticsearch.org/overview/ • Document oriented • Real time analytics • Distributed • High availability • Restful api • Multi-tenancy • Full text search • Conflict management • Schema free • Real time data • Per-operation persistence • Apache 2 open source license • Build on top of apache lucene ™ What I needed to understand to be productive
14. • Document oriented • Restful api • Full text search • Schema free • Real time data What I needed to understand to be productive
15. • Document oriented • Restful api • Full text search • Schema free • Real time data What I needed to understand to be productive
16. • Document oriented • Restful api • Full text search • Schema free • Real time data What I needed to understand to be productive
17. 1. Document Oriented JSON : The Language of Elasticsearch
18. Analogy elasticsearch to RDMS Index : Database Type : table Type mapping: schema document : table row document : table row document : table row document : table row document : table row document : table row Type : table Type mapping: schema
19. Documents
20. Documents id
21. document : like a row in RDBMS id : like a primary key in RDBMS http://wac.450f.edgecastcdn.net/80450F/k2radio.com/files/2012/11/food-for-thought-630x596.jpg
22. Field Field
23. Field Field
24. Field • Smallest individual unit of data • Has a defined type and has one or many values of that type. • Contains a single piece of data: – Like the number 42 – The string "Hello, World!" – An array like: [5, 6, 7, 8].
25. Type • List of fields that can be specified for documents of that type. • Each document has a type mapping – Either user-defined, or – Inferred • Defines the types of its fields (integer, string, etc.) http://exploringelasticsearch.com/
26. Type : analogous to a database table RDBMS http://wac.450f.edgecastcdn.net/80450F/k2radio.com/files/2012/11/food-for-thought-630x596.jpg
27. Data Types
28. Example Type Mapping
29. Example Type Mapping
30. Example Type Mapping
31. Example Type Mapping
32. Example Type Mapping
33. Index • The largest single unit of data is an index • Documents are unique per-index http://exploringelasticsearch.com/
34. Index Type Type
35. Indexes : analogous to a database in traditional RDBMS http://wac.450f.edgecastcdn.net/80450F/k2radio.com/files/2012/11/food-for-thought-630x596.jpg
36. Restful • Tends to match HTTP verbs up to the Create, Read, Update, and Delete operations that are fundamental to most databases http://exploringelasticsearch.com/
37. Restful examples Create Index Create Document Retrieve Document Update Document Delete Document
38. Movie Web Application • Want to be able to search movies by – Movie description – Movie name – Actor name • Sort results in direction • Sort results by an attribute • Facet on genre – Aggregate counts of distinct genres within the result-set
39. Database schema • Database tables – Movies table • Name : “string” • description : “string” – Genres table • Name : “string” – Actors table • first_name : “string” • last_name : “string”
40. 1 1 1 many many
41. 1 many 1 many
42. Elasticsearch : Search Basics Create Index
43. Search Basics : Integrating Tire • Domain Specific Language (DSL) for Elasticsearch. • ActiveModel integration in Rails applications • At this point, ElasticSearch and Tire would just work • No type mapping provided • Elasticsearch infers type mapping • But....would only be able to search attributes on Movie model/table Include Tire Index name
44. elasticsearch : Search Basics • Create 4 Documents
45. • When Create in database, then create a document • Indexed document represents a denormalized table – Do not perform joins in index Tire : Search Basics Create Elasticsearch documents. Now, we need custom type mapping Tire method
46. • When Create, Update, Delete records in database, then do the same to the document in the index • Real Time Tire : Search Basics Stay in Sync
47. elasticsearch : Search Basics • All elasticsearch queries boil down to the task of restricting the result set, scoring, then sorting • Searches are handled by Search API. This API has several other APIs nested inside of it: • Query DSL, • Filter API, • Facet API, and • Sort API.
48. elasticsearch : Search Basics Custom type mapping • Search by • Title
49. elasticsearch : Search Basics Custom type mapping • Search by • Title • Actor name
50. elasticsearch : Search Basics Custom type mapping • Search by • Title • Actor name • Facet on genre (“index” : “not analyzed”)
51. elasticsearch : Search Basics Custom type mapping • Search by • Title • Actor name • Facet on genre (“index” : “not analyzed”) • Description
52. Tire : Search Basics Custom Type Mapping
53. elasticsearch : String Query Index Search endpoint Http verb Query field description for “rollerblading”
54. Tire : String Query
55. Tire : String Query Make search dynamic: query_string → 'rollerblading'
56. elasticsearch : String Query
57. elasticsearch : String Query ….but what ???? How did elasticsearch understand that 'rollerblading' and 'rollerblades' were related words?
58. elasticsearch : String Query ….but what ???? How did elasticsearch understand that 'rollerblading' and 'rollerblades' were related words? 'snowball analyzer' in the type mapping
59. elasticsearch : String Query Remember, the analyzer defined in the custom 'type mapping'!
60. elasticsearch : Snowball Analyzer “rollerblading”, “rollerblader”, and “rollerbladed” has been cut down to its stem, “rollerblad”
61. elasticsearch : Scoring Scores the documents based on similarity to the query • The similarity’s value is usually known as the document’s score
62. elasticsearch : Scoring Scores the documents based on similarity to the query • The similarity’s value is usually known as the document’s score
63. elasticsearch : Results It's JSON !!!!!
64. elasticsearch : Results It's JSON !!!!! score
65. elasticsearch : Results It's JSON !!!!! score Number of hits
66. elasticsearch : Results It's JSON !!!!! score Number of hits Entire document returned
67. elasticsearch : Query Options Other parameters and options, such as: • Maximum result set size, • Result offset location • Search phrases • 'fuzzy' querying ("skateboarig" matches 'skateboarding”) • 36 different query types, and 25 different filter types
68. Tire : Query Options
69. Tire : Dynamic Query Options Sort results: Sort direction Sort on Result set size Make search dynamic
70. 3.4 Facets
71. elasticsearch : Facets • Aggregate statistics for query results • Example: – Consider a user searching for movies by title – Provide aggregate counts of distinct genres within the result-set.
72. elasticsearch : Facets • Disable analysis with "index": "not_analyzed". Otherwise: • "Romance" would be transformed to "romanc" • "Science Fiction" would be aggregated as two separate categories “science” and “fiction”.
73. elasticsearch : Facets • Searches for movies with a description that contains the word “rollerblading”
74. elasticsearch : Facets • Results contains another 'key', 'facets' • 3 facets
75. Tire : Facets Facet name Facet on
76. Other Popular functionalities • Filters • Highliting • Pagination • Search suggest • Many more
77. Goals : Movie Web Application • Want to be able to search movies by – Movie description – Movie name – Actor name • Sort results in direction • Sort results by • Facet on genre – Aggregate counts of distinct genres within the result-set
78. Goals Achieved! Search by Actor name Movie description Movie name Sort on Sort by Facet on genre
79. Helpful bits of technology • https://github.com/mobz/elasticsearch-head • https://httpkit.com/resources/HTTP-from-the-Comm
80. Similar Technologies • Solr • Sphinx
81. Resources • https://httpkit.com/resources/HTTP-from-the-Command-Line/#make-a-request • http://exploringelasticsearch.com/book/an-overview/what-is-elasticsearch.htmL • https://github.com/karmi/retire
82. • curl -X POST -d '{"query": {"match": {"_all": "story"}}}' http://localhost:9200/movie_db/_search?pretty=true • curl -X POST -d '{"query": {"match": {"description": "hacking"}}, "facets": {"genre": {"terms":{"field": "genre"}}}}' http://localhost:9200/movie_db/_search? pretty=true
83. Elasticsearch Searching Steps • The first step is matching all documents that meet the given criteria • The second step is scoring the documents based on similarity to the query •