Elasticsearch Introduction

My slides about elasticsearch and especially about text analysis. Video (in Russian): http://www.youtube.com/watch?v=fe6kLv4UcVY

From Minsk Ruby Meetup: http://brug.by/events/ruby-meetup-september-14.

Transcript

2. elasticsearch
3. me Developer at Twinslash email: dima.zhlobo@gmail.com skype: dima.zhlobo github: proghat twitter: @proghat Dmitry Zhlobo
4. search is hard • speed vs. relevancy
5. search is hard • speed vs. relevancy • real time
6. search is hard • speed vs. relevancy • real time • different kinds of data
7. usual approach • SELECT * FROM posts WHERE `body` LIKE '%query%'
8. usual approach • SELECT * FROM posts WHERE `body` LIKE '%query%' • gem 'thinking-sphinx' … Article.search(params[:q])
9. usual approach • SELECT * FROM posts WHERE `body` LIKE '%query%' • gem 'thinking-sphinx' … Article.search(params[:q])
10. how search works? • document 1: flexible and powerful open source, distributed real- time search and analytics engine for the cloud... • document 2: Apache Mahout has implementations of a wide range of machine learning and data mining... • document 3: Our core algorithms for clustering, classification and batch based collaborative filtering are implemented on top of Apache Hadoop using the MapReduce...
11. how search works? data mapreduce learning classification recommenders analysis
12. how search works? data mapreduce learning classification recommenders analysis 1 2 3 1 2 2 3 2 3 2 3
13. how search works? data mapreduce learning classification recommenders analysis 1 2 3 1 2 2 3 2 3 2 3
14. how search works? data mapreduce learning classification recommenders analysis 1 2 3 1 2 2 3 2 3 2 3
15. how search works? data mapreduce learning classification recommenders analysis 1 2 3 1 2 2 3 2 3 2 3
16. elasticsearch
17. • full text search • real time data • opensource • restful api • distributed • schema free & document oriented elasticsearch
18. analysis Flexible and powerful <strong>search</strong> engine
19. analysis Flexible and powerful <strong>search</strong> engine char filters Mapping HTML Strip Pattern Replace
20. analysis char filters Flexible and powerful search engine
21. analysis char filters tokenizer Flexible and powerful search engine Path Hierarchy Keyword Letter Lowercase NGram Standard Whitespace Pattern Edge NGram
22. analysis char filters tokenizer Flexible and powerful search engine
23. analysis char filters tokenizer Flexible and powerful search engine token filters Stop Lowercase Snowball Synonym Trim Unique Normalization Stemmer Shingle Truncate Reverse
24. analysis tokenizer token filters char filters flexible powerful search engine
25. russian morphology “Он отлично рассказал о лучшем поисковом движке”
26. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный
27. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать
28. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать хороший
29. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать хороший поисковый
30. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать хороший движок поисковый
31. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать хороший движок поисковый
32. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать хороший движок “хороший поисковой движок” хороший движок поисковый поисковый
33. russian morphology “Он отлично рассказал о лучшем поисковом движке” отлично отличный рассказать хороший движок “хороший поисковой движок” хороший поисковый движок поисковый
34. phonetic analysis
36. phonetic analysis Eyjafjallajökull
37. phonetic analysis Eyjafjallajökull Eiyafyalayokul iofiolDkul
38. analysis char filters tokenizer token filters
39. analysis char filters tokenizer token filters analyzer
40. analysis char filters tokenizer token filters analyzer • has name
41. analysis char filters tokenizer token filters analyzer • has name • reusable
42. analysis char filters tokenizer token filters analysis: analyzer: rus_morphology: type: "custom" char_filter: ["html_strip"] tokenizer: "standard" filter: ["lowercase", "russian_morphology", "stopwords"]
43. getting started # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'
44. getting started # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'
45. getting started # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'
46. getting started # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: "Search" }'
47. # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }' curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }' getting started
48. # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }' curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }' # Search documents curl -XGET "localhost:9200/posts/_search?q=data" curl -XGET "localhost:9200/posts/_search?q=title:elasticsearch" getting started
49. # Add document to index curl -XPOST "localhost:9200/posts/post/1" -d '{ title: “Search" }' curl -XPOST "localhost:9200/posts/post/whatever" -d '{ title: "ES" }' # Search documents curl -XGET "localhost:9200/posts/_search?q=data" curl -XGET "localhost:9200/posts/_search?q=title:elasticsearch" # Update and delete documents curl -XPUT "localhost:9200/posts/post/1" -d '{ title: “Data" }' curl -XDELETE "localhost:9200/posts/post/whatever" getting started
50. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" }
51. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } repository: { type: "string" }
52. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } repository: { type: "string", boost: 5 }
53. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } repository: { type: "string", boost: 5, analyzer: "repo_name" } repo_name: { tokenizer: "letter", filter: ["lowercase","phonetic"] }
54. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } description: { type: "string" }
55. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } description: { type: "string", analyzer: "english_text" } english_text: { tokenizer: "standard", filter: ["lowercase", "stopwords", "snowball"] }
56. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } maintainer: { properties: { } }
57. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } maintainer: { properties: { name: { type: "string" } } }
58. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } maintainer: { properties: { name: { type: "string", analyzer: "phonetic" } } } phonetic: { tokenizer: "standard", filter: ["lowercase", "stopwords", "beidermorse"] }
59. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } maintainer: { properties: { email: { type: "string" } } }
60. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } maintainer: { properties: { email: { type: "string", index: "not_analyzed" } } }
61. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } language: { type: "string" }
62. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } language: { type: "string", analyzer: "programming_lang" } programming_lang: { tokenizer: "keyword", filter: ["lowercase"] }
63. mapping { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" } created_at: { type: "date", format: "YYYY-MM-DD" }
64. mapping curl -XPOST "localhost:9200/repositories" -d ' settings: { analysis: { analyzer: { ... }, filter: { ... } } }, mappings: { repository: { properties: { ... } } }'
65. mapping curl -XPOST "localhost:9200/repositories" -d '...' curl -XPOST "localhost:9200/repositories/repository" -d ' { repository: "elasticsearch/elasticsearch", description: "Open Source, Distributed, RESTful Search Engine", maintainer: { name: "Shay Banon", email: "kimchy@gmail.com" }, languages: ["Java", "Shell"], created_at: "2010-02-08" }'
66. search curl -XGET "localhost:9200/repositories/repository/_search?q=engine"
67. search curl -XPOST "localhost:9200/repositories/repository/_search" -d ' { query: { match: { description: "search" } } }'
68. search "hits" : { "total" : 3, "hits" : [ { "_score" : 0.22295055, "_source" : { repository: "elasticsearch/elasticsearch" } }, { "_score" : 0.22295055, "_source" : { repository: "ankane/searchkick" } }, { "_score" : 0.095891505, "_source" : { repository: "karmi/tire" } } ] }
69. search curl -XPOST "localhost:9200/repositories/repository/_search" -d ' { query: { match: { _all: "elasticsearch" } } }'
70. search "hits" : { "total" : 2, "hits" : [ { "_score" : 5.46875, "_source" : { repository: "elasticsearch/elasticsearch" } }, { "_score" : 0.04746387, "_source" : { repository: "karmi/tire" } } ] }
71. facets curl -XPOST "localhost:9200/repositories/repository/_search" -d ' { query: { match: { _all: "search" } }, facets: { language: { terms: { field: "languages" } } } }'
72. facets "hits" : { "total" : 2, "hits" : [ ... ] }, "facets" : { "language" : { "terms" : [ { "term" : "ruby", "count" : 2 }, { "term" : "shell", "count" : 1 }, { "term" : "java", "count" : 1 } ] } }
73. filters curl -XPOST "localhost:9200/repositories/repository/_search" -d ' { query: { filtered: { query: { match: { _all: "search" } }, filter: { term: { "languages": "java" } } } } }'
74. filters "hits" : { "total" : 1, "hits" : [ { "_score" : 5.46875, "_source" : { repository: "elasticsearch/elasticsearch" } } ] }
75. performance and scaling
76. performance and scaling elasticsearch is web scale
77. random facts • bulk operations • real time • highlights • geo types and geo distance facets • attachments • “did you mean?” and completions • common terms • filters and caching • river
78. You know. For search.