Beyond the basics with Elasticsearch

Transcript

1. Beyond the basics with Elasticsearch Honza Král @honzakral
2. Search
3. Bible concordance A simple form lists Biblical words alphabetically, with indications to enable the inquirer to find the passages of the Bible where the words occur. The first concordance, completed in 1230, was undertaken under the guidance of Hugo de Saint-Cher (Hugo de Sancto Charo), assisted by fellow Dominicans.
4. Inverted index python file_1.txt file_2.txt file_3.txt web file_2.txt file_3.txt file_2.txt file_4.txt django file_3.txt flask jazz file_4.txt
5. "Python" and "Django" python file_1.txt file_2.txt file_3.txt file_2.txt file_4.txt django file_3.txt flask jazz file_4.txt web file_2.txt file_3.txt
6. Reversed search index queries, run documents Alerting threshold reached Classification language, geo Percolator
7. Geo Classification curl -XPUT 'localhost:9200/events/.percolator/city-warsaw' -d '{ "query": {"filtered": {"filter": { "geo_shape": { "location": { "indexed_shape": { "index": "shapes", "type": "city", "id": "warsaw", "path": "area" } } } } } } }'
8. Language Detection curl -XPUT 'localhost:9200/events/.percolator/lang-polish' -d '{ "query": { "match": { "description": { "query": "cześć chrząszcz Żubrówka Wyborowa kur...", "type": "boolean", "minimum_should_match": 4 } } } }'
9. Classification curl -XPUT 'localhost:9200/events/conf/_percolate' -d '{ "doc": { "title": "pywaw summit", "description": "konferencja zorganizowana dla wszystkich zainteresowanych w technologii związanej z Pythona...", "location" : "40.73,-74.1" } }' { ... "matches" : [ {"_index" : "events", "_id" : "city-warsaw"}, {"_index" : "events", "_id" : "lang-polish"}, {"_index" : "events", "_id" : "topic-python"} ] }
10. Relevancy
11. Standard is TF-IDF good for text Lucene adds on top factors text length etc. Sometimes not good enough user contributed "quality" time/space decay ... Relevancy 0 0.2 0.4 0.6 0.8 1 20 30 40 50 60 score age "gauss" "exp" "lin" decay reference scale offset
12. Hotel Search "function_score": { "query": {"match": {"name": "grand hotel"}}, "functions": [ { "filter": {"terms": {"facilities": ["balcony"]}}, "boost_factor": 2 },{ "gauss": { "field": "location", "scale": "1km", "reference": [51,0] } },{ "field_value_factor": {"field“: "popularity"} },{ "random_score": {} } ] }
13. Aggregations
15. Recommendations
16. Example: recommendations Product A user A purchase history Product B Product C user B purchase history Product D Users represented as documents Products represented as terms
17. Dumb approach to recommendation (no relevance) Popular != Relevant { "query": { "terms": { "artists": user_likes} }, "aggs":{ "popular":{ "terms":{ "field": "artists", "exclude": user_likes } } } }
18. Strong recommendations based on relevance Use the score! Compare to background { "query": { "terms": { "artists": user_likes} }, "aggs":{ "significant":{ "significant_terms":{ "field": "artists", "exclude": user_likes } } } }
19. www.elastic.co Super-connected nodes in graphs We just figured out the way to surf only the meaningful connections in a graph! Concept A Concept B Concept C useful useless
20. Meaningful Connections
21. www.elastic.co Example: wikipedia Yoko Ono John Lennon article John Lennon United States Beatles article Ringo Starr Wikipedia articles are documents Links and page titles are terms
22. Aggregation + Relevancy is how we look at world
23. Thanks! Honza Král @honzakral