Elasticsearch and Recommendations

Quick overview about:
* Elasticsearch
* The Elastic Stack
* How you can get recommendations from Elasticsearch. Either in combination with Spark or through Elasticsearch aggregations.

Transcript

1. and Recommendations ̴̴Philipp Krenn̴̴̴@xeraa
2. Agenda Elasticsearch Intro Recommender Systems Recommendations with Elasticsearch
3. https:/ /db-engines.com/en/ranking
5. https:/ /www.reddit.com/r/SQL/comments/7i828i/ms_sql_query_is_too_complex/
7. Who uses Elasticsearch?
8. Infrastructure | Developer Advocate
9. http:/ /thedudeabides.com/articles/the_birth_of_compass
11. $ curl http://localhost:9200 { "name": "instance-0000000003", "cluster_name": "44d59d42507ebde50eec3dd749c45b86", "cluster_uuid": "4mZwuq5eS4ieREDzVq0B_Q", "version": { "number": "6.1.1", "build_hash": "bd92e7f", "build_date": "2017-12-17T20:23:25.338Z", "build_snapshot": false, "lucene_version": "7.1.0", "minimum_wire_compatibility_version": "5.6.0", "minimum_index_compatibility_version": "5.0.0" }, "tagline": "You Know, for Search" }
12. $ curl http://localhost:9200 { "name": "instance-0000000003", "cluster_name": "44d59d42507ebde50eec3dd749c45b86", "cluster_uuid": "4mZwuq5eS4ieREDzVq0B_Q", "version": { "number": "6.1.1", "build_hash": "bd92e7f", "build_date": "2017-12-17T20:23:25.338Z", "build_snapshot": false, "lucene_version": "7.1.0", "minimum_wire_compatibility_version": "5.6.0", "minimum_index_compatibility_version": "5.0.0" }, "tagline": "You Know, for Search" }
14. Search?
16. Apache Lucene Elasticsearch
18. Example These are <em>not</em> the droids you are looking for.
19. html_strip Char Filter These are not the droids you are looking for.
20. standard Tokenizer These̴are̴not̴the̴droids̴you̴ are̴looking̴for
21. lowercase Token Filter these̴are̴not̴the̴droids̴you̴ are̴looking̴for
22. stop Token Filter droids̴you̴looking
23. snowball Token Filter droid̴you̴look
24. GET /_analyze { "analyzer": "english", "text": "These are not the droids you are looking for." }
25. { "tokens": [ { "token": "droid", "start_offset": 18, "end_offset": 24, "type": "<ALPHANUM>", "position": 4 }, { "token": "you", "start_offset": 25, "end_offset": 28, "type": "<ALPHANUM>", "position": 5 }, ... ] }
26. GET /_analyze { "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop", "snowball" ], "text": "These are <em>not</em> the droids you are looking for." }
27. { "tokens": [ { "token": "droid", "start_offset": 27, "end_offset": 33, "type": "<ALPHANUM>", "position": 4 }, { "token": "you", "start_offset": 34, "end_offset": 37, "type": "<ALPHANUM>", "position": 5 }, ... ] }
28. Search++
37. Aggregations Logs, Metrics, Pings,...
39. Recommender Systems
40. Recommendations with Elasticsearch
41. https:/ /github.com/IBM/elasticsearch-spark-recommender
43. Elasticsearch: Common vs Significant
44. Common POST movies/user/_search { "size": 0, "query": { "match": { "movies_liked": "Terminator" } }, "aggregations": { "movies_like_terminator": { "terms": { "field": "movies_liked.keyword", "min_doc_count": 1 } } } }
45. Significant POST movies/user/_search { "size": 0, "query": { "match": { "movies_liked": "Terminator" } }, "aggregations": { "movies_like_terminator": { "significant_terms": { "field": "movies_liked.keyword", "min_doc_count": 1 } } } }
46. Significance: JLH https:/ /github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/search/aggregations/bucket/significant/heuristics/JLHScore.java double subsetProbability = (double) subsetFreq / (double) subsetSize; double supersetProbability = (double) supersetFreq / (double) supersetSize; double absoluteProbabilityChange = subsetProbability - supersetProbability; if (absoluteProbabilityChange <= 0) { return 0; } double relativeProbabilityChange = (subsetProbability / supersetProbability); return absoluteProbabilityChange * relativeProbabilityChange;
47. Significance of Rambo with Terminator subsetProbability: 50% supersetProbability: 25% JLH: 0.5
48. Conclusion
49. Elasticsearch Elastic Stack Recommendations
50. Questions? ̴̴ Philipp Krenn̴̴̴@xeraa