Search, Stay Relevant

This is an introduction to search using elasticsearch and the elastic stack.
It was hosted at GoMyCode @ Tunis.
https://www.eventbrite.com/e/search-and-stay-relevant-tickets-48487421182

Transcript

1. 1 Taha Derouiche taha@elastic.co Elastic Search, Stay Relevant
2. 2 Taha Derouiche • Consulting Engineer @ elastic • 6 years in various tech roles • 2 years elasticsearch • https://www.linkedin.com/in/tahaderouiche • taha@elastic.co Who am I?
3. 3 Agenda Introduction to search Elasticsearch and elastic stack Full-text search explained Other use cases of the elastic stack Questions 1 2 3 4 5
4. 4 Most think of search as…
5. 5 SEARCH
6. 6 Multilingual Full Text Search Stemming Type ahead
7. 7 Mobile Geo search Influenced by Rating Personalized Ranking Time Range
9. 9 Stemming / Highlighting Geo range Filter Search Time range Filter Pagination Numeric Filter
10. 10 10
11. 11 Building search needs ...
12. 12 All this. Secure Multi-lingual Query language Drill down APIs Enrichment Alerting Query Language Unstructured Search Type ahead High availability Fast Highlighting Scalable
13. 13 Elastic has all this. Secure Multi-lingual Query language Real-time Drill down APIs Data Sources Enrichment Alerting Query Language Unstructured Search Type ahead High availability Fast Highlighting Scalable
14. 14 That's the end goal of elasticsearch: We want to make data exploration, the ability to go ahead and ask questions on your data and get results in milliseconds, available to end users. Shay Bannon dotScale 2013
15. 15 Open Source
16. 16 16 Elastic Stack Store, Search, & Analyze Elasticsearch Visualize & Manage Kibana Beats Ingest Logstash
17. 1 100,000+ Meetup Members 250M+ Product Downloads 5000+ Subscription Customers Statistics since 2012, founding of Elastic
18. Millions of Downloads 225 2012 Cumulative downloads of all Elastic products, including Elasticsearch, Kibana, Beats, and Logstash 1 2018 …
19. 19 E-Commerce Demo
20. 20 Demo
21. 21 21 "query": { "bool": { "must": [ { "multi_match": { "query": "white shirt", "fields": [ "product_name^1", "product_name.english^1", "slug", "description^1", "description.english^1", "manufacturer", "category", "attributes.gender" ], "operator": "or", "minimum_should_match": "50%", "type": "best_fields" } } ] } }
22. 22 Full-text search explained
23. 23 Demo walkthrough • https://github.com/tahaderouiche/elasticsearch-gomycode • Prerequisites: ‒ Elasticsearch Service on Elastic Cloud: Free trial OR ‒ Docker + Docker compose: Setup locally Steps Source: Gray Arial10pt
24. 24 24 Example I will either find a <em>way</em> or make one.
25. 25 I will either find a way or make one.
26. 26 26 Example I will either find a <em>way</em> or make one.
27. 27 html_strip Char Filter I will either find a way or make one.
28. 28 28 standard Tokenizer I will either find a way or make one
29. 29 lowercase Token Filter i will either find a way or make one
30. 30 30 Stop Token Filter i either find way make one
31. 31 Analyzers
32. 32 32 GET /_analyze { "analyzer": "english", "text": "I will either find a way or make one." }
33. 33 "tokens": [ { "token": "i", "start_offset": 0, "end_offset": 1, "type": "<ALPHANUM>", "position": 0 }, { "token": "either", "start_offset": 7, "end_offset": 13, "type": "<ALPHANUM>", "position": 2 }, ... ]
34. 34 34 GET /_analyze { "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop" ], "text": "I will either find a <em>way</em> or make one." }
35. 35 Stop words a an and are as at be but by for if in into is it no not of on or such that the their then there these they this to was will with
36. 36 Always Use Stop Words?
37. 37 GET /_analyze { "analyzer": "english", "text": "To be, or not to be." }
38. 38 38 Languages Arabic, Armenian, Basque, Brazilian, Bulgarian, Catalan, CJK, Czech, Danish, Dutch, English, Finnish, French, Galician, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Latvian, Lithuanian, Norwegian, Persian, Portuguese, Romanian, Russian, Sorani, Spanish, Swedish, Turkish, Thai
39. 39 GET /_analyze { "analyzer": ”french", "text": "Je trouverai un chemin, ou j’en créerai un." }
40. 40 Synonyms
41. 41 41 " I will either find a way or make one.” " I will either find a way or create one.”
42. 42 PUT /hannibal { "settings": { "analysis": { "filter": { "my_synonym_filter": { "type": "synonym", "synonyms": [ "make,create" ] } },
43. 43 "analyzer": { "my_analyzer": { "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop", "my_synonym_filter" ] } } }
44. 44 "mappings": { "_doc": { "properties": { "quote": { "type": "text", "analyzer": "my_analyzer" } } } }
45. 45 45 PUT /hannibal/_doc/1 { "quote": "I will either find a <em>way</em> or make one." } PUT /hannibal/_doc/2 { "quote": "I have come not to make war on the Italians, but to aid the Italians against Rome." } PUT /hannibal/_doc/3 { "quote": "God has given to man no sharper spur to <b>victory</b> than contempt of death." }
46. 46 Search
47. 47 POST /hannibal/_search { "query": { "match_all": { } } }
48. 48 48 POST /hannibal/_search { "query": { "match": { "quote": "make" } } }
49. 49 49 POST /hannibal/_search { "query": { "match": { "quote": ”create" } } }
50. 50 Score
51. 51 "hits": { "total": 2, "max_score": 0.4121213, "hits": [ { "_index": "hannibal", "_type": "_doc", "_id": "1", "_score": 0.4121213, "_source": { "quote": "I will either find a <em>way</em> or make one." } }, { "_index": "hannibal", "_type": "_doc", "_id": "2", "_score": 0.40594208, "_source": { "quote": "I have come not to make war on the Italians, but to aid the Italians against Rome." } } ] }
52. 52 52 BM25 After elasticsearch version 5.0 TF/IDF before https://www.youtube.com/watch?v=v3Ko0CwgTZ0
53. 53 What is BM25?
54. 54 54 Term Frequency Inverse Document Frequency Field-Length Norm
55. 55 !" !#$%, '() = "$#+,#-). Term Frequency
56. 56 !"# $%&' = 1 + log( /012345 634789:;< ) Inverse Document Frequency
57. 57 !"#$ %"& = 1 !)$*+#$, Field-Length Norm
58. 58 58 POST /hannibal/_search?explain=true { "query": { "match": { "quote": "make" } } }
59. 59 There is a lot more Fuzziness Multiple Analyzers Phrase Search Suggester
60. 60 60 So many more use cases with Elastic Stack
61. 61 Elastic Solutions elastic.co/solutions
62. 62 Thank You QUESTIONS? Taha Derouiche @tahaderouiche We are hiring! elastic.co/careers