Full-Text Search: MongoDB vs Elasticsearch

Today’s applications are expected to provide powerful full-text search. But how does that work in general and how do I implement it on my site or in my application?

Actually, this is not as hard as it sounds at first. This talk covers:
* How full-text search works in general and what the differences to databases are.
* How the relevancy of documents is being calculated.
* How search works in MongoDB and Elasticsearch as well as what the differences between the two systems are.

Transcript

1. Full-Text Search Explained Philipp Krenn̴̴̴̴@xeraa
5. Infrastructure | Developer Advocate
6. ViennaDB Papers We Love Vienna
7. Who uses databases?
8. Who uses search?
9. Databases vs Full-text search
11. But I can do... SELECT * FROM my_table WHERE my_text LIKE ‘%blue%’
13. 1. Speed B-Tree
16. 2. Features Fuzziness, synonyms, scoring,...
17. Storing
18. Indexing Remove formatting
19. Indexing Tokenize
20. Indexing Stop words
21. Indexing Stemming http://snowballstem.org
22. Indexing Synonyms
23. MongoDB
24. Before FTS Regular expressions !
25. Before FTS Arrays with relevant terms ! https://docs.mongodb.com/manual/tutorial/model-data-for- keyword-search/
26. { title : "Moby-Dick" , author : "Herman Melville" , published : 1851 , ISBN : 0451526996 , topics : [ "whaling" , "allegory" , "revenge" , "American" , "novel" , "nautical" , "voyage" , "Cape Cod" ] } db.volumes.createIndex({ topics: 1 }) db.volumes.findOne({ topics : "voyage" }, { title: 1 })
27. No Stemming Fuzziness Synonyms Scoring
28. Finally FTS 230+ votes Created 2009 Resolved 2013 https://jira.mongodb.org/browse/SERVER-380
29. FTS in MongoDB Beta since 2.4 Stable since 3.0 "80% solution" — for more Elasticsearch
30. FTS in MongoDB In Latin alphabets Case insensitive (default in 3.2) [A-z] other characters removed (default in 3.2)
31. Indexing String or array of strings Optional language or translations Optional weight if multiple fields indexed
32. $text Updated version in MongoDB 3.2 { $text: { $search: "<string>", $language: "<string>", $caseSensitive: <boolean>, $diacriticSensitive: <boolean> } }
34. Example Text These are not the droids you are looking for.
35. Tokenizer these̴are̴not̴the̴droids̴you̴are̴ looking̴for
36. Stop Words droids̴looking
37. Stemming droid̴look
38. > db.starwars.ensureIndex({ quote: "text" }) { "createdCollectionAutomatically": true, "numIndexesBefore": 1, "numIndexesAfter": 2, "ok": 1 }
39. > db.starwars.getIndices() [ ... { "v": 1, "key": { "_fts": "text", "_ftsx": 1 }, "name": "quote_text", "ns": "starwars.starwars", "weights": { "quote": 1 }, "default_language": "english", "language_override": "language", "textIndexVersion": 3 } ]
40. > db.starwars.insert( { quote: "These are not the droids you are looking for." } ) Inserted 1 record(s) in 39ms WriteResult({ "nInserted": 1 })
41. > db.germanStarwars.ensureIndex( { "$**": "text"}, { default_language: "german" } )
42. Elasticsearch
43. Apache Lucene Elasticsearch
44. Example Text These are <em>not</em> the droids you are looking for.
45. html_strip Char Filter These are not the droids you are looking for.
46. standard Tokenizer These̴are̴not̴the̴droids̴you̴are̴ looking̴for
47. lowercase Token Filter these̴are̴not̴the̴droids̴you̴are̴ looking̴for
48. stop Token Filter droids̴you̴looking
49. snowball Token Filter droid̴you̴look
50. GET /_analyze { "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop", "snowball" ], "text": "These are <em>not</em> the droids you are looking for." }
51. { "tokens": [ { "token": "droid", "start_offset": 27, "end_offset": 33, "type": "<ALPHANUM>", "position": 4 }, { "token": "you", "start_offset": 34, "end_offset": 37, "type": "<ALPHANUM>", "position": 5 }, ... ] }
52. Another Example Obi-Wan never told you what happened to your father.
53. Another Example obi̴wan̴never̴told̴you̴what̴ happen̴your̴father
54. Another Example <b>No</b>. I am your father.
55. Another Example i̴am̴your̴father
56. Language Rules English: Philipp's → philipp French: l'église → eglis German: äußerst → ausserst
57. phonetic Token Filter Plugin Joe Bloggs → J̴joe̴BLKS̴bloggs
58. Elasticsearch Index, type, mapping
59. PUT /my_index { "settings": { "analysis": { "filter": { "my_synonym_filter": { "type": "synonym", "synonyms": [ "word1,synonym", "word2,synonym" ] } },
60. "analyzer": { "my_analyzer": { "char_filter": [ "html_strip" ], "tokenizer": "standard", "filter": [ "lowercase", "stop", "snowball", "my_synonym_filter" ] } } } },
61. "mappings": { "my_type": { "properties": { "my_title": { "type": "text", "analyzer": "my_analyzer" } } } } }
62. PUT /starwars/quotes/1 { "quote": "These are <em>not</em> the droids you are looking for." } PUT /starwars/quotes/2 { "quote": "Obi-Wan never told you what happened to your father." } PUT /starwars/quotes/3 { "quote": "<b>No</b>. I am your father." }
63. Inverted Index ID 1 ID 2 ID 3 am 0 0 1[2] droid 1[4] 0 0 father 0 1[9] 1[4] happen 0 1[6] 0 i 0 0 1[1] look 1[7] 0 0 never 0 1[2] 0 obi 0 1[0] 0 told 0 1[3] 0 wan 0 1[1] 0 what 0 1[5] 0 you 1[5] 1[4] 0 your 0 1[8] 1[3]
64. Searching
65. MongoDB
66. > db.starwars.insert( { quote: "Obi-Wan never told you what happened to your father." } ) > db.starwars.insert( { quote: "No. I am your father." } )
67. > db.starwars.find({ $text: { $search: "droid" }}) { "_id": ObjectId("57f2d54de814412463c3adef"), "quote": "These are not the droids you are looking for." } Fetched 1 record(s) in 35ms
68. > db.starwars.find({ $text: { $search: "father" }}) { "_id": ObjectId("57f2d56fe814412463c3adf0"), "quote": "Obi-Wan never told you what happened to your father." } { "_id": ObjectId("57f2d581e814412463c3adf1"), "quote": "No. I am your father." } Fetched 2 record(s) in 3ms
69. > db.starwars.find({ $text: { $search: "droid" }}).explain() { "queryPlanner": { ... "$text": { "$search": "droid", "$language": "english", "$caseSensitive": false, "$diacriticSensitive": false } },
70. "winningPlan": { "stage": "TEXT", "indexPrefix": { }, "indexName": "quote_text", "parsedTextQuery": { "terms": [ "droid" ], "negatedTerms": [ ], "phrases": [ ], "negatedPhrases": [ ] }, ...
71. > db.starwars.find({ $text: { $search: "father -obi" }}) { "_id": ObjectId("57f2d581e814412463c3adf1"), "quote": "No. I am your father." } Fetched 1 record(s) in 4ms
72. > db.starwars.find({ $text: { $search: "father -obi" }}).explain() ... "parsedTextQuery": { "terms": [ "father" ], "negatedTerms": [ "obi" ], "phrases": [ ], "negatedPhrases": [ ] }, ...
73. Queries // OR > db.starwars.find({ $text: { $search: "look droid" } }) // AND but without input stemming > db.starwars.find({ $text: { $search: "\"look\" \"droid\"" } }) // Negation > db.starwars.find({ $text: { $search: "look -droid" } }) // Phrase > db.starwars.find({ $text: { $search: "\"look droid\"" } }) // Translation > db.starwars.find({ $text: { $search: "suchen", $language: "de" } })
74. Elasticsearch
75. POST /starwars/_search { "query": { "match": { "quote": "droid" } } }
76. { "took": 2, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 1, "max_score": 0.39556286, "hits": [ { "_index": "starwars", "_type": "quotes", "_id": "1", "_score": 0.39556286, "_source": { "quote": "These are <em>not</em> the droids you are looking for." } } ] } }
77. POST /starwars/_search { "query": { "match": { "quote": { "query": "van", "fuzziness": "AUTO" } } } }
78. { "took": 14, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 1, "max_score": 0.18155496, "hits": [ { "_index": "starwars", "_type": "quotes", "_id": "2", "_score": 0.18155496, "_source": { "quote": "Obi-Wan never told you what happened to your father." } } ] } }
79. SELECT * FROM starwars WHERE quote LIKE "?an" OR quote LIKE "V?n" OR quote LIKE "Va?"
80. Scoring
81. MongoDB
82. > db.starwars.find({ $text: { $search: "droid" }}, {score: {$meta: "textScore"}}) { "_id": ObjectId("57f2d54de814412463c3adef"), "quote": "These are not the droids you are looking for.", "score": 0.75 } Fetched 1 record(s) in 14ms
83. One Term https://github.com/mongodb/mongo/blob/v3.2/src/mongo/db/fts/fts_spec.cpp#L219 double coeff = (0.5 * data.count / numTokens) + 0.5; data.count: matches numTokens: stemmed words
84. Search for droid "These are not the droids you are looking for." droid look == 1 match, 2 tokens coeff:
85. Search for father "No. I am your father." father == 1 match, 1 token coeff:
86. Search for father "Obi-Wan never told you what happened to your father." obi wan never told happen father == 1 match, 6 tokens coeff:
87. > db.starwars.find({ $text: { $search: "obi-wan" }}, {score: {$meta: "textScore"}}) { "_id": ObjectId("57f2d56fe814412463c3adf0"), "quote": "Obi-Wan never told you what happened to your father.", "score": 1.1666666666666667 } Fetched 1 record(s) in 6ms
88. Multiple Terms https://github.com/mongodb/mongo/blob/v3.2/src/mongo/db/fts/fts_spec.cpp#L228 score += (weight * data.freq * coeff * adjustment); weight: method parameter data.freq, adjustment: 1
89. Search for obi-wan obi wan never told happen father == 1 match, 6 tokens coeff:
90. Search for obi-wan obi wan never told happen father == 1 match, 6 tokens coeff:
91. Search for obi-wan score: Sum:
92. Elasticsearch
93. Term Frequency / Inverse Document Frequency (TF/IDF) Search one term
94. BM25 https://speakerdeck.com/elastic/ improved-text-scoring-with-bm25
95. Term Frequency
97. Inverse Document Frequency
99. Field-Length Norm
100. Putting it Together score(q,d) = queryNorm(q) · coord(q,d) · ∑ ( tf(t in d) · idf(t)² · t.getBoost() · norm(t,d) ) (t in q)
101. POST /starwars/_search?explain { "query": { "match": { "quote": "father" } } }
102. "_explanation": { "value": 0.2876821, "description": "weight(quote:father in 0) [PerFieldSimilarity], result of:", "details": [ { "value": 0.2876821, "description": "score(doc=0,freq=1.0 = termFreq=1.0\n), product of:", "details": [ { "value": 0.2876821, "description": "idf(docFreq=1, docCount=1)", "details": [] }, ...
103. Score 0.2876821 vs 0.27233246
104. Vector Space Model Search multiple terms
105. Score each term Vector Calculate angle
106. Search your obi
108. Conclusion
109. Indexing Formatting Tokenize Lowercase, stop words, stemming Synonyms
110. Elasticsearch Scoring Term Frequency Inverse Document Frequency Field-Length Norm
111. Elasticsearch Scoring Vector Space Model
112. MongoDB Limitations B-tree vs inverted index Simple scoring
113. Elasticsearch Features Fuzziness Synonyms Suggestions & highlighting Documentation
114. 80 - 20 solution or 20 - 80 solution?
115. More https://www.elastic.co/training Amsterdam: Nov 21-24
116. Thanks! Questions? Philipp Krenn̴̴̴̴̴@xeraa PS: Stickers
117. Image Credit → Schnitzel https://flic.kr/p/9m27wm → Architecture https://flic.kr/p/6dwCAe → Conchita https://flic.kr/p/nBqSHT → Black and grey http://hdimagelib.com/ zedge+quote+wallpapers